On the 6th of October, my blog went offline as a result of what was quickly but adequately branded as the pizmess. The piz(arro)mess started in fact on the 4th of October, as an apparently targeted ddos attack on republican infrastructure including my logger machine hosted with MivoCloud as well as most/all Pizarro-hosted machines. While my Moldova-based provider needed however only a quick email from me to update their rules and mitigate the "attack", the Uruguay-based data centre turned this initial blip into a full-blown catastrophy for Pizarro, nullrouting their IPs for a whole week and acting overall increasingly ineptly over it too.

As the uruguayan noise just increased along the lines of "ra-raaa attack, can't do anything other than nullroute!!", I even pinged again my moldavian provider to get some concrete data as to what terrible thing had hit my logger. And here it is, all of 1.5Gbit/s peak of it, with dump.pcap file attached and coming from just a few IPs really:

The incredible 2019 ddos on logs.ossasepia.com

While my logger was therefore quickly back online and happily serving TMSR, my blog and Minigame's servers were not to recover as the pizmess quickly went from bad to worse. As a result, Pizarro, the in-WoT preferred provider of dedicated servers and colocation services went out of business within a week, as its data centre¹ proved to have irremediably turned into a scam and a rather inept one at that, involving nullrouting of paying customers' IPs on the basis of no data whatsoever and failed/inexistent cooling systems in place. On the very same data centre, a report less than 2 years old stated the rather opposite "Given the size and ratio of coolers to cabinets there seems to be a healthy amount of excess capacity". In any case, overall the exact speed at which this centre->scam conversion happened is unclear but the whole thing seems to have taken Pizarro management entirely by surprise as there was no plan B in place and nothing to do other than to wheel the data and the hardware out of there as soon as possible, contact local lawyers, file a police complaint and otherwise write the adventure's epitaph: servers for sale, in Uruguay. All of this was done and done timely, to Aaron's credit.

Still, despite all the good intentions, my younghands.club remained unreachable and this affected worst precisely newcomers and learners, the exact sort of situation I could have done without. My blog also remained offline and all my up-to-date backups of it (as of all other data, Minigame's as well as mine, of all sorts) could still not paper over the sinking feeling of atrophiating writing muscle that came with the situation. So I had no choice but to look for a new provider to host both younghands.club and ossasepia.com.

The initial logical² option was to attempt expanding the contact with Maxim from MivoCloud rather faster than initially contemplated. Responsive as ever, he came in for a talk when asked and was very willing to host republican machines as they come but in the end the whole thing fell through because of his unexpected refusal to actually use his acquired position in the WoT in any meaningful sense. So with this option closed, what else was there to do then to hit the list of ISPs again and start contacting more of them to see if I could by any chance find one I could actually work with. And as this tiresome talking to all sorts of pretend ISPs went on and on with mostly headache to show for it and very little else³, the younghands.club that could not wait anymore was set up with MivoCloud and this blog here finally made it back online only when Stanislav discovered that he actually had all he needed - and cheaper too! - within 20 minutes drive from his very own desk.

So with many thanks to Stanislav for his quick and smooth bringing up of all my blog-data⁴ on the new Rockchip in the USA, the first thing I did upon getting my blog back was to shudder at the ugly theme I had lived with for lack of anything else to choose from. The thing with crises like this is that they force anyway such an expense of time and effort that in the wake of it I get triggered again by *all* the pending/unsolved/still-broken issues and with an intensity that has deep roots I know only too well about. So I usually weigh very carefully and hold on to acting too quickly on those triggers so as to not end up doing all sorts rather than the most useful and truly needed sorts. Still, I considered that the new home for my new blog could do with a maximum of one hour spent on hammering an existing theme into something I could at least stand slightly better than that green thing.

Funnily enough it turned out that even less than 60 minutes on this task can make quite a difference: the "default" theme with the blue header morphed into what you can see with just a few touches to the .css files (both of them) to change some colours, some fonts and the wasteful padding of single-post pages, as well as a few further touches to .php to put in the server-side selection mechanism and tweak odd bits and pieces so it doesn't look *that* terrible. Hopefully there will be some themes I can choose from before my next "can't stand this anymore" hits again with such intensity.

In other technical terms and at Stanislav's request, I did also a few tests of the new location, here's some data for the hungry reader:

• From ossasepia.com server (USA), the ftp get aimed at logs.ossasepia.com (Moldova) went like this: 1105154420 bytes received in 539.60 secs (2000.1 kB/s)
• The other way around, from logs.ossasepia.com server, a wget of a file ~86MB went like this: 100%[======================================>] 89,944,680 131K/s in 10m 11s
• While the above may look surprising, the issue is not on the USA side I'd think. Here's how the same wget went from my home (UK) to ossasepia.com (USA): 100%[======================================>] 89,944,680 998K/s in 1m 45s

Now that my blog is back up and running on Stanislav-owned iron again, there is of course the whole backlog of write-ups to be done on top of all the crisis-induced tasks plus usual tasks plus new tasks. How do adult people *ever* manage to get bored?

1. Latecho, of Uruguay ↩

2. Not quite ideal though, partially because of grouping again *everything* in a single place and partially because of location (on Internet pipes map rather than purely geographical) and distance from any L1 presence. ↩

3. There is in fact a pile of emails to show for it + at least two ISP representatives that made it to register a key with deedbot if not really any further than that. And now I have of course the added task of going again through the pile of emails and doing a write-up of it all, preferably sooner rather than later. I know it and my every-growing list of stuff-to-be-done overflowingly-knows it too, what more can I say about it. ↩

4. Thanks to Aaron too, for his packing and sending of data from Uruguay. ↩

Category: TMSR

Comments feed: RSS 2.0