A lot's gone on since my last entry.
On burning chrome
My former colo server went down hard a week or two ago. It was running a pretty old version of Ubuntu (17.10) - old enough that the packages no longer worked. This was a pretty solid box; I'd taken over the lease on it, and it was probably seven or eight years old at this point. Anyways, I go to do the upgrade and naturally, it fails. Okay. So time to get out the Ubuntu 19.04 installer, schedule an IP KVM hookup to the machine, and try to upgrade that way. All this manages to do is mangle one of the drives, trying to install on top of it instead instead of upgrading. The drives had been previously set up as RAID1 with LUKS on top of that; so I went ahead and started to dd the good drive (sdb) to the overwritten drive.
Except one little problem:
dd: error reading '/dev/sdb': Input/output error 716513280 bytes (717 MB, 683 MiB) copied, 38.0286 s, 18.8 MB/s
So far, safecopy has only managed to salvage part of the drive:
Recovered bad blocks: 90 Unrecoverable bad blocks (bytes): 59635 (3905703936) Blocks (bytes) copied: 488378646 (2000398934016)
Since this drive is encrypted and there's enough bad blocks, it's pretty unlikely I'll be able to recover my data - especially since I don't have a good way to detect corruption. mdadm and LVM can't find the drive, and I think it's just pretty well hosed. I lost a lot of things I didn't want to lose, like my photography site. So that sucks. I need to get the rest of my sites back online, too - which means trying to finangle Haskell into building my personal site.
I ended up replacing that one server with two other servers: one has NVMe drives and is going to be a services / sites box, and the other is a storage and general development machines. Both run OpenBSD, and it feels good to be back. That being said, you still can't stack softraid modes, which means I can't stack crypto on top of RAID1 - which means I have to figure out my own backup strategy. A plain dd of one NVMe drive onto another took about two hours, which is not acceptable. I imagine it'd be even longer with the 4T drives, which are eight times the size of the SSDs.