Post mortem…
Well, I’m getting closer to getting everything back. It’s going to take days if not weeks to recover everything. I run a RAID-5 SCSI array on my webserver. The array consists of 4 primary disks + one hot spare. Well, what happened was a cascade of crap that caused the loss/misplacement of a bunch of data in the array.
Timeline:
– A spammer decides that he’s going to use one of my email addresses as a fake return email. Ok, it’s happened before.
– The resulting torrent of spam gives my server fits but is no problem. It handles it in stride.
– Road Runner sees a shitload of connects to port 25 from my server trying to send back bounce notifications and cuts me off about 2AM.
– Sometime in here I have a disk go bad in the array.
– I wake up and can’t get online.
– My server is now unreachable for some reason. My only choice, and a bad one at that, was to hard power the machine.
– Upon coming online the RAID array decides it doesn’t want to play nice and trashes its journal file. fsck recovers most stuff but tosses it in lost+found instead of its previous spot.
– My backup drive for all this was useless. For unknown reasons it will not mount on any of my machines.
– I get to now spend hours/days/weeks going through lost+found moving files to their correct places.
Kill me now…