[plug] Dumb stuff recovery

[Dean Bergin]

Nice recovery!

I remember when I managed to trash a raw disk image of a prod server (VM)
on a KVM hypervisor host, which I managed to recover by performing a dd
from *inside* the running VM back to the hypervisor host over SSH which
would not have been possible if the VM was shut down after the disk image
deletion as that would have synchronised the hosts filesystem and all hope
would have been lost especially since there was no backup.

I may have blogged an article on my historical blunder and recovery...

Thanks for sharing this little nugget.

On Mon, 26 Aug 2024, 08:16 Brad Campbell, <brad at fnarfbargle.com> wrote:

> I learned something new today.
>
> I have a duplicate test server here I use to stage and script all updates
> on the prod machine. I had just done a major OS update and was getting
> ready to tweak the package manager. Before I did that I thought I'll backup
> the root fs entirely. I did that using rsync to a mounted RAID6 on /server.
> I performed a few tweaks and wanted to undo them, so I cd to the backup
> directory and ran this :
>
> rsync -avxH --delete /root/ /
>
> What I should have typed was rsync -avxH --delete root/ /
>
> So I wiped out the entire root filesystem excepting the root directory.
> That left me with a running root shell and nothing else. All commands were
> gone. Unless it was a bash builtin I was stuffed.
>
> I never knew you could run the dynamic loader as an executable.
>
> /server/root/lib/x86_64-linux-gnu/ld-2.31.so --library-path
> /server/root/usr/lib:/server/root/usr/lib/x86_64-linux-gnu:/server/root/lib/x86_64-linux-gnu/
> /server/root/usr/bin/rsync -avxH --delete /server/root/ /
>
> And I was back in business. Bad day when you don't learn something!
>
> Thankfully bash completion still worked, although I still had the fs
> skeleton on the prod server to refer to for directory names just in case.
>
> Regards,
> Brad
> _______________________________________________
> PLUG discussion list: plug at plug.org.au
> http://lists.plug.org.au/mailman/listinfo/plug
> Committee e-mail: committee at plug.org.au
> PLUG Membership: http://www.plug.org.au/membership
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.plug.org.au/pipermail/plug/attachments/20240826/8fb0407d/attachment.html>