[plug] Debian 10-11 upgrade issues

[Dean Bergin]

Hello Nick!

I pulled the trigger too soon.

I dug a bit deeper and discovered the problem is indeed mdadm itself, or
more specifically, the config file.

It turns out that either mdadm or kernel behaviour appears to have changed
between versions, so I commented out the apparently-still-valid 'DEVICE'
section in mdadm.conf then rebooted and I'm now back in business! 😁

Thanks for the links though much appreciated. On that note, does
anyone have any recommendations on doing bare-metal snapshot/restores at
the root FS (preferably live)?

I'd love to find a solution to managing a bare-metal host like a VM to
avoid situations like this in future.
I can't really operate the host as a VM as it's a kind of hyperconverged
hypervisor host, so I generally require bare-metal and direct access to all
devices but the GPU.


Kind Regards,

*Dean Bergin*.


On Wed, 28 Jun 2023 at 11:27, Nick Bannon <nick at ucc.gu.uwa.edu.au> wrote:

> On Wed, Jun 28, 2023 at 10:03:54AM +0800, Dean Bergin wrote:
> > I'm wondering if anyone can offer some sage advice around an issue I have
> > post-upgrade of a homelab server I maintain.
> > I'm stuck in emergency mode and it looks as though my LVM (device-mapper)
> > mounts aren't available.
>
> Heya, Dean! Sounds fixable, good luck. Can you boot with an "old" kernel?
>
> While it's usually possible to get away with just the installer
> "rescue" mode, I find it's more reassuring to reach for a different
> "live boot" at this stage and interactively use https://www.finnix.org/
> (small, Debian "testing" based, and also great for a headless server)
> or https://www.debian.org/CD/live/ . Either the latest, or occasionally
> going back to a version that more closely matches your current system.
>
> Boot it, let it scan for what's available, then interactively experiment
> to find exactly what you need to get your system mounted and self-bootable.
>
> > Before upgrading, I did have crypto-mount (or something) enabled, which
> > produced errors but did not prevent boot as I never encrypted the
> > underlying raid block device md0 (it was always planned but never
> > implemented, so it was skipped when building the server and I never
> > disabled the crypto boot service).
>
> It sounds like that specifically was "just" a harmless warning? We need
> to find the errors in console output, or `dmesg`, or what's totally
> missing that "ought" to be there, like a loaded working driver for your
> storage hardware (`lspci`/`lshw` ?).
>
> Maybe those devices were missing because of an `mdadm` error that
> scrolled past unnoticed, or a missing driver module in the `initramfs`,
> and it might be easier to see from a running live USB what was missing.
>
> ( If the LVs are missing (`lvs`/`vgs`/`pvs`), the md0
> has probably not been assembled. Maybe that failed because the devices
> under md0 were missing, so that needs to be fixed before `vgchange -ay`
> will help. )
>
> There may be some known relevant issues that got written down in the
> "Upgrade" chapter 4 or "Issues to be aware of" chapter 5 of:
> https://www.debian.org/releases/oldstable/amd64/release-notes/
> https://www.debian.org/releases/stable/amd64/release-notes/
>
> [...]
> > I'm seeking some advice to try to quickly fix this before I consider
> > rebuilding. I do have a copy of /etc before upgrading FWIW.
>
> Good, could be handy to compare or know what devices or filesystem UUIDs
> to look for.
>
> Nick.
>
> --
>    Nick Bannon   | "I made this letter longer than usual because
> nick-sig at rcpt.to | I lack the time to make it shorter." - Pascal
> _______________________________________________
> PLUG discussion list: plug at plug.org.au
> http://lists.plug.org.au/mailman/listinfo/plug
> Committee e-mail: committee at plug.org.au
> PLUG Membership: http://www.plug.org.au/membership
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.plug.org.au/pipermail/plug/attachments/20230628/e0657bba/attachment.html>