[plug] Looking for assistance with a recent Debian upgrade
Chris Hoy Poy
chris at hoypoy.id.au
Wed Dec 18 13:52:10 AWST 2019
Tbh, it'll probably just work , depending on how much unusual / unpackaged
stuff you have currently installed. But as you can see, stuff is gonna fail
anyway as 686 stuff falls off the back of the adequate testing truck :-(
If it's just running a pretty standard set of debian packaged
samba/nfs/LAMP etc, upgrading the kernel to amd64 probably just works, and
most stuff will continue functioning as a 686 binary , as that is perfectly
fine on a 64bit kernel , except for the usual caveats (that you have
already now) where individual processes can't access all your available
memory, but the kernel can put stuff everywhere so you can still use your
ram.
But I don't know your system, so obviously can't guarantee, and there is
some small element of risk etc etc.
I've done before, and mostly had good results, with the occasional
smattering of "oh dear, guess I'm reinstalling and getting out those
backups now". But it's been a while since I've had to do it! (A decade+?
Sigh)
I would go the 4.19 686 pae route for now, and hope that worked for long
enough to put together a plan for a rebuild, ideally on the next hardware
refresh, as that's a nice boundary. If it doesn't work, good thing its this
time of year, as most people are pretty happy when stuff fails this time of
year, and they get to go home :-) (except the unlucky person rebuilding the
server and testing it)
/Chris
On Wed, 18 Dec 2019, 1:31 pm Joe Aquilina, <joe at chem.com.au> wrote:
> You are right, it is a problem. What I wouldn't give to have someone much
> more experienced than me come in and sit with me for a week or two (or
> however long it took) so that I could significantly improve my knowledge
> and skills. MY/our finances really don't allow for that at present,
> business has been a little slower than usual this past year.
>
> The system was set up many years ago by someone who is vastly more
> knowledgeable and experienced than me, the bosses son who is a former PLUG
> member and now lives in Melbourne. I am just the "lucky" bunny who gets to
> try to keep it running, which has been pretty much successful until now.
>
> I have rebuilt the hardware but when I did, I simply moved the hard drives
> across and didn't cross-grade from i386 to amd64. We have hard drive
> failures along the way but have been able to overcome those by swapping in
> new drives into the raid array.
>
> So, for now, should I upgrade the kernel and see how that goes, or get
> really adventurous and cross-grade to amd64 and upgrade the kernel? How
> difficult is an amd64 cross-grade - that is not something I have ever done.
> In any case, I am thinking that I may not do any of this until the weekend
> when I have time to do a dd clone of the system first before I potentially
> make things worse.
>
> Cheers.
>
> Joe Aquilina
>
> On 18/12/19 1:19 pm, Benjamin wrote:
>
> Being unable to recreate your setup from scratch is... well, a problem.
>
> It's worth investing in something like Ansible or Puppet, using it to
> automate creating complicated setups - that way you're not hosed if your
> hard drive dies and you have to do all this stuff by hand anyway...
>
> On Wed, Dec 18, 2019 at 1:14 PM Joe Aquilina <joe at chem.com.au> wrote:
>
>> I just did an apt-cache search it shows me this:
>>
>> linux-headers-4.19.0-6-686 - Header files for Linux 4.19.0-6-686
>> linux-headers-4.19.0-6-686-pae - Header files for Linux 4.19.0-6-686-pae
>> linux-headers-4.19.0-6-rt-686-pae - Header files for Linux
>> 4.19.0-6-rt-686-pae
>> linux-image-4.19.0-6-686-dbg - Debug symbols for linux-image-4.19.0-6-686
>> linux-image-4.19.0-6-686-pae-dbg - Debug symbols for
>> linux-image-4.19.0-6-686-pae
>> linux-image-4.19.0-6-686-pae-unsigned - Linux 4.19 for modern PCs
>> linux-image-4.19.0-6-686-unsigned - Linux 4.19 for older PCs
>> linux-image-4.19.0-6-rt-686-pae-dbg - Debug symbols for
>> linux-image-4.19.0-6-rt-686-pae
>> linux-image-4.19.0-6-rt-686-pae-unsigned - Linux 4.19 for modern PCs,
>> PREEMPT_RT
>> linux-image-i386-signed-template - Template for signed linux-image
>> packages for i386
>> linux-image-4.19.0-6-686 - Linux 4.19 for older PCs (signed)
>> linux-image-4.19.0-6-686-pae - Linux 4.19 for modern PCs (signed)
>> linux-image-4.19.0-6-rt-686-pae - Linux 4.19 for modern PCs, PREEMPT_RT
>> (signed)
>> linux-image-686 - Linux for older PCs (meta-package)
>> linux-image-686-dbg - Debugging symbols for Linux 686 configuration
>> (meta-package)
>> linux-image-686-pae - Linux for modern PCs (meta-package)
>> linux-image-686-pae-dbg - Debugging symbols for Linux 686-pae
>> configuration (meta-package)
>> linux-image-rt-686-pae - Linux for modern PCs (meta-package), PREEMPT_RT
>> linux-image-rt-686-pae-dbg - Debugging symbols for Linux rt-686-pae
>> configuration (meta-package)
>> linux-image-3.16.0-4-686-pae - Linux 3.16 for modern PCs
>>
>> Is that not showing me that there is a 4.19 PAE branch for buster? Or am
>> I misinterpreting that output?
>>
>> I have been reluctant to jump to amd64 on this system because it is a
>> rather complicated setup, which I am not confident that I could recreate
>> from scratch if the worst happened. But as you say, perhaps it is time to
>> do it anyway.
>>
>> Cheers.
>>
>> Joe Aquilina
>>
>>
>> On 18/12/19 12:55 pm, Chris Hoy Poy wrote:
>>
>> Ahh you are using the PAE branch , which doesn't have a later kernel in
>> Buster
>>
>> Time to make the jump to amd64 !
>>
>> /Chris
>>
>>
>>
>> On Wed, 18 Dec 2019, 12:52 pm Chris Hoy Poy, <chris at hoypoy.id.au> wrote:
>>
>>> Given that other users have reported similiar issues with that exact
>>> kernel coupled with updated openssl + openssh, you want to update that
>>> kernel to something a bit more recent.
>>>
>>> Should be a straight forward apt-get install <linux-image> from memory,
>>> as suggested here :
>>>
>>> https://wiki.debian.org/HowToUpgradeKernel
>>>
>>> It's a pretty safe process these days, though you are making some big
>>> jumps (3.16 to 4.19.x (Buster latest)) - so have some get out of jail cards
>>> handy (backups, console access, coffee, etc)
>>>
>>>
>>> If it was just recently upgraded to buster, you shouldn't have any
>>> issues on latest kernel(s) Being on 686 as opposed to amd64 (pretty much
>>> the default these days, and I guarantee amd64 gets better testing with
>>> stuff then 686 ! ). I wouldn't mangle that unless you feel like a reinstall
>>> tho, it should be fine for 99% of use cases.
>>>
>>> Enjoy
>>> /Chris
>>>
>>>
>>> On Wed, 18 Dec 2019, 12:41 pm Joe Aquilina, <joe at chem.com.au> wrote:
>>>
>>>> I think that is a default sshd_config. I have tried removing (and later
>>>> purging) it recently and that is pretty much as it was after the latest
>>>> reinstall.
>>>>
>>>> The kernel is an older one, which surprises me. It doesn't seem to have
>>>> been updated as part of the upgrade from stretch to buster, which I was
>>>> expecting to have happened. The kernel is still 3.16.0-4-686-pae.
>>>>
>>>> I have never updated a kernel, is there a link to a procedure for this?
>>>> I have found one that suggests using ukuu, but I have not been able to
>>>> install that, there seems to be a problem with the repository.
>>>>
>>>> Cheers.
>>>>
>>>> Joe Aquilina
>>>>
>>>>
>>>> On 18/12/19 12:19 pm, Chris Hoy Poy wrote:
>>>>
>>>> That line shouldn't bother it (the nologin is fine, you don't want it
>>>> logging in)
>>>>
>>>> I can't see "usePrivilegeSeparation" in that config, it's probably
>>>> default.
>>>>
>>>> How old is the overall install, and has the kernel been upgraded
>>>> recently?
>>>>
>>>> I see a number of recent minor issues around openssl versions + kernel
>>>> versions
>>>>
>>>> Probably want to be a later kernel if possible, just to be sure.
>>>>
>>>> https://www.mail-archive.com/debian-ssh@lists.debian.org/msg08820.html
>>>>
>>>> https://www.mail-archive.com/debian-ssh@lists.debian.org/msg08852.html
>>>>
>>>>
>>>> On Wed, 18 Dec 2019, 12:05 pm Joe Aquilina, <joe at chem.com.au> wrote:
>>>>
>>>>> Chris
>>>>>
>>>>> Her is the sshd_config file on the server:
>>>>>
>>>>> $ cat /etc/ssh/sshd_config
>>>>> # $OpenBSD: sshd_config,v 1.103 2018/04/09 20:41:22 tj Exp $
>>>>>
>>>>> # This is the sshd server system-wide configuration file. See
>>>>> # sshd_config(5) for more information.
>>>>>
>>>>> # This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin
>>>>>
>>>>> # The strategy used for options in the default sshd_config shipped with
>>>>> # OpenSSH is to specify options with their default value where
>>>>> # possible, but leave them commented. Uncommented options override the
>>>>> # default value.
>>>>>
>>>>> Port 22
>>>>> #AddressFamily any
>>>>> #ListenAddress 0.0.0.0
>>>>> #ListenAddress ::
>>>>>
>>>>> #HostKey /etc/ssh/ssh_host_rsa_key
>>>>> #HostKey /etc/ssh/ssh_host_ecdsa_key
>>>>> #HostKey /etc/ssh/ssh_host_ed25519_key
>>>>>
>>>>> # Ciphers and keying
>>>>> #RekeyLimit default none
>>>>>
>>>>> # Logging
>>>>> #SyslogFacility AUTH
>>>>> #LogLevel INFO
>>>>>
>>>>> # Authentication:
>>>>>
>>>>> #LoginGraceTime 2m
>>>>> #PermitRootLogin prohibit-password
>>>>> AllowUsers joe
>>>>> #StrictModes yes
>>>>> #MaxAuthTries 6
>>>>> #MaxSessions 10
>>>>>
>>>>> #PubkeyAuthentication yes
>>>>>
>>>>> # Expect .ssh/authorized_keys2 to be disregarded by default in future.
>>>>> #AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2
>>>>>
>>>>> #AuthorizedPrincipalsFile none
>>>>>
>>>>> #AuthorizedKeysCommand none
>>>>> #AuthorizedKeysCommandUser nobody
>>>>>
>>>>> # For this to work you will also need host keys in
>>>>> /etc/ssh/ssh_known_hosts
>>>>> #HostbasedAuthentication no
>>>>> # Change to yes if you don't trust ~/.ssh/known_hosts for
>>>>> # HostbasedAuthentication
>>>>> #IgnoreUserKnownHosts no
>>>>> # Don't read the user's ~/.rhosts and ~/.shosts files
>>>>> #IgnoreRhosts yes
>>>>>
>>>>> # To disable tunneled clear text passwords, change to no here!
>>>>> #PasswordAuthentication yes
>>>>> #PermitEmptyPasswords no
>>>>>
>>>>> # Change to yes to enable challenge-response passwords (beware issues
>>>>> with
>>>>> # some PAM modules and threads)
>>>>> ChallengeResponseAuthentication no
>>>>>
>>>>> # Kerberos options
>>>>> #KerberosAuthentication no
>>>>> #KerberosOrLocalPasswd yes
>>>>> #KerberosTicketCleanup yes
>>>>> #KerberosGetAFSToken no
>>>>>
>>>>> # GSSAPI options
>>>>> #GSSAPIAuthentication no
>>>>> #GSSAPICleanupCredentials yes
>>>>> #GSSAPIStrictAcceptorCheck yes
>>>>> #GSSAPIKeyExchange no
>>>>>
>>>>> # Set this to 'yes' to enable PAM authentication, account processing,
>>>>> # and session processing. If this is enabled, PAM authentication will
>>>>> # be allowed through the ChallengeResponseAuthentication and
>>>>> # PasswordAuthentication. Depending on your PAM configuration,
>>>>> # PAM authentication via ChallengeResponseAuthentication may bypass
>>>>> # the setting of "PermitRootLogin without-password".
>>>>> # If you just want the PAM account and session checks to run without
>>>>> # PAM authentication, then enable this but set PasswordAuthentication
>>>>> # and ChallengeResponseAuthentication to 'no'.
>>>>> UsePAM yes
>>>>> UseLogin no
>>>>>
>>>>> #AllowAgentForwarding yes
>>>>> #AllowTcpForwarding yes
>>>>> #GatewayPorts no
>>>>> X11Forwarding yes
>>>>> #X11DisplayOffset 10
>>>>> #X11UseLocalhost yes
>>>>> #PermitTTY yes
>>>>> PrintMotd no
>>>>> #PrintLastLog yes
>>>>> #TCPKeepAlive yes
>>>>> #PermitUserEnvironment no
>>>>> #Compression delayed
>>>>> #ClientAliveInterval 0
>>>>> #ClientAliveCountMax 3
>>>>> #UseDNS no
>>>>> #PidFile /var/run/sshd.pid
>>>>> #MaxStartups 10:30:100
>>>>> #PermitTunnel no
>>>>> #ChrootDirectory none
>>>>> #VersionAddendum none
>>>>>
>>>>> # no default banner path
>>>>> #Banner none
>>>>>
>>>>> # Allow client to pass locale environment variables
>>>>> AcceptEnv LANG LC_*
>>>>>
>>>>> # override default of no subsystems
>>>>> Subsystem sftp /usr/lib/openssh/sftp-server
>>>>>
>>>>> # Example of overriding settings on a per-user basis
>>>>> #Match User anoncvs
>>>>> # X11Forwarding no
>>>>> # AllowTcpForwarding no
>>>>> # PermitTTY no
>>>>> # ForceCommand cvs server
>>>>>
>>>>> I just checked the passwd file on the server and both accounts I use
>>>>> to login finish with /bin/bash. However, I also noticed that the last line
>>>>> of the passwd file looks like this:
>>>>>
>>>>> sshd:x:100:65534::/run/sshd:/usr/sbin/nologin
>>>>>
>>>>> Looking at the passwd file from a backup done before the upgrade, and
>>>>> when ssh logins were working, this line is a recent addition - it does not
>>>>> appear in past instances of the passwd file. Is this the cause of my
>>>>> problems? Can I simply delete this line and try again?
>>>>>
>>>>> Cheers.
>>>>>
>>>>> Joe Aquilina
>>>>>
>>>>>
>>>>> On 18/12/19 11:49 am, Chris Hoy Poy wrote:
>>>>>
>>>>> Hey Joe,
>>>>>
>>>>> Can you check what "usePrivilegeSeparation" is defined as in the
>>>>> server sshd_config is ?
>>>>>
>>>>> Cheers
>>>>> /Chris
>>>>>
>>>>> On Wed, 18 Dec 2019, 11:42 am Joe Aquilina, <joe at chem.com.au> wrote:
>>>>>
>>>>>> sestatus and getenforce both show selinux as disabled.
>>>>>>
>>>>>> There is already another account that is occasionally used to login
>>>>>> to the server - it fails exactly the same as my (joe) account. I don't
>>>>>> believe that any scripts at login.
>>>>>>
>>>>>> And yes I did edit the output to protect the "guilty" ... replaced
>>>>>> the real server name with <server> and the server's IP address. I presumed
>>>>>> that is what was requested when it was suggested that I post a sanitised
>>>>>> copy of the login attempt output.
>>>>>>
>>>>>> Cheers.
>>>>>>
>>>>>> Joe Aquilina
>>>>>>
>>>>>> On 18/12/19 11:08 am, mike wrote:
>>>>>>
>>>>>> On 18/12/2019 10:43, Joe Aquilina wrote:
>>>>>>
>>>>>> I have no idea about selinux, whether it is installed/enabled. How do
>>>>>> I check that and disable it if necessary, and then re-enable?
>>>>>>
>>>>>>
>>>>>> sestatus or getenforce
>>>>>>
>>>>>> If file not found then not in use.
>>>>>>
>>>>>> Are you removing details from the output? IE:
>>>>>> Authenticated to <server> ([ip.address of server]:22).
>>>>>>
>>>>>> Mine says
>>>>>> debug1: Authentication succeeded (publickey).
>>>>>> Authenticated to nos ([10.222.0.4]:22).
>>>>>>
>>>>>> Another thought is what does the passwd file say for your login? I have /bin/bash on the end
>>>>>>
>>>>>> What user are you trying to login as?
>>>>>>
>>>>>> Are you running any scripts at login that may be failing?
>>>>>>
>>>>>> Have you tried another user?
>>>>>>
>>>>>> Maybe create a new user and try logging in with that just to remove the user as being an issue.
>>>>>>
>>>>>>
>>>>>> --
>>>>>> 'ooroo
>>>>>>
>>>>>> Mike...(:)-)
>>>>>> ---------------------------------------------------
>>>>>> Email: mike at wolf-rock.com o
>>>>>> You need only two tools. o /////
>>>>>> A hammer and duct tape. If it /@ `\ /) ~
>>>>>> doesn't move and it should use > (O) X< ~ Fish!!
>>>>>> the hammer. If it moves and `\___/' \) ~
>>>>>> shouldn't, use the tape. \\\
>>>>>> ---------------------------------------------------
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Joe Aquilina
>>>>>> Central Chemical Consulting Pty Ltd
>>>>>> PO Box 2546 Malaga WA 6944 Australia
>>>>>> 1/11 Narloo St Malaga 6090 Australia
>>>>>> Tel: +61 8 9248 2739 Fax: +61 8 9248 2749joe at chem.com.au www.chem.com.au
>>>>>>
>>>>>> _______________________________________________
>>>>>> PLUG discussion list: plug at plug.org.au
>>>>>> http://lists.plug.org.au/mailman/listinfo/plug
>>>>>> Committee e-mail: committee at plug.org.au
>>>>>> PLUG Membership: http://www.plug.org.au/membership
>>>>>
>>>>>
>>>>> --
>>>>> Joe Aquilina
>>>>> Central Chemical Consulting Pty Ltd
>>>>> PO Box 2546 Malaga WA 6944 Australia
>>>>> 1/11 Narloo St Malaga 6090 Australia
>>>>> Tel: +61 8 9248 2739 Fax: +61 8 9248 2749joe at chem.com.au www.chem.com.au
>>>>>
>>>>> _______________________________________________
>>>>> PLUG discussion list: plug at plug.org.au
>>>>> http://lists.plug.org.au/mailman/listinfo/plug
>>>>> Committee e-mail: committee at plug.org.au
>>>>> PLUG Membership: http://www.plug.org.au/membership
>>>>
>>>>
>>>> --
>>>> Joe Aquilina
>>>> Central Chemical Consulting Pty Ltd
>>>> PO Box 2546 Malaga WA 6944 Australia
>>>> 1/11 Narloo St Malaga 6090 Australia
>>>> Tel: +61 8 9248 2739 Fax: +61 8 9248 2749joe at chem.com.au www.chem.com.au
>>>>
>>>> _______________________________________________
>>>> PLUG discussion list: plug at plug.org.au
>>>> http://lists.plug.org.au/mailman/listinfo/plug
>>>> Committee e-mail: committee at plug.org.au
>>>> PLUG Membership: http://www.plug.org.au/membership
>>>
>>>
>> --
>> Joe Aquilina
>> Central Chemical Consulting Pty Ltd
>> PO Box 2546 Malaga WA 6944 Australia
>> 1/11 Narloo St Malaga 6090 Australia
>> Tel: +61 8 9248 2739 Fax: +61 8 9248 2749joe at chem.com.au www.chem.com.au
>>
>> _______________________________________________
>> PLUG discussion list: plug at plug.org.au
>> http://lists.plug.org.au/mailman/listinfo/plug
>> Committee e-mail: committee at plug.org.au
>> PLUG Membership: http://www.plug.org.au/membership
>
>
> --
> Joe Aquilina
> Central Chemical Consulting Pty Ltd
> PO Box 2546 Malaga WA 6944 Australia
> 1/11 Narloo St Malaga 6090 Australia
> Tel: +61 8 9248 2739 Fax: +61 8 9248 2749joe at chem.com.au www.chem.com.au
>
> _______________________________________________
> PLUG discussion list: plug at plug.org.au
> http://lists.plug.org.au/mailman/listinfo/plug
> Committee e-mail: committee at plug.org.au
> PLUG Membership: http://www.plug.org.au/membership
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.plug.org.au/pipermail/plug/attachments/20191218/34c230ed/attachment.html>
More information about the plug
mailing list