[plug] Looking for assistance with a recent Debian upgrade
Joe Aquilina
joe at chem.com.au
Wed Dec 18 14:22:05 AWST 2019
Looks like I could have a busy weekend, as I think I will do a dd clone
of the system first before I upgrade the kernel. Better learn my lessons
from my first effort, and ensure that I can get the system back to
working order in case I stuff things up.
We are not running anything too exotic on that box, it is pretty much
all from debian packages so I hope there will be no further (major)
calamities to deal with.
So for now I will stop pestering people here (and elsewhere) and prepare
for and plan my weekend of fun!?
Cheers.
Joe Aquilina
On 18/12/19 1:52 pm, Chris Hoy Poy wrote:
> Tbh, it'll probably just work , depending on how much unusual /
> unpackaged stuff you have currently installed. But as you can see,
> stuff is gonna fail anyway as 686 stuff falls off the back of the
> adequate testing truck :-(
>
> If it's just running a pretty standard set of debian packaged
> samba/nfs/LAMP etc, upgrading the kernel to amd64 probably just works,
> and most stuff will continue functioning as a 686 binary , as that is
> perfectly fine on a 64bit kernel , except for the usual caveats (that
> you have already now) where individual processes can't access all your
> available memory, but the kernel can put stuff everywhere so you can
> still use your ram.
>
> But I don't know your system, so obviously can't guarantee, and there
> is some small element of risk etc etc.
>
> I've done before, and mostly had good results, with the occasional
> smattering of "oh dear, guess I'm reinstalling and getting out those
> backups now". But it's been a while since I've had to do it! (A
> decade+? Sigh)
>
> I would go the 4.19 686 pae route for now, and hope that worked for
> long enough to put together a plan for a rebuild, ideally on the next
> hardware refresh, as that's a nice boundary. If it doesn't work, good
> thing its this time of year, as most people are pretty happy when
> stuff fails this time of year, and they get to go home :-) (except the
> unlucky person rebuilding the server and testing it)
>
>
>
> /Chris
>
> On Wed, 18 Dec 2019, 1:31 pm Joe Aquilina, <joe at chem.com.au
> <mailto:joe at chem.com.au>> wrote:
>
> You are right, it is a problem. What I wouldn't give to have
> someone much more experienced than me come in and sit with me for
> a week or two (or however long it took) so that I could
> significantly improve my knowledge and skills. MY/our finances
> really don't allow for that at present, business has been a little
> slower than usual this past year.
>
> The system was set up many years ago by someone who is vastly more
> knowledgeable and experienced than me, the bosses son who is a
> former PLUG member and now lives in Melbourne. I am just the
> "lucky" bunny who gets to try to keep it running, which has been
> pretty much successful until now.
>
> I have rebuilt the hardware but when I did, I simply moved the
> hard drives across and didn't cross-grade from i386 to amd64. We
> have hard drive failures along the way but have been able to
> overcome those by swapping in new drives into the raid array.
>
> So, for now, should I upgrade the kernel and see how that goes, or
> get really adventurous and cross-grade to amd64 and upgrade the
> kernel? How difficult is an amd64 cross-grade - that is not
> something I have ever done. In any case, I am thinking that I may
> not do any of this until the weekend when I have time to do a dd
> clone of the system first before I potentially make things worse.
>
> Cheers.
>
> Joe Aquilina
>
> On 18/12/19 1:19 pm, Benjamin wrote:
>> Being unable to recreate your setup from scratch is... well, a
>> problem.
>>
>> It's worth investing in something like Ansible or Puppet, using
>> it to automate creating complicated setups - that way you're not
>> hosed if your hard drive dies and you have to do all this stuff
>> by hand anyway...
>>
>> On Wed, Dec 18, 2019 at 1:14 PM Joe Aquilina <joe at chem.com.au
>> <mailto:joe at chem.com.au>> wrote:
>>
>> I just did an apt-cache search it shows me this:
>>
>> linux-headers-4.19.0-6-686 - Header files for Linux 4.19.0-6-686
>> linux-headers-4.19.0-6-686-pae - Header files for Linux
>> 4.19.0-6-686-pae
>> linux-headers-4.19.0-6-rt-686-pae - Header files for Linux
>> 4.19.0-6-rt-686-pae
>> linux-image-4.19.0-6-686-dbg - Debug symbols for
>> linux-image-4.19.0-6-686
>> linux-image-4.19.0-6-686-pae-dbg - Debug symbols for
>> linux-image-4.19.0-6-686-pae
>> linux-image-4.19.0-6-686-pae-unsigned - Linux 4.19 for modern PCs
>> linux-image-4.19.0-6-686-unsigned - Linux 4.19 for older PCs
>> linux-image-4.19.0-6-rt-686-pae-dbg - Debug symbols for
>> linux-image-4.19.0-6-rt-686-pae
>> linux-image-4.19.0-6-rt-686-pae-unsigned - Linux 4.19 for
>> modern PCs, PREEMPT_RT
>> linux-image-i386-signed-template - Template for signed
>> linux-image packages for i386
>> linux-image-4.19.0-6-686 - Linux 4.19 for older PCs (signed)
>> linux-image-4.19.0-6-686-pae - Linux 4.19 for modern PCs (signed)
>> linux-image-4.19.0-6-rt-686-pae - Linux 4.19 for modern PCs,
>> PREEMPT_RT (signed)
>> linux-image-686 - Linux for older PCs (meta-package)
>> linux-image-686-dbg - Debugging symbols for Linux 686
>> configuration (meta-package)
>> linux-image-686-pae - Linux for modern PCs (meta-package)
>> linux-image-686-pae-dbg - Debugging symbols for Linux 686-pae
>> configuration (meta-package)
>> linux-image-rt-686-pae - Linux for modern PCs (meta-package),
>> PREEMPT_RT
>> linux-image-rt-686-pae-dbg - Debugging symbols for Linux
>> rt-686-pae configuration (meta-package)
>> linux-image-3.16.0-4-686-pae - Linux 3.16 for modern PCs
>>
>> Is that not showing me that there is a 4.19 PAE branch for
>> buster? Or am I misinterpreting that output?
>>
>> I have been reluctant to jump to amd64 on this system because
>> it is a rather complicated setup, which I am not confident
>> that I could recreate from scratch if the worst happened. But
>> as you say, perhaps it is time to do it anyway.
>>
>> Cheers.
>>
>> Joe Aquilina
>>
>>
>> On 18/12/19 12:55 pm, Chris Hoy Poy wrote:
>>> Ahh you are using the PAE branch , which doesn't have a
>>> later kernel in Buster
>>>
>>> Time to make the jump to amd64 !
>>>
>>> /Chris
>>>
>>>
>>>
>>> On Wed, 18 Dec 2019, 12:52 pm Chris Hoy Poy,
>>> <chris at hoypoy.id.au <mailto:chris at hoypoy.id.au>> wrote:
>>>
>>> Given that other users have reported similiar issues
>>> with that exact kernel coupled with updated openssl +
>>> openssh, you want to update that kernel to something a
>>> bit more recent.
>>>
>>> Should be a straight forward apt-get install
>>> <linux-image> from memory, as suggested here :
>>>
>>> https://wiki.debian.org/HowToUpgradeKernel
>>>
>>> It's a pretty safe process these days, though you are
>>> making some big jumps (3.16 to 4.19.x (Buster latest)) -
>>> so have some get out of jail cards handy (backups,
>>> console access, coffee, etc)
>>>
>>>
>>> If it was just recently upgraded to buster, you
>>> shouldn't have any issues on latest kernel(s) Being on
>>> 686 as opposed to amd64 (pretty much the default these
>>> days, and I guarantee amd64 gets better testing with
>>> stuff then 686 ! ). I wouldn't mangle that unless you
>>> feel like a reinstall tho, it should be fine for 99% of
>>> use cases.
>>>
>>> Enjoy
>>> /Chris
>>>
>>>
>>> On Wed, 18 Dec 2019, 12:41 pm Joe Aquilina,
>>> <joe at chem.com.au <mailto:joe at chem.com.au>> wrote:
>>>
>>> I think that is a default sshd_config. I have tried
>>> removing (and later purging) it recently and that is
>>> pretty much as it was after the latest reinstall.
>>>
>>> The kernel is an older one, which surprises me. It
>>> doesn't seem to have been updated as part of the
>>> upgrade from stretch to buster, which I was
>>> expecting to have happened. The kernel is still
>>> 3.16.0-4-686-pae.
>>>
>>> I have never updated a kernel, is there a link to a
>>> procedure for this? I have found one that suggests
>>> using ukuu, but I have not been able to install
>>> that, there seems to be a problem with the repository.
>>>
>>> Cheers.
>>>
>>> Joe Aquilina
>>>
>>>
>>> On 18/12/19 12:19 pm, Chris Hoy Poy wrote:
>>>> That line shouldn't bother it (the nologin is fine,
>>>> you don't want it logging in)
>>>>
>>>> I can't see "usePrivilegeSeparation" in that
>>>> config, it's probably default.
>>>>
>>>> How old is the overall install, and has the kernel
>>>> been upgraded recently?
>>>>
>>>> I see a number of recent minor issues around
>>>> openssl versions + kernel versions
>>>>
>>>> Probably want to be a later kernel if possible,
>>>> just to be sure.
>>>>
>>>> https://www.mail-archive.com/debian-ssh@lists.debian.org/msg08820.html
>>>>
>>>> https://www.mail-archive.com/debian-ssh@lists.debian.org/msg08852.html
>>>>
>>>>
>>>> On Wed, 18 Dec 2019, 12:05 pm Joe Aquilina,
>>>> <joe at chem.com.au <mailto:joe at chem.com.au>> wrote:
>>>>
>>>> Chris
>>>>
>>>> Her is the sshd_config file on the server:
>>>>
>>>> $ cat /etc/ssh/sshd_config
>>>> # $OpenBSD: sshd_config,v 1.103
>>>> 2018/04/09 20:41:22 tj Exp $
>>>>
>>>> # This is the sshd server system-wide
>>>> configuration file. See
>>>> # sshd_config(5) for more information.
>>>>
>>>> # This sshd was compiled with
>>>> PATH=/usr/bin:/bin:/usr/sbin:/sbin
>>>>
>>>> # The strategy used for options in the default
>>>> sshd_config shipped with
>>>> # OpenSSH is to specify options with their
>>>> default value where
>>>> # possible, but leave them commented.
>>>> Uncommented options override the
>>>> # default value.
>>>>
>>>> Port 22
>>>> #AddressFamily any
>>>> #ListenAddress 0.0.0.0
>>>> #ListenAddress ::
>>>>
>>>> #HostKey /etc/ssh/ssh_host_rsa_key
>>>> #HostKey /etc/ssh/ssh_host_ecdsa_key
>>>> #HostKey /etc/ssh/ssh_host_ed25519_key
>>>>
>>>> # Ciphers and keying
>>>> #RekeyLimit default none
>>>>
>>>> # Logging
>>>> #SyslogFacility AUTH
>>>> #LogLevel INFO
>>>>
>>>> # Authentication:
>>>>
>>>> #LoginGraceTime 2m
>>>> #PermitRootLogin prohibit-password
>>>> AllowUsers joe
>>>> #StrictModes yes
>>>> #MaxAuthTries 6
>>>> #MaxSessions 10
>>>>
>>>> #PubkeyAuthentication yes
>>>>
>>>> # Expect .ssh/authorized_keys2 to be
>>>> disregarded by default in future.
>>>> #AuthorizedKeysFile .ssh/authorized_keys
>>>> .ssh/authorized_keys2
>>>>
>>>> #AuthorizedPrincipalsFile none
>>>>
>>>> #AuthorizedKeysCommand none
>>>> #AuthorizedKeysCommandUser nobody
>>>>
>>>> # For this to work you will also need host keys
>>>> in /etc/ssh/ssh_known_hosts
>>>> #HostbasedAuthentication no
>>>> # Change to yes if you don't trust
>>>> ~/.ssh/known_hosts for
>>>> # HostbasedAuthentication
>>>> #IgnoreUserKnownHosts no
>>>> # Don't read the user's ~/.rhosts and ~/.shosts
>>>> files
>>>> #IgnoreRhosts yes
>>>>
>>>> # To disable tunneled clear text passwords,
>>>> change to no here!
>>>> #PasswordAuthentication yes
>>>> #PermitEmptyPasswords no
>>>>
>>>> # Change to yes to enable challenge-response
>>>> passwords (beware issues with
>>>> # some PAM modules and threads)
>>>> ChallengeResponseAuthentication no
>>>>
>>>> # Kerberos options
>>>> #KerberosAuthentication no
>>>> #KerberosOrLocalPasswd yes
>>>> #KerberosTicketCleanup yes
>>>> #KerberosGetAFSToken no
>>>>
>>>> # GSSAPI options
>>>> #GSSAPIAuthentication no
>>>> #GSSAPICleanupCredentials yes
>>>> #GSSAPIStrictAcceptorCheck yes
>>>> #GSSAPIKeyExchange no
>>>>
>>>> # Set this to 'yes' to enable PAM
>>>> authentication, account processing,
>>>> # and session processing. If this is enabled,
>>>> PAM authentication will
>>>> # be allowed through the
>>>> ChallengeResponseAuthentication and
>>>> # PasswordAuthentication. Depending on your PAM
>>>> configuration,
>>>> # PAM authentication via
>>>> ChallengeResponseAuthentication may bypass
>>>> # the setting of "PermitRootLogin
>>>> without-password".
>>>> # If you just want the PAM account and session
>>>> checks to run without
>>>> # PAM authentication, then enable this but set
>>>> PasswordAuthentication
>>>> # and ChallengeResponseAuthentication to 'no'.
>>>> UsePAM yes
>>>> UseLogin no
>>>>
>>>> #AllowAgentForwarding yes
>>>> #AllowTcpForwarding yes
>>>> #GatewayPorts no
>>>> X11Forwarding yes
>>>> #X11DisplayOffset 10
>>>> #X11UseLocalhost yes
>>>> #PermitTTY yes
>>>> PrintMotd no
>>>> #PrintLastLog yes
>>>> #TCPKeepAlive yes
>>>> #PermitUserEnvironment no
>>>> #Compression delayed
>>>> #ClientAliveInterval 0
>>>> #ClientAliveCountMax 3
>>>> #UseDNS no
>>>> #PidFile /var/run/sshd.pid
>>>> #MaxStartups 10:30:100
>>>> #PermitTunnel no
>>>> #ChrootDirectory none
>>>> #VersionAddendum none
>>>>
>>>> # no default banner path
>>>> #Banner none
>>>>
>>>> # Allow client to pass locale environment variables
>>>> AcceptEnv LANG LC_*
>>>>
>>>> # override default of no subsystems
>>>> Subsystem sftp /usr/lib/openssh/sftp-server
>>>>
>>>> # Example of overriding settings on a per-user
>>>> basis
>>>> #Match User anoncvs
>>>> # X11Forwarding no
>>>> # AllowTcpForwarding no
>>>> # PermitTTY no
>>>> # ForceCommand cvs server
>>>>
>>>> I just checked the passwd file on the server
>>>> and both accounts I use to login finish with
>>>> /bin/bash. However, I also noticed that the
>>>> last line of the passwd file looks like this:
>>>>
>>>> sshd:x:100:65534::/run/sshd:/usr/sbin/nologin
>>>>
>>>> Looking at the passwd file from a backup done
>>>> before the upgrade, and when ssh logins were
>>>> working, this line is a recent addition - it
>>>> does not appear in past instances of the passwd
>>>> file. Is this the cause of my problems? Can I
>>>> simply delete this line and try again?
>>>>
>>>> Cheers.
>>>>
>>>> Joe Aquilina
>>>>
>>>>
>>>> On 18/12/19 11:49 am, Chris Hoy Poy wrote:
>>>>> Hey Joe,
>>>>>
>>>>> Can you check what "usePrivilegeSeparation" is
>>>>> defined as in the server sshd_config is ?
>>>>>
>>>>> Cheers
>>>>> /Chris
>>>>>
>>>>> On Wed, 18 Dec 2019, 11:42 am Joe Aquilina,
>>>>> <joe at chem.com.au <mailto:joe at chem.com.au>> wrote:
>>>>>
>>>>> sestatus and getenforce both show selinux
>>>>> as disabled.
>>>>>
>>>>> There is already another account that is
>>>>> occasionally used to login to the server -
>>>>> it fails exactly the same as my (joe)
>>>>> account. I don't believe that any scripts
>>>>> at login.
>>>>>
>>>>> And yes I did edit the output to protect
>>>>> the "guilty" ... replaced the real server
>>>>> name with <server> and the server's IP
>>>>> address. I presumed that is what was
>>>>> requested when it was suggested that I
>>>>> post a sanitised copy of the login attempt
>>>>> output.
>>>>>
>>>>> Cheers.
>>>>>
>>>>> Joe Aquilina
>>>>>
>>>>> On 18/12/19 11:08 am, mike wrote:
>>>>>> On 18/12/2019 10:43, Joe Aquilina wrote:
>>>>>>> I have no idea about selinux, whether it
>>>>>>> is installed/enabled. How do I check
>>>>>>> that and disable it if necessary, and
>>>>>>> then re-enable?
>>>>>>
>>>>>> sestatus orgetenforce
>>>>>>
>>>>>> If file not found then not in use.
>>>>>>
>>>>>> Are you removing details from the output? IE:
>>>>>> Authenticated to <server> ([ip.address of server]:22).
>>>>>>
>>>>>> Mine says
>>>>>> debug1: Authentication succeeded (publickey).
>>>>>> Authenticated to nos ([10.222.0.4]:22).
>>>>>>
>>>>>> Another thought is what does the passwd file say for your login? I have /bin/bash on the end
>>>>>>
>>>>>> What user are you trying to login as?
>>>>>>
>>>>>> Are you running any scripts at login that may be failing?
>>>>>>
>>>>>> Have you tried another user?
>>>>>>
>>>>>> Maybe create a new user and try logging in with that just to remove the user as being an issue.
>>>>>>
>>>>>> --
>>>>>> 'ooroo
>>>>>>
>>>>>> Mike...(:)-)
>>>>>> ---------------------------------------------------
>>>>>> Email:mike at wolf-rock.com <mailto:mike at wolf-rock.com> o
>>>>>> You need only two tools. o /////
>>>>>> A hammer and duct tape. If it /@ `\ /) ~
>>>>>> doesn't move and it should use > (O) X< ~ Fish!!
>>>>>> the hammer. If it moves and `\___/' \) ~
>>>>>> shouldn't, use the tape. \\\
>>>>>> ---------------------------------------------------
>>>>>
>>>>>
>>>>> --
>>>>> Joe Aquilina
>>>>> Central Chemical Consulting Pty Ltd
>>>>> PO Box 2546 Malaga WA 6944 Australia
>>>>> 1/11 Narloo St Malaga 6090 Australia
>>>>> Tel: +61 8 9248 2739 Fax: +61 8 9248 2749
>>>>> joe at chem.com.au <mailto:joe at chem.com.au> www.chem.com.au <http://www.chem.com.au>
>>>>>
>>>>> _______________________________________________
>>>>> PLUG discussion list: plug at plug.org.au
>>>>> <mailto:plug at plug.org.au>
>>>>> http://lists.plug.org.au/mailman/listinfo/plug
>>>>> Committee e-mail: committee at plug.org.au
>>>>> <mailto:committee at plug.org.au>
>>>>> PLUG Membership:
>>>>> http://www.plug.org.au/membership
>>>>>
>>>>
>>>> --
>>>> Joe Aquilina
>>>> Central Chemical Consulting Pty Ltd
>>>> PO Box 2546 Malaga WA 6944 Australia
>>>> 1/11 Narloo St Malaga 6090 Australia
>>>> Tel: +61 8 9248 2739 Fax: +61 8 9248 2749
>>>> joe at chem.com.au <mailto:joe at chem.com.au> www.chem.com.au <http://www.chem.com.au>
>>>>
>>>> _______________________________________________
>>>> PLUG discussion list: plug at plug.org.au
>>>> <mailto:plug at plug.org.au>
>>>> http://lists.plug.org.au/mailman/listinfo/plug
>>>> Committee e-mail: committee at plug.org.au
>>>> <mailto:committee at plug.org.au>
>>>> PLUG Membership: http://www.plug.org.au/membership
>>>>
>>>
>>> --
>>> Joe Aquilina
>>> Central Chemical Consulting Pty Ltd
>>> PO Box 2546 Malaga WA 6944 Australia
>>> 1/11 Narloo St Malaga 6090 Australia
>>> Tel: +61 8 9248 2739 Fax: +61 8 9248 2749
>>> joe at chem.com.au <mailto:joe at chem.com.au> www.chem.com.au <http://www.chem.com.au>
>>>
>>> _______________________________________________
>>> PLUG discussion list: plug at plug.org.au
>>> <mailto:plug at plug.org.au>
>>> http://lists.plug.org.au/mailman/listinfo/plug
>>> Committee e-mail: committee at plug.org.au
>>> <mailto:committee at plug.org.au>
>>> PLUG Membership: http://www.plug.org.au/membership
>>>
>>
>> --
>> Joe Aquilina
>> Central Chemical Consulting Pty Ltd
>> PO Box 2546 Malaga WA 6944 Australia
>> 1/11 Narloo St Malaga 6090 Australia
>> Tel: +61 8 9248 2739 Fax: +61 8 9248 2749
>> joe at chem.com.au <mailto:joe at chem.com.au> www.chem.com.au <http://www.chem.com.au>
>>
>> _______________________________________________
>> PLUG discussion list: plug at plug.org.au <mailto:plug at plug.org.au>
>> http://lists.plug.org.au/mailman/listinfo/plug
>> Committee e-mail: committee at plug.org.au
>> <mailto:committee at plug.org.au>
>> PLUG Membership: http://www.plug.org.au/membership
>>
>
> --
> Joe Aquilina
> Central Chemical Consulting Pty Ltd
> PO Box 2546 Malaga WA 6944 Australia
> 1/11 Narloo St Malaga 6090 Australia
> Tel: +61 8 9248 2739 Fax: +61 8 9248 2749
> joe at chem.com.au <mailto:joe at chem.com.au> www.chem.com.au <http://www.chem.com.au>
>
> _______________________________________________
> PLUG discussion list: plug at plug.org.au <mailto:plug at plug.org.au>
> http://lists.plug.org.au/mailman/listinfo/plug
> Committee e-mail: committee at plug.org.au <mailto:committee at plug.org.au>
> PLUG Membership: http://www.plug.org.au/membership
>
--
Joe Aquilina
Central Chemical Consulting Pty Ltd
PO Box 2546 Malaga WA 6944 Australia
1/11 Narloo St Malaga 6090 Australia
Tel: +61 8 9248 2739 Fax: +61 8 9248 2749
joe at chem.com.au www.chem.com.au
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.plug.org.au/pipermail/plug/attachments/20191218/ab8aa7e1/attachment.html>
More information about the plug
mailing list