<div dir="auto">Tbh, it'll probably just work , depending on how much unusual / unpackaged stuff you have currently installed. But as you can see, stuff is gonna fail anyway as 686 stuff falls off the back of the adequate testing truck :-(<div dir="auto"><br></div><div dir="auto">If it's just running a pretty standard set of debian packaged samba/nfs/LAMP etc, upgrading the kernel to amd64 probably just works, and most stuff will continue functioning as a 686 binary , as that is perfectly fine on a 64bit kernel , except for the usual caveats (that you have already now) where individual processes can't access all your available memory, but the kernel can put stuff everywhere so you can still use your ram.</div><div dir="auto"><br></div><div dir="auto">But I don't know your system, so obviously can't guarantee, and there is some small element of risk etc etc. </div><div dir="auto"><br></div><div dir="auto">I've done before, and mostly had good results, with the occasional smattering of "oh dear, guess I'm reinstalling and getting out those backups now". But it's been a while since I've had to do it! (A decade+? Sigh)</div><div dir="auto"><br></div><div dir="auto">I would go the 4.19 686 pae route for now, and hope that worked for long enough to put together a plan for a rebuild, ideally on the next hardware refresh, as that's a nice boundary. If it doesn't work, good thing its this time of year, as most people are pretty happy when stuff fails this time of year, and they get to go home :-) (except the unlucky person rebuilding the server and testing it)</div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto">/Chris</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 18 Dec 2019, 1:31 pm Joe Aquilina, <<a href="mailto:joe@chem.com.au">joe@chem.com.au</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div>You are right, it is a problem. What I
wouldn't give to have someone much more experienced than me come
in and sit with me for a week or two (or however long it took) so
that I could significantly improve my knowledge and skills. MY/our
finances really don't allow for that at present, business has been
a little slower than usual this past year.<br>
</div>
<div><br>
</div>
<div>The system was set up many years ago by
someone who is vastly more knowledgeable and experienced than me,
the bosses son who is a former PLUG member and now lives in
Melbourne. I am just the "lucky" bunny who gets to try to keep it
running, which has been pretty much successful until now.</div>
<div><br>
</div>
<div>I have rebuilt the hardware but when I
did, I simply moved the hard drives across and didn't cross-grade
from i386 to amd64. We have hard drive failures along the way but
have been able to overcome those by swapping in new drives into
the raid array.</div>
<div><br>
</div>
<div>So, for now, should I upgrade the
kernel and see how that goes, or get really adventurous and
cross-grade to amd64 and upgrade the kernel? How difficult is an
amd64 cross-grade - that is not something I have ever done. In any
case, I am thinking that I may not do any of this until the
weekend when I have time to do a dd clone of the system first
before I potentially make things worse.<br>
</div>
<div><br>
</div>
<div>Cheers.</div>
<div><br>
</div>
<div>Joe Aquilina<br>
</div>
<div><br>
</div>
<div>On 18/12/19 1:19 pm, Benjamin wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Being unable to recreate your setup from scratch
is... well, a problem.
<div><br>
</div>
<div>It's worth investing in something like Ansible or Puppet,
using it to automate creating complicated setups - that way
you're not hosed if your hard drive dies and you have to do
all this stuff by hand anyway...</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, Dec 18, 2019 at 1:14
PM Joe Aquilina <<a href="mailto:joe@chem.com.au" target="_blank" rel="noreferrer">joe@chem.com.au</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>I just did an apt-cache search it shows me this:</div>
<div><br>
</div>
<div><tt>linux-headers-4.19.0-6-686 - Header files for Linux
4.19.0-6-686</tt><tt><br>
</tt><tt>linux-headers-4.19.0-6-686-pae - Header files for
Linux 4.19.0-6-686-pae</tt><tt><br>
</tt><tt>linux-headers-4.19.0-6-rt-686-pae - Header files
for Linux 4.19.0-6-rt-686-pae</tt><tt><br>
</tt><tt>linux-image-4.19.0-6-686-dbg - Debug symbols for
linux-image-4.19.0-6-686</tt><tt><br>
</tt><tt>linux-image-4.19.0-6-686-pae-dbg - Debug symbols
for linux-image-4.19.0-6-686-pae</tt><tt><br>
</tt><tt>linux-image-4.19.0-6-686-pae-unsigned - Linux
4.19 for modern PCs</tt><tt><br>
</tt><tt>linux-image-4.19.0-6-686-unsigned - Linux 4.19
for older PCs</tt><tt><br>
</tt><tt>linux-image-4.19.0-6-rt-686-pae-dbg - Debug
symbols for linux-image-4.19.0-6-rt-686-pae</tt><tt><br>
</tt><tt>linux-image-4.19.0-6-rt-686-pae-unsigned - Linux
4.19 for modern PCs, PREEMPT_RT</tt><tt><br>
</tt><tt>linux-image-i386-signed-template - Template for
signed linux-image packages for i386</tt><tt><br>
</tt><tt>linux-image-4.19.0-6-686 - Linux 4.19 for older
PCs (signed)</tt><tt><br>
</tt><tt>linux-image-4.19.0-6-686-pae - Linux 4.19 for
modern PCs (signed)</tt><tt><br>
</tt><tt>linux-image-4.19.0-6-rt-686-pae - Linux 4.19 for
modern PCs, PREEMPT_RT (signed)</tt><tt><br>
</tt><tt>linux-image-686 - Linux for older PCs
(meta-package)</tt><tt><br>
</tt><tt>linux-image-686-dbg - Debugging symbols for Linux
686 configuration (meta-package)</tt><tt><br>
</tt><tt>linux-image-686-pae - Linux for modern PCs
(meta-package)</tt><tt><br>
</tt><tt>linux-image-686-pae-dbg - Debugging symbols for
Linux 686-pae configuration (meta-package)</tt><tt><br>
</tt><tt>linux-image-rt-686-pae - Linux for modern PCs
(meta-package), PREEMPT_RT</tt><tt><br>
</tt><tt>linux-image-rt-686-pae-dbg - Debugging symbols
for Linux rt-686-pae configuration (meta-package)</tt><tt><br>
</tt><tt>linux-image-3.16.0-4-686-pae - Linux 3.16 for
modern PCs</tt></div>
<div><tt><br>
</tt></div>
<div>Is that not showing me that there is a 4.19 PAE branch
for buster? Or am I misinterpreting that output?</div>
<div><br>
</div>
<div>I have been reluctant to jump to amd64 on this system
because it is a rather complicated setup, which I am not
confident that I could recreate from scratch if the worst
happened. But as you say, perhaps it is time to do it
anyway.<br>
</div>
<div><br>
</div>
<div>Cheers.</div>
<div><br>
</div>
<div>Joe Aquilina</div>
<div><br>
</div>
<div><br>
</div>
<div>On 18/12/19 12:55 pm, Chris Hoy Poy wrote:<br>
</div>
<blockquote type="cite">
<div dir="auto">Ahh you are using the PAE branch , which
doesn't have a later kernel in Buster
<div dir="auto"><br>
</div>
<div dir="auto">Time to make the jump to amd64 ! </div>
<div dir="auto"><br>
</div>
<div dir="auto">/Chris</div>
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, 18 Dec 2019,
12:52 pm Chris Hoy Poy, <<a href="mailto:chris@hoypoy.id.au" target="_blank" rel="noreferrer">chris@hoypoy.id.au</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="auto">
<div>Given that other users have reported similiar
issues with that exact kernel coupled with updated
openssl + openssh, you want to update that kernel
to something a bit more recent.
<div dir="auto"><br>
</div>
<div dir="auto">Should be a straight forward
apt-get install <linux-image> from memory,
as suggested here :</div>
<div dir="auto"><br>
</div>
<div dir="auto"><a href="https://wiki.debian.org/HowToUpgradeKernel" rel="noreferrer noreferrer noreferrer" target="_blank">https://wiki.debian.org/HowToUpgradeKernel</a><br>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">It's a pretty safe process these
days, though you are making some big jumps (3.16
to 4.19.x (Buster latest)) - so have some get
out of jail cards handy (backups, console
access, coffee, etc)</div>
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">If it was just recently upgraded
to buster, you shouldn't have any issues on
latest kernel(s) Being on 686 as opposed to
amd64 (pretty much the default these days, and I
guarantee amd64 gets better testing with stuff
then 686 ! ). I wouldn't mangle that unless you
feel like a reinstall tho, it should be fine for
99% of use cases.</div>
<div dir="auto"><br>
</div>
<div dir="auto">Enjoy</div>
<div dir="auto">/Chris</div>
<br>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, 18 Dec
2019, 12:41 pm Joe Aquilina, <<a href="mailto:joe@chem.com.au" rel="noreferrer noreferrer noreferrer" target="_blank">joe@chem.com.au</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>I think that is a default sshd_config.
I have tried removing (and later purging)
it recently and that is pretty much as it
was after the latest reinstall.</div>
<div><br>
</div>
<div>The kernel is an older one, which
surprises me. It doesn't seem to have been
updated as part of the upgrade from
stretch to buster, which I was expecting
to have happened. The kernel is still
3.16.0-4-686-pae.</div>
<div><br>
</div>
<div>I have never updated a kernel, is there
a link to a procedure for this? I have
found one that suggests using ukuu, but I
have not been able to install that, there
seems to be a problem with the repository.</div>
<div><br>
</div>
<div>Cheers.</div>
<div><br>
</div>
<div>Joe Aquilina<br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>On 18/12/19 12:19 pm, Chris Hoy Poy
wrote:<br>
</div>
<blockquote type="cite">
<div dir="auto">That line shouldn't bother
it (the nologin is fine, you don't want
it logging in)
<div dir="auto"><br>
</div>
<div dir="auto">I can't see
"usePrivilegeSeparation" in that
config, it's probably default.</div>
<div dir="auto"><br>
</div>
<div dir="auto">How old is the overall
install, and has the kernel been
upgraded recently?</div>
<div dir="auto"><br>
</div>
<div dir="auto">I see a number of recent
minor issues around openssl versions +
kernel versions </div>
<div dir="auto"><br>
</div>
<div dir="auto">Probably want to be a
later kernel if possible, just to be
sure.</div>
<div dir="auto"><br>
</div>
<div dir="auto"><a href="https://www.mail-archive.com/debian-ssh@lists.debian.org/msg08820.html" rel="noreferrer noreferrer
noreferrer noreferrer noreferrer" target="_blank">https://www.mail-archive.com/debian-ssh@lists.debian.org/msg08820.html</a><br>
</div>
<div dir="auto"><br>
</div>
<div dir="auto"><a href="https://www.mail-archive.com/debian-ssh@lists.debian.org/msg08852.html" rel="noreferrer noreferrer
noreferrer noreferrer" target="_blank">https://www.mail-archive.com/debian-ssh@lists.debian.org/msg08852.html</a><br>
</div>
<div dir="auto"><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On
Wed, 18 Dec 2019, 12:05 pm Joe
Aquilina, <<a href="mailto:joe@chem.com.au" rel="noreferrer noreferrer
noreferrer noreferrer noreferrer
noreferrer noreferrer" target="_blank">joe@chem.com.au</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Chris</div>
<div><br>
</div>
<div>Her is the sshd_config file on
the server:</div>
<div><br>
</div>
<div>$ cat /etc/ssh/sshd_config
<br>
<tt># $OpenBSD:
sshd_config,v 1.103 2018/04/09
20:41:22 tj Exp $</tt><tt><br>
</tt><tt><br>
</tt><tt># This is the sshd server
system-wide configuration file.
See</tt><tt><br>
</tt><tt># sshd_config(5) for more
information.</tt><tt><br>
</tt><tt><br>
</tt><tt># This sshd was compiled
with
PATH=/usr/bin:/bin:/usr/sbin:/sbin</tt><tt><br>
</tt><tt><br>
</tt><tt># The strategy used for
options in the default
sshd_config shipped with</tt><tt><br>
</tt><tt># OpenSSH is to specify
options with their default value
where</tt><tt><br>
</tt><tt># possible, but leave
them commented. Uncommented
options override the</tt><tt><br>
</tt><tt># default value.</tt><tt><br>
</tt><tt><br>
</tt><tt>Port 22</tt><tt><br>
</tt><tt>#AddressFamily any</tt><tt><br>
</tt><tt>#ListenAddress 0.0.0.0</tt><tt><br>
</tt><tt>#ListenAddress ::</tt><tt><br>
</tt><tt><br>
</tt><tt>#HostKey
/etc/ssh/ssh_host_rsa_key</tt><tt><br>
</tt><tt>#HostKey
/etc/ssh/ssh_host_ecdsa_key</tt><tt><br>
</tt><tt>#HostKey
/etc/ssh/ssh_host_ed25519_key</tt><tt><br>
</tt><tt><br>
</tt><tt># Ciphers and keying</tt><tt><br>
</tt><tt>#RekeyLimit default none</tt><tt><br>
</tt><tt><br>
</tt><tt># Logging</tt><tt><br>
</tt><tt>#SyslogFacility AUTH</tt><tt><br>
</tt><tt>#LogLevel INFO</tt><tt><br>
</tt><tt><br>
</tt><tt># Authentication:</tt><tt><br>
</tt><tt><br>
</tt><tt>#LoginGraceTime 2m</tt><tt><br>
</tt><tt>#PermitRootLogin
prohibit-password</tt><tt><br>
</tt><tt>AllowUsers joe</tt><tt><br>
</tt><tt>#StrictModes yes</tt><tt><br>
</tt><tt>#MaxAuthTries 6</tt><tt><br>
</tt><tt>#MaxSessions 10</tt><tt><br>
</tt><tt><br>
</tt><tt>#PubkeyAuthentication yes</tt><tt><br>
</tt><tt><br>
</tt><tt># Expect
.ssh/authorized_keys2 to be
disregarded by default in
future.</tt><tt><br>
</tt><tt>#AuthorizedKeysFile
.ssh/authorized_keys
.ssh/authorized_keys2</tt><tt><br>
</tt><tt><br>
</tt><tt>#AuthorizedPrincipalsFile
none</tt><tt><br>
</tt><tt><br>
</tt><tt>#AuthorizedKeysCommand
none</tt><tt><br>
</tt><tt>#AuthorizedKeysCommandUser
nobody</tt><tt><br>
</tt><tt><br>
</tt><tt># For this to work you
will also need host keys in
/etc/ssh/ssh_known_hosts</tt><tt><br>
</tt><tt>#HostbasedAuthentication
no</tt><tt><br>
</tt><tt># Change to yes if you
don't trust ~/.ssh/known_hosts
for</tt><tt><br>
</tt><tt># HostbasedAuthentication</tt><tt><br>
</tt><tt>#IgnoreUserKnownHosts no</tt><tt><br>
</tt><tt># Don't read the user's
~/.rhosts and ~/.shosts files</tt><tt><br>
</tt><tt>#IgnoreRhosts yes</tt><tt><br>
</tt><tt><br>
</tt><tt># To disable tunneled
clear text passwords, change to
no here!</tt><tt><br>
</tt><tt>#PasswordAuthentication
yes</tt><tt><br>
</tt><tt>#PermitEmptyPasswords no</tt><tt><br>
</tt><tt><br>
</tt><tt># Change to yes to enable
challenge-response passwords
(beware issues with</tt><tt><br>
</tt><tt># some PAM modules and
threads)</tt><tt><br>
</tt><tt>ChallengeResponseAuthentication
no</tt><tt><br>
</tt><tt><br>
</tt><tt># Kerberos options</tt><tt><br>
</tt><tt>#KerberosAuthentication
no</tt><tt><br>
</tt><tt>#KerberosOrLocalPasswd
yes</tt><tt><br>
</tt><tt>#KerberosTicketCleanup
yes</tt><tt><br>
</tt><tt>#KerberosGetAFSToken no</tt><tt><br>
</tt><tt><br>
</tt><tt># GSSAPI options</tt><tt><br>
</tt><tt>#GSSAPIAuthentication no</tt><tt><br>
</tt><tt>#GSSAPICleanupCredentials
yes</tt><tt><br>
</tt><tt>#GSSAPIStrictAcceptorCheck
yes</tt><tt><br>
</tt><tt>#GSSAPIKeyExchange no</tt><tt><br>
</tt><tt><br>
</tt><tt># Set this to 'yes' to
enable PAM authentication,
account processing,</tt><tt><br>
</tt><tt># and session processing.
If this is enabled, PAM
authentication will</tt><tt><br>
</tt><tt># be allowed through the
ChallengeResponseAuthentication
and</tt><tt><br>
</tt><tt>#
PasswordAuthentication.
Depending on your PAM
configuration,</tt><tt><br>
</tt><tt># PAM authentication via
ChallengeResponseAuthentication
may bypass</tt><tt><br>
</tt><tt># the setting of
"PermitRootLogin
without-password".</tt><tt><br>
</tt><tt># If you just want the
PAM account and session checks
to run without</tt><tt><br>
</tt><tt># PAM authentication,
then enable this but set
PasswordAuthentication</tt><tt><br>
</tt><tt># and
ChallengeResponseAuthentication
to 'no'.</tt><tt><br>
</tt><tt>UsePAM yes</tt><tt><br>
</tt><tt>UseLogin no</tt><tt><br>
</tt><tt><br>
</tt><tt>#AllowAgentForwarding yes</tt><tt><br>
</tt><tt>#AllowTcpForwarding yes</tt><tt><br>
</tt><tt>#GatewayPorts no</tt><tt><br>
</tt><tt>X11Forwarding yes</tt><tt><br>
</tt><tt>#X11DisplayOffset 10</tt><tt><br>
</tt><tt>#X11UseLocalhost yes</tt><tt><br>
</tt><tt>#PermitTTY yes</tt><tt><br>
</tt><tt>PrintMotd no</tt><tt><br>
</tt><tt>#PrintLastLog yes</tt><tt><br>
</tt><tt>#TCPKeepAlive yes</tt><tt><br>
</tt><tt>#PermitUserEnvironment no</tt><tt><br>
</tt><tt>#Compression delayed</tt><tt><br>
</tt><tt>#ClientAliveInterval 0</tt><tt><br>
</tt><tt>#ClientAliveCountMax 3</tt><tt><br>
</tt><tt>#UseDNS no</tt><tt><br>
</tt><tt>#PidFile
/var/run/sshd.pid</tt><tt><br>
</tt><tt>#MaxStartups 10:30:100</tt><tt><br>
</tt><tt>#PermitTunnel no</tt><tt><br>
</tt><tt>#ChrootDirectory none</tt><tt><br>
</tt><tt>#VersionAddendum none</tt><tt><br>
</tt><tt><br>
</tt><tt># no default banner path</tt><tt><br>
</tt><tt>#Banner none</tt><tt><br>
</tt><tt><br>
</tt><tt># Allow client to pass
locale environment variables</tt><tt><br>
</tt><tt>AcceptEnv LANG LC_*</tt><tt><br>
</tt><tt><br>
</tt><tt># override default of no
subsystems</tt><tt><br>
</tt><tt>Subsystem sftp
/usr/lib/openssh/sftp-server</tt><tt><br>
</tt><tt><br>
</tt><tt># Example of overriding
settings on a per-user basis</tt><tt><br>
</tt><tt>#Match User anoncvs</tt><tt><br>
</tt><tt># X11Forwarding no</tt><tt><br>
</tt><tt>#
AllowTcpForwarding no</tt><tt><br>
</tt><tt># PermitTTY no</tt><tt><br>
</tt><tt># ForceCommand cvs
server</tt><tt><br>
</tt></div>
<div><br>
</div>
<div>I just checked the passwd file
on the server and both accounts I
use to login finish with
/bin/bash. However, I also noticed
that the last line of the passwd
file looks like this:</div>
<div><br>
</div>
<div><tt>sshd:x:100:65534::/run/sshd:/usr/sbin/nologin</tt></div>
<div><br>
</div>
<div>Looking at the passwd file from
a backup done before the upgrade,
and when ssh logins were working,
this line is a recent addition -
it does not appear in past
instances of the passwd file. Is
this the cause of my problems? Can
I simply delete this line and try
again?</div>
<div><br>
</div>
<div>Cheers.</div>
<div><br>
</div>
<div>Joe Aquilina</div>
<div><br>
</div>
<div><br>
</div>
<div>On 18/12/19 11:49 am, Chris Hoy
Poy wrote:<br>
</div>
<blockquote type="cite">
<div dir="auto">Hey Joe,
<div dir="auto"><br>
</div>
<div dir="auto">Can you check
what "usePrivilegeSeparation"
is defined as in the server
sshd_config is ?</div>
<div dir="auto"><br>
</div>
<div dir="auto">Cheers</div>
<div dir="auto">/Chris</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, 18
Dec 2019, 11:42 am Joe
Aquilina, <<a href="mailto:joe@chem.com.au" rel="noreferrer noreferrer
noreferrer noreferrer
noreferrer noreferrer
noreferrer noreferrer" target="_blank">joe@chem.com.au</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>sestatus and getenforce
both show selinux as
disabled.</div>
<div><br>
</div>
<div>There is already
another account that is
occasionally used to login
to the server - it fails
exactly the same as my
(joe) account. I don't
believe that any scripts
at login.<br>
</div>
<div><br>
</div>
<div>And yes I did edit the
output to protect the
"guilty" ... replaced the
real server name with
<server> and the
server's IP address. I
presumed that is what was
requested when it was
suggested that I post a
sanitised copy of the
login attempt output.</div>
<div><br>
</div>
<div>Cheers.</div>
<div><br>
</div>
<div>Joe Aquilina<br>
</div>
<div><br>
</div>
<div>On 18/12/19 11:08 am,
mike wrote:<br>
</div>
<blockquote type="cite">
<div>On 18/12/2019 10:43,
Joe Aquilina wrote:<br>
</div>
<blockquote type="cite">I
have no idea about
selinux, whether it is
installed/enabled. How
do I check that and
disable it if necessary,
and then re-enable?</blockquote>
<br>
<pre>sestatus or <span></span>getenforce
If file not found then not in use.
Are you removing details from the output? IE:
Authenticated to <server> ([ip.address of server]:22).
Mine says
debug1: Authentication succeeded (publickey).
Authenticated to nos ([10.222.0.4]:22).
Another thought is what does the passwd file say for your login? I have /bin/bash on the end
What user are you trying to login as?
Are you running any scripts at login that may be failing?
Have you tried another user?
Maybe create a new user and try logging in with that just to remove the user as being an issue.
</pre>
<pre cols="72">--
'ooroo
Mike...(:)-)
---------------------------------------------------
Email: <a href="mailto:mike@wolf-rock.com" rel="noreferrer noreferrer noreferrer noreferrer noreferrer noreferrer noreferrer noreferrer noreferrer" target="_blank">mike@wolf-rock.com</a> o
You need only two tools. o /////
A hammer and duct tape. If it /@ `\ /) ~
doesn't move and it should use > (O) X< ~ Fish!!
the hammer. If it moves and `\___/' \) ~
shouldn't, use the tape. \\\
---------------------------------------------------</pre>
</blockquote>
<p><br>
</p>
<pre cols="72">--
Joe Aquilina
Central Chemical Consulting Pty Ltd
PO Box 2546 Malaga WA 6944 Australia
1/11 Narloo St Malaga 6090 Australia
Tel: +61 8 9248 2739 Fax: +61 8 9248 2749
<a href="mailto:joe@chem.com.au" rel="noreferrer noreferrer noreferrer noreferrer noreferrer noreferrer noreferrer noreferrer noreferrer" target="_blank">joe@chem.com.au</a> <a href="http://www.chem.com.au" rel="noreferrer noreferrer noreferrer noreferrer noreferrer noreferrer noreferrer noreferrer noreferrer" target="_blank">www.chem.com.au</a> </pre>
</div>
_______________________________________________<br>
PLUG discussion list: <a href="mailto:plug@plug.org.au" rel="noreferrer noreferrer
noreferrer noreferrer
noreferrer noreferrer
noreferrer noreferrer noreferrer" target="_blank">plug@plug.org.au</a><br>
<a href="http://lists.plug.org.au/mailman/listinfo/plug" rel="noreferrer noreferrer
noreferrer noreferrer
noreferrer noreferrer
noreferrer noreferrer
noreferrer noreferrer" target="_blank">http://lists.plug.org.au/mailman/listinfo/plug</a><br>
Committee e-mail: <a href="mailto:committee@plug.org.au" rel="noreferrer noreferrer
noreferrer noreferrer
noreferrer noreferrer
noreferrer noreferrer noreferrer" target="_blank">committee@plug.org.au</a><br>
PLUG Membership: <a href="http://www.plug.org.au/membership" rel="noreferrer noreferrer
noreferrer noreferrer
noreferrer noreferrer
noreferrer noreferrer
noreferrer noreferrer" target="_blank">http://www.plug.org.au/membership</a></blockquote>
</div>
</blockquote>
<p><br>
</p>
<pre cols="72">--
Joe Aquilina
Central Chemical Consulting Pty Ltd
PO Box 2546 Malaga WA 6944 Australia
1/11 Narloo St Malaga 6090 Australia
Tel: +61 8 9248 2739 Fax: +61 8 9248 2749
<a href="mailto:joe@chem.com.au" rel="noreferrer noreferrer noreferrer noreferrer noreferrer noreferrer noreferrer noreferrer" target="_blank">joe@chem.com.au</a> <a href="http://www.chem.com.au" rel="noreferrer noreferrer noreferrer noreferrer noreferrer noreferrer noreferrer noreferrer" target="_blank">www.chem.com.au</a></pre>
</div>
_______________________________________________<br>
PLUG discussion list: <a href="mailto:plug@plug.org.au" rel="noreferrer noreferrer
noreferrer noreferrer noreferrer
noreferrer noreferrer noreferrer" target="_blank">plug@plug.org.au</a><br>
<a href="http://lists.plug.org.au/mailman/listinfo/plug" rel="noreferrer noreferrer
noreferrer noreferrer noreferrer
noreferrer noreferrer noreferrer noreferrer" target="_blank">http://lists.plug.org.au/mailman/listinfo/plug</a><br>
Committee e-mail: <a href="mailto:committee@plug.org.au" rel="noreferrer noreferrer
noreferrer noreferrer noreferrer
noreferrer noreferrer noreferrer" target="_blank">committee@plug.org.au</a><br>
PLUG Membership: <a href="http://www.plug.org.au/membership" rel="noreferrer noreferrer
noreferrer noreferrer noreferrer
noreferrer noreferrer noreferrer noreferrer" target="_blank">http://www.plug.org.au/membership</a></blockquote>
</div>
</blockquote>
<p><br>
</p>
<pre cols="72">--
Joe Aquilina
Central Chemical Consulting Pty Ltd
PO Box 2546 Malaga WA 6944 Australia
1/11 Narloo St Malaga 6090 Australia
Tel: +61 8 9248 2739 Fax: +61 8 9248 2749
<a href="mailto:joe@chem.com.au" rel="noreferrer noreferrer noreferrer noreferrer" target="_blank">joe@chem.com.au</a> <a href="http://www.chem.com.au" rel="noreferrer noreferrer noreferrer noreferrer" target="_blank">www.chem.com.au</a></pre>
</div>
_______________________________________________<br>
PLUG discussion list: <a href="mailto:plug@plug.org.au" rel="noreferrer noreferrer noreferrer noreferrer" target="_blank">plug@plug.org.au</a><br>
<a href="http://lists.plug.org.au/mailman/listinfo/plug" rel="noreferrer noreferrer noreferrer
noreferrer noreferrer" target="_blank">http://lists.plug.org.au/mailman/listinfo/plug</a><br>
Committee e-mail: <a href="mailto:committee@plug.org.au" rel="noreferrer noreferrer noreferrer noreferrer" target="_blank">committee@plug.org.au</a><br>
PLUG Membership: <a href="http://www.plug.org.au/membership" rel="noreferrer noreferrer noreferrer
noreferrer noreferrer" target="_blank">http://www.plug.org.au/membership</a></blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
<p><br>
</p>
<pre cols="72">--
Joe Aquilina
Central Chemical Consulting Pty Ltd
PO Box 2546 Malaga WA 6944 Australia
1/11 Narloo St Malaga 6090 Australia
Tel: +61 8 9248 2739 Fax: +61 8 9248 2749
<a href="mailto:joe@chem.com.au" target="_blank" rel="noreferrer">joe@chem.com.au</a> <a href="http://www.chem.com.au" target="_blank" rel="noreferrer">www.chem.com.au</a></pre>
</div>
_______________________________________________<br>
PLUG discussion list: <a href="mailto:plug@plug.org.au" target="_blank" rel="noreferrer">plug@plug.org.au</a><br>
<a href="http://lists.plug.org.au/mailman/listinfo/plug" rel="noreferrer noreferrer" target="_blank">http://lists.plug.org.au/mailman/listinfo/plug</a><br>
Committee e-mail: <a href="mailto:committee@plug.org.au" target="_blank" rel="noreferrer">committee@plug.org.au</a><br>
PLUG Membership: <a href="http://www.plug.org.au/membership" rel="noreferrer noreferrer" target="_blank">http://www.plug.org.au/membership</a></blockquote>
</div>
</blockquote>
<p><br>
</p>
<pre cols="72">--
Joe Aquilina
Central Chemical Consulting Pty Ltd
PO Box 2546 Malaga WA 6944 Australia
1/11 Narloo St Malaga 6090 Australia
Tel: +61 8 9248 2739 Fax: +61 8 9248 2749
<a href="mailto:joe@chem.com.au" target="_blank" rel="noreferrer">joe@chem.com.au</a> <a href="http://www.chem.com.au" target="_blank" rel="noreferrer">www.chem.com.au</a></pre>
</div>
_______________________________________________<br>
PLUG discussion list: <a href="mailto:plug@plug.org.au" target="_blank" rel="noreferrer">plug@plug.org.au</a><br>
<a href="http://lists.plug.org.au/mailman/listinfo/plug" rel="noreferrer noreferrer" target="_blank">http://lists.plug.org.au/mailman/listinfo/plug</a><br>
Committee e-mail: <a href="mailto:committee@plug.org.au" target="_blank" rel="noreferrer">committee@plug.org.au</a><br>
PLUG Membership: <a href="http://www.plug.org.au/membership" rel="noreferrer noreferrer" target="_blank">http://www.plug.org.au/membership</a></blockquote></div>