[plug] How to diagnose a crashing Linux server?

[Craig Ringer]

> The odd thing about my lastlog being 18Mb in size is that there are three
> valid users on the system, two of whom rarely log on, and the machine was
> set up a couple of months ago ~feb/mar.
> 
> I followed the other thread on this topic, and followed a similar tact,
> which recreated the lastlog as a 143kb file.

Interesting.

>>setterm -blank 0
> 
> Thanks - however my screen keeps turning off, I've turned 'off' energy
> saving in both the screen itself, and RH(Gnome) -> Preferences ->
> Screensaver | Advanced and also applied the CLI parameters as supplied
> above. Trying the 'failsafe' login mode to see if that stops the screen turn
> off.

Aha, you didn't mention you were running X anywhere. Can you run the 
server w/o X for normal operations, and just fire up X when needed? 
What's the motherboard chipset and video card? X can cause all sorts of 
stability issues occasionally, and its always a good idea to disable X 
logins for servers unless you have a good reason not to.

How to disable graphical logins depends on distro. In Red Hat you can vi 
/etc/inittab and change
id:5:initdefault:
to
id:3:initdefault:
but first make sure that all your other services will start as expected 
in runlevel 3. Unless you've customised your starting services heavily 
and only in runlevel 5, it should be the the same. Make sure with 
"chkconfig" and by looking at the /etc/rc3.d and /etc/rc5.d directories.

For debian, you can always simply remove /etc/rc2.d/S99gdm or mv 
/etc/rc2.d/{S,K}99gdm if you like (and assuming you're using gdm).

You can always fire up X manually by logging in on the console and 
running "startx". Alternately, if you're using GDM (method is different 
for other ?DMs) you can vi /etc/X11/gdm/gdm.conf and in the [servers] 
section comment out the line that starts an X server on GDM startup. 
This allows you to have GDM running but not starting local X servers - 
important if you use a server for serving remote X clients with XDMCP. 
If its set up to reply to XDMCP queries, you can get a login with
	X -query <server-running-gdm>

As for preventing the display from turning off, you can usually just 
"xset dpms 0 0 0". However something like GNOME might override that, 
and/or anything set in the XF86Config. I've never run a desktop 
environment on a server (and barely at all on a desktop box, I use a 
customised IceWM usually) so I can't really help you there. Suggested 
solution: disable X.

>>Does it reply to ICMP?
> 
> Will a 'ping' suffice? I haven't tried this, but can next time it fails.

A ping is what you want, yeah.

> One thing I've noticed, however is the memory usage is up reasonably high,
> at 3PM yesterday, following a reboot at around 9am, there was 293Mb of 1.5Gb
> of physical memory in use, with none of the 1.9Gb swap file being used.
> Today at 9.30AM there was 1.4Gb of memory in use, with no swap file being
> used. Currently (3PM today) there is still 1.4Gb of memory in use, the top
> usage programs are nautilus (~14Mb), python (~15Mb), gnome-settings-daemon
> (~12.2Mb), gnome-panel (~10Mb), gnome-system-monitor (~8.5Mb) and httpd
> (~8Mb) ... the total of these seems to be nowhere near the 1.4Gb being
> reported used. This info was collected via System Monitor. 'free' seems to
> indicate roughly the same figures.

That all sounds reasonable, most of your RAM useage will be cached files 
from the disk. For example:

[craig at bucket craig]$ free -m
            total       used       free     shared    buffers     cached
Mem:       2015       1808        207          0        304       1300
-/+ buffers/cache:     202       1812
Swap:      3999          0       3999

Here, I have 1.8 gigs of RAM "in use" but in reality, all but 200mb of 
that is available for programs to use - the kernel is just using it as a 
monster disk cache until its needed. I assume that's what you're seeing 
from your description, but I could easily be wrong - can you post a 
"free -m"?