The other day my Linux box crashed while I was running a bunch of stuff including VMware Player (by crash, I mean no display, no SSH, no virtual consoles; the only thing that still worked was it still responded to pings). Since it is rare for Linux to crash like this, after rebooting I looked at the logs and saw:

Dec 19 04:57:52 shifter kernel: Swap cache: add 3743740, delete 3740409, find 1403709/2392711, race 41+5059
Dec 19 04:59:31 shifter kernel: 0 bounce buffer pages
Dec 19 04:56:50 shifter mgetty[22816]: failed dev=ttyS0, pid=22816, got signal 14, exiting
Dec 19 05:03:08 shifter kernel: Free swap:       1816376kB
Dec 19 05:06:28 shifter kernel: 129008 pages of RAM
Dec 19 05:06:42 shifter kernel: 0 pages of HIGHMEM
Dec 19 05:08:41 shifter kernel: 2372 reserved pages
Dec 19 05:09:29 shifter kernel: 150030 pages shared
Dec 19 05:14:46 shifter kernel: 3243 pages swap cached
Dec 19 05:22:22 shifter kernel: Out of Memory: Killed process 8446 (eggcups).
Dec 19 05:29:35 shifter kernel: oom-killer: gfp_mask=0xd2

When a Linux system is super low on memory (including swap space), oom-killer kicks in and starts to well, kill. It's extremely indiscriminate about what it starts killing. After all, there is no swap space left so the system has already exhausted other options like paging things out of memory to swap space and swapping whole processes out of memory to swap space. Now it's time to simply get out the gun and start shooting. Usually this is really bad, because it's bound to kill things that are important to you and to the proper running of the system.

Now my box only had 512 MB of RAM (upgraded to 2 GB yesterday) and had been configured with only a 2 GB swap partition, so with a bunch of stuff running, you can see how things got tight.

Time to make more swap space...

To see how much swap space you have configured:

% swapon -s

Then to add more:

% sudo dd if=/dev/zero of=/var/tmp/swapfile bs=1G count=2      # Create 2 GB file
2+0 records in
2+0 records out
% sudo mkswap /var/tmp/swapfile
Setting up swapspace version 1, size = 2147479 kB
% sudo swapon /var/tmp/swapfile
% swapon -s
Filename                                Type            Size    Used    Priority
/dev/hda2                               partition       2048276 176     -1
/home/var/tmp/swapfile                  file            2097144 0       -2