networked day to day technical issues


Disabling swap parititions which have swapped out pages

Due some circumstances i needed to expand a swap device which was sitting on top of a logical device (LVM) . Nothing spectacular and in this case i didn't want to add another swap device as there was enough free physical memory in order to hold all pages which were going to be moved from swap.

This got me thinking regarding what would happen if when running swapoff there wasn't enough free physical memory available to hold the pages from swap. As i seen it there were two options:

  • swapoff will refuse to disable swapping from the device and return an error
  • swapoff will start moving pages to free physical memory and once that is full OOM.kill would be called to wreck havoc in the system

As i didn't want to find out the hard way i did a test on Rhel 5.5 (CentOS 5.5) running kernel 2.6.18 and it turns out that at swapoff refuses the disable swapping on that device

If anyone interested in disabling swap, then in order to be sure there is enough space available the disk cache,dentries and inodes can be flushed (though the kernel will do this for you when running swapoff by flushing part or all of those) by doing echo 3 > /proc/sys/vm/drop_cache. Take into account that flushing the whole disk cache will affect to some extent the performance of your machine.

Later edit: on one occasion on a rhel 5.5 server doing the above echo did clear the cache as expected but that echo (bash process) didn't return and it started to consume 99% cpu (on one core) in system calls , was not killable (kill -9), it was in Running state (no waiting for I/O). No errors were reported by dmesg and eventually the server was rebooted so it seems that in some circumstances it might yield unexpected results, probably a bug.


Linux – disk caching and swap usage

The Linux kernel allocates unused physical memory(RAM) to disk caching in order to improve performance, how much was allocated can be seen by using the command free (or top or others) .

The memory used for caching is available on demand , so if the kernel needs it then it will clear flush an amount of the cache (or all) depending on the needs and it will allocate that memory to processes.

The kernel may decide to swap unused pages even if memory is allocated for caching (instead of clearing some of the cache) as it makes no sense to keep in memory data which hasn't been accessed for a long while and this data can be swapped out without affecting the performance while that amount of memory will be far more useful (performance wise) being used for disk caching.

We can force a cache cleaning (echo 1 > /proc/sys/vm/drop_caches) and event adjust the swappiness kernel parameter

vm.swappiness is a tunable kernel parameter that controls how much the kernel favors swap over RAM. A high swappiness value means that the kernel will be more apt to unmap mapped pages. A low swappiness value means the opposite, the kernel will be less apt to unmap mapped pages. In other words, the higher the vm.swappiness value, the more the system will swap. The default value is 60% .

One of the kernel developers, Andrew Morton , advises:

My point is that decreasing the tendency of the kernel to swap stuff out is wrong.  You really don't want hundreds of megabytes of BloatyApp's untouched memory floating about in the machine.  Get it out on the disk, use the memory for something useful.

If you have a portion of swap used and also a lot of memory allocated to disk caching , you can double check that the kernel is working as explained above by running vmstat and checking the si and so columns. Vmstat's man page states:

"si: Amount of memory swapped in from disk (/s).
so: Amount of memory swapped to disk (/s).

Now if those two show low to none activity then it is clear that the pages that were swapped out are not being used by the processes owning them and there was no point in keeping them in physical memory.


Linux LVM snapshot merge

Finally, one of the features i have been waiting for a very long time is available.
Today i was going over the release notes for Red Hat Enterprise 6 and i noticed that LVM snapshot merging is available.
Now this opens the possibility to revert changes like package upgrades (failed or successful) which before this i had to do using virtual machine snapshots or other disk/file system copy mechanisms .

Snapshot merging is accomplished with lvconvert and --merge flag. The man page states:

--merge Merges a snapshot into its origin volume. If both the origin and snapshot volume are not open the merge will start immediately. Otherwise, the merge will start the first time either the origin or snapshot are activated and both are closed. Merging a snapshot into an origin that cannot be closed, for example a root filesystem, is deferred until the next time the origin volume is activated. When merging starts, the resulting logical volume will have the origin’s name, minor number and UUID. While the merge is in progress, reads or writes to the origin appear as they were directed to the snapshot being merged. When the merge finishes, the merged snapshot is removed. Multiple snapshots may be specified on the commandline or a @tag may be used to specify multiple snapshots be merged to their respective origin.