networked day to day technical issues


Linux: realtime traffic monitoring and path determination

There are situations when one needs to give the answer to questions like:

- a) - what application/process is listening for inbound connections
- b) - what application/process is causing network traffic
- c) - what hosts are right now doing network traffic with our server
- d) - current rate of traffic going through the network interfaces
- e) - how much traffic is causing each workstation/server directly connected to the Linux server
- f) - which path is an outgoing packet going to take when you have multiple network cards and several routes (and more than one routing tables)


Visualize sar reports with awk and gnuplot

On several systems where only sar (part of sysstat) is collecting and storing performance data i needed to troubleshoot performance issues which occurred several hours earlier . Sar is a great tool but  it is annoying that it doesn't have any option to output at the same time , on the same page, output from different reports (like cpu usage, memory usage and disk usage). If you try to request those three at the same time, it will output each report on it's on page and from there it's hard to visualize how each performance indicator evolved at a specific point in time. A solution would have been to load the data in a spreadsheet application and use vlookup function to group the data but this is time consuming and with my spreadsheet skills i don't think it can be automated.

I used awk and order to create a report from sar output, choosing the fields i considered useful in 95% of the times. Because my display resolution width is 900 i managed too squeeze in a lot of fields. In order to get a report for the date of 18th from 10 AM to 6 PM i use:


The Confusion Between Gigabyte And Gibibyte

Recently i have been confronted with several cases where on the SAN side a LUN was configured, of a certain size and on the Linux side it was detected with a larger size. The SAN vendor's support team didn't ever reply to why this is happening and after some digging i managed to find the explanation. I presumed it was some kind of issue with how each part (SAN Storage Array and Linux) measured the LUN and it turned out to be so. The Linux in question is RHEL 5.5 64bit , kernel 2.6.18

It turns out that on the SAN side the size was presented in Gibibytes and Linux presented the size as Gigabytes .

1 Gigabyte (GB) = 1 x 1000 x 1000 x 1000 = 1000000000 bytes

1 Gibibyte (GiB) = 2^30 = 1 x 1024 x 1024 x 1024 = 1073741824 bytes

So a 500 GiB LUN was reported on Linux as ~536.8 GB

Unfortunately this confusion will haunt for a long time.

More details on wikipedia and  man 7 units


Disabling swap parititions which have swapped out pages

Due some circumstances i needed to expand a swap device which was sitting on top of a logical device (LVM) . Nothing spectacular and in this case i didn't want to add another swap device as there was enough free physical memory in order to hold all pages which were going to be moved from swap.

This got me thinking regarding what would happen if when running swapoff there wasn't enough free physical memory available to hold the pages from swap. As i seen it there were two options:

  • swapoff will refuse to disable swapping from the device and return an error
  • swapoff will start moving pages to free physical memory and once that is full OOM.kill would be called to wreck havoc in the system

As i didn't want to find out the hard way i did a test on Rhel 5.5 (CentOS 5.5) running kernel 2.6.18 and it turns out that at swapoff refuses the disable swapping on that device

If anyone interested in disabling swap, then in order to be sure there is enough space available the disk cache,dentries and inodes can be flushed (though the kernel will do this for you when running swapoff by flushing part or all of those) by doing echo 3 > /proc/sys/vm/drop_cache. Take into account that flushing the whole disk cache will affect to some extent the performance of your machine.

Later edit: on one occasion on a rhel 5.5 server doing the above echo did clear the cache as expected but that echo (bash process) didn't return and it started to consume 99% cpu (on one core) in system calls , was not killable (kill -9), it was in Running state (no waiting for I/O). No errors were reported by dmesg and eventually the server was rebooted so it seems that in some circumstances it might yield unexpected results, probably a bug.


Linux – disk caching and swap usage

The Linux kernel allocates unused physical memory(RAM) to disk caching in order to improve performance, how much was allocated can be seen by using the command free (or top or others) .

The memory used for caching is available on demand , so if the kernel needs it then it will clear flush an amount of the cache (or all) depending on the needs and it will allocate that memory to processes.

The kernel may decide to swap unused pages even if memory is allocated for caching (instead of clearing some of the cache) as it makes no sense to keep in memory data which hasn't been accessed for a long while and this data can be swapped out without affecting the performance while that amount of memory will be far more useful (performance wise) being used for disk caching.

We can force a cache cleaning (echo 1 > /proc/sys/vm/drop_caches) and event adjust the swappiness kernel parameter

vm.swappiness is a tunable kernel parameter that controls how much the kernel favors swap over RAM. A high swappiness value means that the kernel will be more apt to unmap mapped pages. A low swappiness value means the opposite, the kernel will be less apt to unmap mapped pages. In other words, the higher the vm.swappiness value, the more the system will swap. The default value is 60% .

One of the kernel developers, Andrew Morton , advises:

My point is that decreasing the tendency of the kernel to swap stuff out is wrong.  You really don't want hundreds of megabytes of BloatyApp's untouched memory floating about in the machine.  Get it out on the disk, use the memory for something useful.

If you have a portion of swap used and also a lot of memory allocated to disk caching , you can double check that the kernel is working as explained above by running vmstat and checking the si and so columns. Vmstat's man page states:

"si: Amount of memory swapped in from disk (/s).
so: Amount of memory swapped to disk (/s).

Now if those two show low to none activity then it is clear that the pages that were swapped out are not being used by the processes owning them and there was no point in keeping them in physical memory.


Linux LVM snapshot merge

Finally, one of the features i have been waiting for a very long time is available.
Today i was going over the release notes for Red Hat Enterprise 6 and i noticed that LVM snapshot merging is available.
Now this opens the possibility to revert changes like package upgrades (failed or successful) which before this i had to do using virtual machine snapshots or other disk/file system copy mechanisms .

Snapshot merging is accomplished with lvconvert and --merge flag. The man page states:

--merge Merges a snapshot into its origin volume. If both the origin and snapshot volume are not open the merge will start immediately. Otherwise, the merge will start the first time either the origin or snapshot are activated and both are closed. Merging a snapshot into an origin that cannot be closed, for example a root filesystem, is deferred until the next time the origin volume is activated. When merging starts, the resulting logical volume will have the origin’s name, minor number and UUID. While the merge is in progress, reads or writes to the origin appear as they were directed to the snapshot being merged. When the merge finishes, the merged snapshot is removed. Multiple snapshots may be specified on the commandline or a @tag may be used to specify multiple snapshots be merged to their respective origin.


Linux memory management

Recently i noticed that when running top command if you enable the SWAP column and add all of the values on that column you will get at least more swap usage than what you can see when running free , vmstat or checking the header of the top command; but it may often happen that the sum of the SWAP column actually is larger than physical swap space. In the top output below you can observe that by adding the value of SWAP column for the first 5 rows you will have a value larger the the current swap usage (listed on the 4th row in the header)

I checked that top actually gets correctly the data from /proc/process-id/status ; and it does for each running process so i moved on to see how Linux does memory management.