networked day to day technical issues

11Mar/111

Linux: realtime traffic monitoring and path determination

There are situations when one needs to give the answer to questions like:

- a) - what application/process is listening for inbound connections
- b) - what application/process is causing network traffic
- c) - what hosts are right now doing network traffic with our server
- d) - current rate of traffic going through the network interfaces
- e) - how much traffic is causing each workstation/server directly connected to the Linux server
- f) - which path is an outgoing packet going to take when you have multiple network cards and several routes (and more than one routing tables)

a) what application/process is listening for inbound connections

This can be simply solved with netstat tool by running netstat --inet -lnp in order to show only IP related info

www:~# netstat --inet -lnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:5280            0.0.0.0:*               LISTEN      1259/beam
tcp        0      0 0.0.0.0:993             0.0.0.0:*               LISTEN      21620/cyrmaster
tcp        0      0 127.0.0.1:7777          0.0.0.0:*               LISTEN      1259/beam
tcp        0      0 0.0.0.0:389             0.0.0.0:*               LISTEN      1765/slapd
tcp        0      0 0.0.0.0:5222            0.0.0.0:*               LISTEN      1259/beam
tcp        0      0 127.0.0.1:10024         0.0.0.0:*               LISTEN      7938/amavisd (maste
tcp        0      0 0.0.0.0:53352           0.0.0.0:*               LISTEN      1259/beam
tcp        0      0 127.0.0.1:10025         0.0.0.0:*               LISTEN      3438/master
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      3618/mysqld
tcp        0      0 127.0.0.1:10026         0.0.0.0:*               LISTEN      3438/master
tcp        0      0 0.0.0.0:587             0.0.0.0:*               LISTEN      3438/master
tcp        0      0 127.0.0.1:10028         0.0.0.0:*               LISTEN      3438/master
tcp        0      0 0.0.0.0:143             0.0.0.0:*               LISTEN      15732/imapd
tcp        0      0 127.0.0.1:9999          0.0.0.0:*               LISTEN      3542/perl
tcp        0      0 0.0.0.0:2000            0.0.0.0:*               LISTEN      21620/cyrmaster
tcp        0      0 0.0.0.0:465             0.0.0.0:*               LISTEN      3438/master
tcp        0      0 0.0.0.0:4369            0.0.0.0:*               LISTEN      1242/epmd
tcp        0      0 127.0.0.1:2003          0.0.0.0:*               LISTEN      21620/cyrmaster
tcp        0      0 192.168.0.1:53        0.0.0.0:*               LISTEN      28530/named
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      28530/named
tcp        0      0 0.0.0.0:5269            0.0.0.0:*               LISTEN      1259/beam
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      9836/sshd
tcp        0      0 127.0.0.1:953           0.0.0.0:*               LISTEN      28530/named
tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN      3438/master
tcp        0      0 0.0.0.0:636             0.0.0.0:*               LISTEN      1765/slapd
tcp        0      0 0.0.0.0:65534           0.0.0.0:*               LISTEN      9836/sshd
udp        0      0 0.0.0.0:36252           0.0.0.0:*                           3778/openvpn
udp        0      0 127.0.0.1:161           0.0.0.0:*                           1312/snmpd
udp        0      0 192.168.0.1:53        0.0.0.0:*                           28530/named
udp        0      0 127.0.0.1:53            0.0.0.0:*                           28530/named
www:~#

b) what application/process is causing network traffic

This is achievable with the nethogs utility (http://nethogs.sourceforge.net)

"NetHogs is a small 'net top' tool. Instead of breaking the traffic down per protocol or per subnet, like most tools do, it groups bandwidth by process. NetHogs does not rely on a special kernel module to be loaded. If there's suddenly a lot of network traffic, you can fire up NetHogs and immediately see which PID is causing this. This makes it easy to indentify programs that have gone wild and are suddenly taking up your bandwidth."

c) what hosts are right now doing network traffic with our server

If you need a top like display of host and traffic rates then iftop ( http://www.ex-parrot.com/pdw/iftop ) is the tool for you

"iftop does for network usage what top(1) does for CPU usage. It listens to network traffic on a named interface and displays a table of current bandwidth usage by pairs of hosts. Handy for answering the question "why is our ADSL link so slow?"."

Similar output can be obtained from iptraf's (http://iptraf.seul.org) IP traffic monitor module/mode which can also show Paket Size and Window Size :

d) current rate of traffic going through the network interfaces

Again iptraf is the tool for the job. The module General interface statistics shows a brief summary for each interface while the module Detailed interface statistics will print a lot of useful information regarding incoming/outgoing traffic, separation by TCP / UDP , traffic rate measured in bytes and total packets per second count

General interface statistics:

Detailed interface statistics:

e) how much traffic is causing each workstation/server directly connected to the Linux server

By directly connected i mean a device which is in the same ethernet broadcast domain which most of the time is also the same IP broadcast domain. Examples include things like workstations connected to your server via switches (and no routers in between).

Iptraf allows you to monitor the ethernet traffic on a particular interface via its LAN station monitor module so you can see the list of MAC addresses doing traffic .

Once you the the MAC address(es) you can use the arp or ip tool in order to convert them to ip addresses.

www:~# arp -n | grep 00:30:48:df:27:24
1.2.3.4            ether   00:30:48:df:27:24   C                     eth0
www:~# ip neigh show | grep 00:30:48:df:27:24
1.2.3.4 dev eth0 lladdr 00:30:48:df:27:24 REACHABLE
www:~#

f) which path is an outgoing packet going to take when you have multiple network cards and several routes (and more than one routing tables)

While this might sound a simple task if you have only two - three routes ; it does get complicate when there are a lot of routes in the routing table from something like a dynamic routing protocol daemon and or if you have multiple routing tables (which can be set up with iproute2) .
Traditionally one would run netstat -rn or route -n and examine the few routes showed but as described above this might not be sane when there are a lot of routes and interfaces and not even possible with those two tools if you are using multiple routing tables.
iproute2 comes to the rescue with the ip utility which allows you to easly find out the route for a particular destination by running ip route get ADDRESS [ from ADDRESS iif STRING ] [ oif STRING ] [ tos TOS ]:

www:~# ip route get 3.4.5.6
3.4.5.6 via 1.2.3.1 dev eth0  src 1.2.3.4
    cache  mtu 1500 advmss 1460 hoplimit 64
www:~#

A nice thing is that it also shows you the source address used when sending the packed. The man page states the following:
"Note that this operation is not equivalent to ip route show. show shows existing routes. get resolves them and creates new clones if necessary. Essentially, get is equivalent to sending a packet along this path. If the iif argument is not given, the kernel creates a route to output packets towards the requested destination. This is equivalent to pinging the destination with a subsequent ip route ls cache, however, no packets are actually sent. With the iif argument, the kernel pretends that a packet arrived from this interface and searches for a path to forward the packet."

To see the full path to the destination you can use traceroute which is slow and not that useful nowadays or better use mtr (Matt's traceroute http://www.bitwizard.nl/mtr ) which "combines the functionality of the traceroute and ping programs in a single network diagnostic tool"

  • Great post, thank you!

    You should probably also mention tcpdump and wireshark, which are both “Swiss army knife” software where it comes to network debugging.