Picture this, you have a loadbalanced environment running with separate web servers, cache servers and database servers. Everything is running great, until suddenly the monitoring is yelling, the load on several servers is rising, MySQL queries/second and the memcached commands/second going through the roof, Apache processes are higher than usual, and the website starts giving timeouts.
Yes, that sucks. Of course you’ll first check everything that comes up in your mind like logs, diskspace, swap, etc. But then you come to a point that you really have to dive into it to find the cause of this sudden problem.
There are several ways of doing this. One way I learned the last time we were in this scenario is using tcpdump and Wireshark.
Wireshark is a powerful network protocol analyzer. It can go to the deepest level of packet inspection. The great thing about Wireshark is that it understands the protocols, giving the ability to filter on specific characteristics of a protocol.
I haven’t seen many servers that have a GUI running on it, so you need a command-line tool to capture the packets and inspect them with Wireshark.
There are two programs that do this: tcpdump and tshark.
Tshark is part of Wireshark and even has the ability to give realtime output directly to Wireshark. The downside of Tshark is that the package is not widely available in the common package managers.
Tcpdump on the other hand can be found by most package managers and chances are that it’s already installed on the server. So in this case I’ll go for tcpdump.
Tcpdump is also a very powerful network inspector and has much more features then we use here. See the MAN page for examples.
First you need to install tcpdump. This can be done with most package managers or download the latest release on tcpdump.org.
You need to run tcpdump as root, because it needs access to the ethernet device.
Start tcpdump and log every packet to a file:
tcpdump -i <interface> -s 65535 -w <file>
<interface> should be the interface through which the traffic you want to inspect is going. This wil almost always be eth0 if you have only one interface.
<file> is the output file. This can be anything, but watch out, the file can grow fast when there’s a lot of traffic.
Run this program for a while and then stop it with ctrl + c.
Then download the file to your computer with something like scp (if the file is very large then compress it first with bzip2 or gunzip)
Filtering with Wireshark
Now you have a file full of packets, it’s time to start up Wireshark and load the file by choosing File -> Open in the menu en selecting the right file.
Now the magic happens.. In my case I want to filter all the memcache traffic to see what keys are requested most. It’s easy to create a filter for a specific protocol by typing the protocol in the filter bar or click ‘Expression..’ to click your filter together.
To filter all the memcache packets you use the filter ‘memcache‘. We also want to see which cache key is requested per packet. This can be done by adding a extra column to the list.
- Right-click on the columns and choose Column Preferences.
- Click Add
- Choose field type custom
- Use ‘memcache.key’ as field name.
- Click Ok.
Now you can see which key is requested. There are many more filters for memcache, see the full list here.
You can do the same for MySQL. Most of the time you don’t have a general query log on production. With Wireshark you can catch all queries and display them as a query log. To filter out the mysql packets you just use the filter ‘mysql‘ or ‘mysql.query != “”‘ when you only want packets that request a query. After that you can add a custom column with the field name ‘mysql.query’ to have a list of queries that where executed.
More interessting protocols for webapplications;
The possibilities are endless. Everything that communicates over ethernet is visible and all the common protocols are recognized by Wireshark and thus you’re able to filter on protocol specific characteristics.
In my case I found out that some of the packets that where going to memcache were larger than 1MB, and that’s the default limit for memcache without compression..