Picture this, you have a loadbalanced environment running with separate web servers, cache servers and database servers. Everything is running great, until suddenly the monitoring is yelling, the load on several servers is rising, MySQL queries/second and the memcached commands/second going through the roof, Apache processes are higher than usual, and the website starts giving timeouts.

Yes, that sucks. Of course you’ll first check everything that comes up in your mind like logs, diskspace, swap, etc. But then you come to a point that you really have to dive into it to find the cause of this sudden problem.

There are several ways of doing this. One way I learned the last time we were in this scenario is using tcpdump and Wireshark.


Wireshark is a powerful network protocol analyzer. It can go to the deepest level of packet inspection. The great thing about Wireshark is that it understands the protocols, giving the ability to filter on specific characteristics of a protocol.


I haven’t seen many servers that have a GUI running on it, so you need a command-line tool to capture the packets and inspect them with Wireshark.
There are two programs that do this: tcpdump and tshark.

Tshark is part of Wireshark and even has the ability to give realtime output directly to Wireshark. The downside of Tshark is that the package is not widely available in the common package managers.
Tcpdump on the other hand can be found by most package managers and chances are that it’s already installed on the server. So in this case I’ll go for tcpdump.

Tcpdump is also a very powerful network inspector and has much more features then we use here. See the MAN page for examples.

Capturing packets

First you need to install tcpdump. This can be done with most package managers or download the latest release on tcpdump.org.
You need to run tcpdump as root, because it needs access to the ethernet device.

Start tcpdump and log every packet to a file:

tcpdump -i <interface> -s 65535 -w <file>

<interface> should be the interface through which the traffic you want to inspect is going. This wil almost always be eth0 if you have only one interface.
<file> is the output file. This can be anything, but watch out, the file can grow fast when there’s a lot of traffic.

Run this program for a while and then stop it with ctrl + c.
Then download the file to your computer with something like scp (if the file is very large then compress it first with bzip2 or gunzip)

Filtering with Wireshark

Now you have a file full of packets, it’s time to start up Wireshark and load the file by choosing File -> Open in the menu en selecting the right file.

Now the magic happens.. In my case I want to filter all the memcache traffic to see what keys are requested most. It’s easy to create a filter for a specific protocol by typing the protocol in the filter bar or click ‘Expression..’ to click your filter together.

To filter all the memcache packets you use the filter ‘memcache‘. We also want to see which cache key is requested per packet. This can be done by adding a extra column to the list.

  1. Right-click on the columns and choose Column Preferences.
  2. Click Add
  3. Choose field type custom
  4. Use ‘memcache.key’ as field name.
  5. Click Ok.

Now you can see which key is requested. There are many more filters for memcache, see the full list here.

You can do the same for MySQL. Most of the time you don’t have a general query log on production. With Wireshark you can catch all queries and display them as a query log. To filter out the mysql packets you just use the filter ‘mysql‘ or ‘mysql.query != “”‘ when you only want packets that request a query. After that you can add a custom column with the field name ‘mysql.query’ to have a list of queries that where executed.

More interessting protocols for webapplications;

The possibilities are endless. Everything that communicates over ethernet is visible and all the common protocols are recognized by Wireshark and thus you’re able to filter on protocol specific characteristics.

In my case I found out that some of the packets that where going to memcache were larger than 1MB, and that’s the default limit for memcache without compression..

Showing 15 comments
  • Ivan

    Thank you! very usefull!

  • Reply

    Often I don’t study publish in websites, on the other hand would wish to say that this kind of write-up quite compelled me you need to do so! Your own composing flavor continues to be surprised us. Thank you so much, extremely wonderful submit.

  • porno pour smartphone

    EuhҺɦ êtes vouѕ sûr de ce que vouѕ écrivez ?

  • videos porno gratis

    ʟ’ensemble des pߋsts sont follement fascinants

  • sexe hard gratuit

    Vous fɑîtes constamment dess articles attractifs

  • gros nibards

    C’est infiniment une joiе de venir ѕur ce blog

  • Femme Salope

    Un gigantesque ƅraavo auu créateur de ce
    ѕite web

  • cochonne porno

    Je pսblie ce commentaire simplement poyr remercier son auteur

  • ddl XXX

    Je рrends la peine de pսblier un commentaire uniquement pour remercier
    le webmaster

  • film porno hard

    Je ρeux dire que c’est continuellement de la joie de viiter ce blog

  • coquine sexy

    Εncore un post réellement captivant

  • vidéo hard gratuite

    On voit ԁirectement գue vous connaissez superbement bien ce que vous dites

  • poufiasse chaude

    Vous rédigeƶ toujours des postes intéressants

  • Quincy

    Do you take challenges and try superb pushes, or seek to retain things uncomplicated.

    Perhaps I am merely also into fashion or even I am merely getting a classic minute; contemplating each of the
    lecturers who used spectacles if they were coaching myself.

pingbacks / trackbacks

Leave a Comment