Mobile device detection with WURFL and Varnish

Avatar

If you have seen my presentation about Varnish on the PHPBenelux Conference 2011, you know already that Varnish is a really great reverse proxy caching system that can boost your website performance massively. A somewhat lesser-known feature of Varnish is that its VCL-configuration is very powerful. So powerful in fact, that we could easily add, replace or modify headers to and from Varnish. I have shown for instance in my presentation that you can send a X-Cache-Hit or X-Cache-Miss header to display if a request has been fetched from the either the Varnish cache or has been retrieved from the back-end.

After my presentation, Andreas Creten approached me, asking about mobile device detection in Varnish. He had problems on a website keeping up with all types of mobile devices to detect in the correct way. That question didn’t result in a direct answer, but did trigger something…

What if Varnish could figure out if the visitor was working on a mobile browser, and if so, set a flag to the back-end servers so it could display a mobile site? Impossible? No. Actually, it’s quite easy…Word of caution: we are going to leave the PHP domains and enter C. Varnish has the possibility to implement C-code inside the Varnish configuration file (yes, you can literally program inside the configuration file).

Meet WURFL: Wireless Universal Resource FiLe

It would be really nice if a browser just returned us if they were mobile or not, since we could just check that flag inside our php-code and be done with it. Unfortunately, this is not the case. But what we CAN check is the user-agent that a client send. If we can find out of the user-agent came from a browser on a mobile device, we could set that “mobile”-flag ourselves.

WURFL is a big data file with all the user-agents on mobile systems. We can safely assume that if the user-agent is defined in WURFL, it’s a mobile browser. WURFL has got loads of extra data like how big the mobile screen is, if it has wifi etc. Check out the possibilities on the WURFL site.

The plan

  1. Load WURFL XML into memory
  2. Get the user-agent from the incoming Varnish request
  3. Check if user-agent is inside the WURFL file
  4. Set X-Mobile flag in the Varnish response (either to yes or no)
  5. Profit!

Now, the main approach would be to implement this completely inside the VCL-configuration (/etc/varnish/default.vcl for instance), which is possible since you can use plain-C inside this. However, it bit of cramps our style, since we don’t have full control over everything so we use another way: we create our own detection-library (a dynamic shared object) and load that from the VCL.

Creating the .so:

Even if you don’t know C that well, the main idea is simple: load the XML (through a external library), make an “xpath” and see if we get any result. It will only return a 1 or 0 (or negative number on error) so we can use that as a boolean value.

and you need a header file (more or less, an interface)

Compilation should be done by:

gcc -c -o wurfl.o wurfl.c -I/usr/include/libxml2
gcc -shared -Wl,-soname,libwurfl.so.1 -o libwurfl.so.1.0.1 wurfl.o -lxml2

which creates the shared object, which uses libxml2 (so make sure you’ve got libxml2 libraries AND development files on your system).

Implementing in Varnish

Now that we have installed our shared object, implementation is a breeze.

Not really that difficult but let’s explain. First of all, we need to include our header file so the C-compiler knows about the wurfl_* functionality and we define a global “is_mobile” variable. Then we create a subroutine that fetches the user-agent header from the request through the VRT_GetHdr call. Notice the \013 in front of the User-Agent:, which is the length of the string we want to fetch in OCTAL. \013 means 11 in decimal. If you made a mistake, varnish crashes on the request calls with a “Condition(l == strlen(hdr + 1)) not true.” error.

Once we have fetched the user-agent, we can set the is_mobile variable (we use that later) and we the backend-response header. In our case, we set the X-Mobile: header (mind the strlength again), and add the “yes” or “no” behind it. Note we need to add vrt_magic_string_end to close the string.

Varnish has 3 different hooks that are processed before entering a backend. We need to hook our functionality to all of them. Finally, we want to display the resulting x-mobile flag to the user as well, so the vcl_deliver will add the header to the output that is returned to the visitor.

Testing the setup

So let’s try out the complete setup. I just use a simple php file that does only the CHECKING of the mobile flag:

We run this on the standard port 80. Next up: we start Varnish. We need to use some different startup parameters just to make sure varnish will actually use our WURFL library:

  varnishd -s malloc,32M -a 0.0.0.0:81 -f /etc/varnish/default.vcl \
  -p 'cc_command=exec cc -fpic -shared -Wl,-x -lwurfl -o %o %s'

I also use a small 32M malloc’ed buffer, and use port 81 to accept incoming connections. Now, when we direct the browser to port 81, we get something like this:

As you can see, the output has got a X-Mobile response, and the php-code actually can use HTTP_X_MOBILE. Since we use Firefox’s default user-agent (Mozilla 5.0 etc…), the X-mobile is set to “No”.

Now when we change the user-agent (with a simple Firefox extension) to an iphone 3.0 user-agent, this is our result:

All the mobile-flags are set to “yes” since it detected the iphone user-agent. It works… sweet!

Why not do this in PHP?

Two reasons: first of all, it’s much quicker to do it outside PHP since you don’t have all the php/zend overhead. Second reason: It might be even possible that either Varnish or the web-server behind Varnish will redirect the request to a dedicated web-server that only does the mobile site and maybe that website isn’t programmed in PHP but in something else (god forbid :p). Either way, you would need to program the same logic twice (or maybe even more) and you can redirect more quickly when you let varnish handle the detection.

Todo’s

This blog-post is just a proof-of-concept. We haven’t ran any benchmarks yet on how fast or slow it is and there is lots of room for improvement. For instance: instead of loading a big-HTML file, why not flatten the XML file to only user-agent (md5)-hashes and compare the md5-hash with an md5 from the user-agent. This would remove the need of a bulky XML overhead which saves time and memory, but I will leave this exercise up to you. 🙂

Big credits also to my colleague Joshua for helping me with this proof-of-concept!

CodeCuisine® Monthly

Onze stories in tech & business

1x per maand geserveerd in je mailbox

You are awesome.

Leuk dat je erbij bent!
Binnenkort ontvang je jouw eerste editie.

Start typing to search