Using MemcacheQ as message queue

Using a message queue can help to suspend heavy processes and execute them later so you won’t bother your visitors with long waiting times. There are a few solutions for queuing like Gearman, ActiveMQ and Zend Server Job Queue. For (a Dutch newspaper) we wanted a simple and free queue mechanism that integrates with Zend Framework for handling a number of jobs. We found MemcacheQ.


MemcacheQ has some nice features that makes it an ideal candidate for our purpose:

  • damn simple
  • very fast
  • multiple queues
  • great concurrency
  • memcache compatible protocol

Especially the last item is very interesting. This means that you can use the native memcache API for communicating with MemcacheQ, so you don’t need additional PHP extensions.

The installation has some dependencies like libevent and BerkleyDB. The latter is used for the persistent storage of the data inside the message queue. If something kills MemcacheQ you won’t loose the queued data.

MemcacheQ will run as a daemon and communicates over the UDP protocol. It can handle concurrent connections and can be used in a load-balanced environment.


There are some arguments to use when starting the service. We’re currently using the following on

memcacheq -d -r -H /data/memcacheq -N -R -v -L 1024 -u nobody &> \

Explanation of the used arguments:

  • -d – Run as daemon
  • -r – Maximize core file limit
  • -v – Verbose logging
  • -u <user> – Run as user nobody
  • -H – Directory to store the BDB files
  • -N – Performance improvement
  • -R – Remove logfiles when no longer needed
  • -L – Log buffer size, default is 32KB, we want 1024KB

Other helpful arguments:

  • -h – Show the help with all the arguments
  • -vv – More verbose mode
  • Run without -d to directly see the output


Zend Framework provides an adapter to talk with MemcacheQ: Zend_Queue_Adapter_Memcacheq
This makes it easy to integrate the queue in your website.


We have a news site and want to show the number of visits to an article. When storing this number in a database, we need to execute a query every time a visitor reads an article. That’s something you don’t want because you can’t cache it and the visitor has to wait until the INSERT/UPDATE has finished. So we’ll have to store the hit temporarily in a fast queue and insert it later on in the database.

On the article page we add the new hit to the queue.

And in a separate process (e.g. cronjob) we retrieve the queued hits and insert them in the database.

Known bugs

There’s currently (ZF version 1.11.2) one known bug in the ZF issue tracker: #ZF-9969.
This is a bug when using MemcacheQ 0.2.0.

Showing 9 comments
  • Youri

    Interesting technique. I was wondering what it is that can’t be cached? When you say ‘it’, do you mean the the db query that updates the hits count? I assume that you use (partial?) output caching?

    • Tim de Pater

      I indeed meant the query that updates the hit counter. We use some output caching but mostly data (e.g. resultsets) caching.

      • Youri

        I don’t really get the point how/why you would want to cache a hits count update query. Why can’t a news articlebe cached, and update a hits count at the same time? (apart from the fact that the user has to wait for the hits update query to finish).

        • Tim de Pater

          I now know what you mean.
          It’s not that I want to cache the query that updates the hit. I don’t want a update query being executed every time a visitor visits a page.
          That can cause a high load on the database on peak times. When using memcacheq we can bundle all queries and execute a query per page for all the visits, not per visit.
          And when a plane crashes and we get a major peak on the website, we can temporary turn the off the cronjob that processes the queue to take some load off the database.

  • Youri

    So by the term ‘caching’ in this context you mean saving up the queries and executing them periodically? That not really the definition of caching in my opinion.

    Very interesting that you put the real life examples here!

  • supriya

    will it work fine in ZF2.

  • Ira

    Thanks for a marvelous posting! I definitely enjoyed reading
    it, you could be a great author.I will always bookmark your blog and will come back in the
    future. I want to encourage yourself to continue your great writing,
    have a nice day!

  • Con

    Excellent way of explaining, and nice piece of writing to take information regarding my presentation subject matter, which i am going to
    present in college.

pingbacks / trackbacks

Start typing to search