A few days ago I read an article that compared some frameworks. Amongst others, both Zend Framework 1 and Zend Framework 2 were included. What struck me was that ZF2 was 5 times slower then ZF1. I know, it’s in beta, not yet optimized for production use. But I couldn’t believe the difference was so great. So I’ve tested it myself and want to share my results in this blog.

Setup specs

I’m running the benchmark tests in a VirtualBox machine on my Thinkpad with the following specs;

Host

• Thinkpad T520
• Intel(R) Core(TM) i7-2620M CPU @ 2.70GHz
• 8GB RAM
• 500GB 7200 HDD
• Ubuntu 11.10

VirtualBox machine

• 1 CPU
• 1GB RAM
• 3GB HDD
• Clean Debian 6 install
• PHP 5.3.3-7+squeeze8 with Suhosin-Patch
• Apache/2.2.16 with KeepAlive disabled
• APC enabled

How I benchmark

To correctly interpret the benchmark results, it’s important to know how the results have been achieved.

Zend Framework 1

I’ve downloaded the latest Zend Framework 1 from the website, this is version 1.11.11. With the ZF.sh tool I’ve created a standard skeleton project.

Zend Framework 2

For Zend Framework 2 I’ve used the Zend Skeleton Application from GitHub. This includes Zend Framework 2, to be precise; version b3d970ee42.

ApacheBench

I’ve created a simple script that uses ApacheBench 2.3 and gnuplot to create the test results.
The scripts loops through the following steps;

1. Restart Apache on the VirtualBox machine and wait 5 seconds.
2. Run ab on ZF1, write the results to file.
3. Restart Apache on the VirtualBox machine and wait 5 seconds.
4. Run ab on ZF2, write the results to file.
5. Generate a plot of the results and wait 5 seconds.
6. Increase the number of concurrent connections and run again.

Benchmark results

I did the following benchmarks:

1. 2000 requests, 5 concurrent connections
2. 2000 requests, 10 concurrent connections
3. 2000 requests, 20 concurrent connections
4. 2000 requests, 40 concurrent connections
5. 2000 requests, 80 concurrent connections

I tried this with 512MB RAM in the VirtualBox machine, but this wasn’t enough for 80 concurrent connections (the OOM-killer had some fun) so I had to upgrade it to 1GB. That worked better, the swap file wasn’t used in any of the tests.

Graph results

The lower the response time, the better.

5 concurrent connections

20 concurrent connections

40 concurrent connections

80 concurrent connections

Conclusion

• ZF2’s response times are about 4 times slower than ZF1 in all tests.
• ZF1 is able to handle 51 requests/second, ZF2 12 requests/second.

So, can we conclude that ZF2’s performance sucks? No, definitely not.
Don’t forget it’s still in beta and I’m using a standard skeleton without any optimizations like caching and class map autoloading. And in this respect ZF2 has much more to offer than ZF1.

Before publishing I’ve discussed the results of this benchmark on freenode.net/#zftalk.2 to find out what the problem could be. Matthew Weier O’Phinney pointed out that the bulk of program execution right now is in the DI container. They know how to address this, but have lowered its priority in favor of finishing features needed.

Matthew said; “I’d also argue it’s premature to report benchmarks at this point. We’re well aware of issues currently, but plan to address them pre-stable release” And he’s completely right. This benchmark was meant purely to see if the article from piprime.fr made any sense. And it’s a starting point for me to see what you can do with the different optimizations and what their performance impact is.

There will be a follow-up to this article, hopefully with the conclusion that, with all possible optimizations correctly configured, ZF2 will kick ZF1′s ass!

What are your thoughts about this article? Please let me know in the comments!