hits
The fastest and best designed among the modern web servers
Benchmarking software is not an easy thing to do. It may look trivial, but it requires a bast amount of time and effort when you want to do it right.
In this case, I've been working on a series of web server benchmarks. The target was to measure how the modern web servers perform in different scenarios: different contents, requests, load, and even different hardware.
Most of our previous benchmarks were based on the result of ab requesting a few hundred thousand times the very same file, and that isn't obviously measuring how the server would perform in a real production environment.
So, in this post I'm going to present the result of a more realistic benchmark. This time, instead of requesting a single file, there were 300 different files for the clients to request. Files were between 156 bytes and 14MB.
By the way, as you may have noticed, I said that I'd benchmark the "modern web server". That's an important point. I have chosen to benchmark: Cherokee, Lighttpd and Nginx. All the rest did not made it to be included; the two most notorious cases:
The always interesting but 15 years old Apache: There was a problem about it. I had to choose whether to use the pre-fork or threads based back-end. If I used pre-fork the performance would be -let's say- quite poor. But I couldn't run it with threads because I needed the same server to execute PHP (with mod_php) and that sort of set up is not thread-safe. So, by one reason or the other it wouldn't really fit in the test, unless I cheated by running it with threads for static content, and pre-fork for mod_php.
Non-FOSS servers: The big problem about these servers is that they don't allow to publish public performance benchmarks. So, there is nothing I can do here. We'll have to keep those number internal.
So, without any further delay, here you have the results: (X=concurrent clients, Y=reqs/s)
Note that the difference between Cherokee and the rest increases with the number of concurrent clients. That means that it'll be the best performing web server while your client number raises (when you need it the most, actually).
So, as you can see, Cherokee is the fastest and best designed among the modern web servers. But it's even better than that. The next benchmark will be focused on dynamic content delivery, and it will show how Cherokee outperforms all the rest by even a bigger difference. Stay tuned!
Edit (Sat Oct 18 17:16:23 CEST 2008): For the sake of correctness, here you have a couple of scripts (1, 2), and configuration files (1, 2, 3) that the benchmark has been based upon.
Comments
Dude, Tufte would kill you if he saw that graph. It would be a whole lot easier to read without the perspective.
Including Apache 2.x worker with php5/fastcgi would be a nice addition to this benchmark though..
As you get your blog syndicated at least on planet GNOME you should put a disclaimer on your benchmarks that reveal that a) you are the creator / developer of cherokee and b) you have a start-up behind it. :-)
@Marcus: Yes, you are right on that; although that doesn't make Cherokee any slower. Besides, as you pointed out, another big advantage of the project is that Octality is behind it (offering professional support). That's extremely important for the adoption in enterprise environments.
Alvaro: no, it will not make Cherokee slower but it would help people to take that result with a grain of salt (you are obviously biased, although I'm not saying you're publishing incorrect results, pay attention!). Moreover it would be interesting to know more about this test: hardware, software configuration, test methodology etc ect. Thanks anyway for these numbers.
mod_fcgid exists. Isnt fcgi what all other servers use for php anyway? I also have to agree with Juri the 3d graph style makes it way much harder to interpret the results - without any benefit.
Is there any way to *securely* configure cherokee? I mean, listening to localhost isn't exactly secure, as *every* program on localhost can then configure cherokee? Surely I must miss something here?
You lose credibility when you dismiss the market leader based on pure misinformation. 1. The "15 year old" Apache has three children alive and well today (as well as the parent, if you count Apache 1.3 as alive with lots of users but no real developer interest). Only one of them is still called Apache. The fact that the others are called lighttpd and nginx doesn't distance them from old-apache; it just means they've developed in different directions to modern-apache. 2. The "prefork MPM slow" assertion is untested. Well OK, it's tested elsewhere, and found both right and wrong depending on the nature of the backend load. 3. As others have said, running PHP with a threaded MPM is not a problem. FastCGI supports that with Apache, just as with other servers. Having said all that, the major thrust of Apache 2.x is in much-improved application support, and most of that is wasted with PHP. Unless you're a very unusual PHP user, capable of mix-and-matching different technologies in an application.
@Alvaro: No, there is nobody selling support for other webservers... Do yourself a favour and try to stay fair.
nginx is configured incorrectly. Add multiple workers, so it is a fair comparison. worker_processes 6; Make sure enough max files descriptors are allocated. worker_rlimit_nofile 10240; Add more connections per worker. worker_connections 8024; You will see nginx will perform a lot better once you apply the mentioned configuration options.
I tried to run Django on Cherokee without success, I have filled a bug, of course. I like the clean interface Cherokee has, but to be fair actually web servers need to serve more than static content. I have seen that Cherokee has an special way to do that, but in my oppinion it needs more work on the mod_x/fastcgi apache equivalent.
It will be nice if Cherokee could interpret .htaccess, I think that it's not a big problem to add/write a module which could interpret .htaccess that was written for Apache. Most people will have another good reason to switch to Cherokee.
Why is yaws excluded from that list?
@yaws: good point! In fact, I was also thinking of adding yaws to the comparison while reading the post. I think it could drop very interesting results as well
Why is cherokee that fast? Is that explained somewhere? Afai remember Lighty has some good explanation somewhere.
does cherokee support ftp?
it would be nice to learn about the hardware used for this test.
Can you update the result using the more recent version of Cherokee?
most important would be to have the MEMORY FOOTPRINT of all servers while this test - of course for a fair test you should ask the developers of each server to contribute with a finetuned config for the given test situation! Also it would be very nice to have more realistic long-term test, to learn how stable the servers are. Thanks!
Corsale Benchmark Methodology. 1. List all intents in each of the tests. 2. Illustrate and re-iterate correlation to "real world" scenarios. 3. Run each bench test twelve times in non-liner procedures. As in, do not simply run the same test back to back. 4. For each iteration, move the test at the bottom of the stack up one positin, displacing the first test to the bottom in spill over buffer fashion. 5. Publish the tools, software, hardware ID's including all firmware versions. 6. Disclose any conceivable bias in the first paragraph of the results and make sure that it's dually noted in syndication.
