Paul M. Jones

As many of you know, I maintain a series of web framework benchmarks. The project codebase is here and the most recent report is here.

It was with some interest, then, that I viewed Rasmus Lerdorf's slides on the subject of performance benchmarking. I'm beginning to think there's something unexpected or unexamined in his testing methodology.

Note: see the update at the end of this entry.

On this slide, Rasmus notes the following:

• Static HTML nets 611.78 trans/sec
• Trivial PHP nets 606.77 trans/sec

This would seem to indicate that the mere invocation of PHP, on Rasmus' setup, reduces Apache's performance from serving static pages by less than 1%.

In my testing on Amazon EC2 small instances, I note somewhat different results:

• Static HTML nets 2309.14 req/sec
• Trivial PHP nets 1320.47 req/sec

The net reduction there is about 43%. Yes, this is with opcode caching turned on.

I then became really curious as to how Rasmus might have his system set up, to see only a 1% bit of "added overhead" from PHP being invoked. It would be nice if I could set up my own systems the same way.

When I asked about that at work, my colleague Elizabeth Smith opined that maybe Rasmus' web server is running all requests through the PHP interpreter, not just PHP files. That sounded like a good intuition to me, so I set up an EC2 instance to try it out.

Per the setup instructions on this page I built an EC2 server the same as I've done for my own benchmarking reports. I didn't check out the whole project, though; this time we just need the "bench", "baseline-html", and "baseline-php" directories.

As a reminder, the baseline index.html page is just the following plain text ...

Hello World!

... and the baseline index.php page is the following single line of PHP code:

<?php echo 'Hello World!'; ?>

The php5.conf file for Apache looks like this by default ...

<IfModule mod_php5.c>
  AddType application/x-httpd-php .php .phtml .php3
  AddType application/x-httpd-php-source .phps
</IfModule>

... and we're going to leave it alone for a bit.

Using ab -c 10 -t 60 to benchmark baseline-html and baseline-php with the default php5.conf gives us these results (an average over 5 runs each):

               |      avg
-------------- | --------
baseline-html  |  2367.02
baseline-php   |  1270.15

That's a 47% drop for invoking PHP. (That is itself 4 points different than the numbers I show above, so it appears there are some variables I have not controlled for, or maybe I just need to let this run longer than 5 minutes to smooth out the deviations.)

To test our hypothesis, we modify the php5.conf file to add .html to the list of files that get passed through PHP ...

<IfModule mod_php5.c>
  AddType application/x-httpd-php .html .php .phtml .php3
  AddType application/x-httpd-php-source .phps
</IfModule>

... restart Apache, and run the same ab tests again:

framework      |      avg
-------------- | --------
baseline-html  |  1348.80
baseline-php   |  1341.31

That's less than a 1% drop -- close enough to make make me think that Rasmus might be pushing everything through the PHP interpreter, regardless of whether or not it's a PHP file.

If that is true (and it's a big "if"), then merely invoking PHP does appear to cause about a 45% drop (give or take) in Apache's responsiveness, which is contrary to the point Rasmus makes on this slide about PHP "rarely" being a bottleneck -- and I say that as someone who works with PHP almost exclusively. In fairness, I am depending only on the text of his slides here, so he may have said something to that effect in the presentation itself.

Failure modes on this analysis:

• I am using XCache and not APC for the opcode cache. (Why? Because it works with both Lighttpd+FCGI and Apache+modphp, at least the last time I checked, and I'm interested in the differences between those two setups.)

• I am using an EC2 server, which is more production-ish than Rasmus' laptop.

• I am using ab to benchmark with, not siege. I tried using Siege and did not notice any significant differences, so I'm sticking with the ab tools I've built for now.

I can't imagine those three differences would lead to the kind of disparity in performance that I'm seeing, but it's possible.

Has anyone else tried doing this?

Rasmus, if you have the time and inclination, would you care to shed some light on these prognostications?

Update: Is It EC2?

Rasmus replies below that he did not, in fact, have PHP running for all .html files.

For me, the next question was to see what the real difference on EC2 is between no cache, XCache, and APC.

No cache:

               |      avg
-------------- | --------
baseline-html  |  2339.25
baseline-php   |  1197.28

XCache (copied from above)

               |      avg
-------------- | --------
baseline-html  |  2367.02
baseline-php   |  1270.15

APC:

               |      avg
-------------- | --------
baseline-html  |  2315.83
baseline-php   |  1433.91

So on EC2, you get about 1200 req/sec without caching, about 1300 req/sec with XCache, and about 1400 req/sec with APC, in the "hello world" baseline scenarios.

Maybe this is all an artifact of how EC2 works, then? I have no idea. Next step is to test on non EC2 systems, if I can find one that others can reasonably build on themselves (since one of the goals here is for others to be able to duplicate the results).