Paul M. Jones

Fabien Potencier released Symfony 2.0.0alpha1 last week, along with some benchmarks showing its performance. I am glad to see that Fabien used my benchmarking system and methodology, and am happy to see that he is paying attention to the performance of his framework. I take this as an acceptance on his part that my methodology is legitimate and valid, and that it has value when comparing framework responsiveness.

However, in attempting to reproduce the published Symfony 2 benchmarking results, I found Fabien's reporting to be inaccurate (or at least incomplete). Read on for a very, very long post detailing my attempt to replicate his results for the "hello world" basic framework overhead comparison, and my conclusions.

For the impatient, here are my conclusions in advance:

1. Fabien's benchmark report, as shown at http://symfony-reloaded.org/fast, is inaccurate for the setup he describes. Lithium and Flow3 do not work in Fabien's benchmark codebase at Github. Also, Symfony 2 is faster than Solar beta 3 by 5%, not 20%, on a "c1.xlarge" instance; to get a relative difference like Fabien describes, one has to use an "m1.large" instance. (It is entirely possible that the process Fabien used for benchmarking is incompletely described, and that the codebase is not fully updated, thus contributing to this disparity in results.)

2. We should use Siege 2.69, not 2.66, for more accurate benchmarking of baseline responsiveness. If we notice that HTML is slower than PHP, it's a sign that something is wrong.

3. Symfony 2 preloads its foundation and application classes, something no other framework does in the benchmarked code. When we treat Solar and Symfony 2 the same way, by preloading the foundation classes for each, we find that Solar is roughly 28% faster than Symfony 2.

Overview

Full disclosure: I am the architect of the Solar Framework for PHP discussed herein. I have been doing these benchmarks for years; see the "benchmarks" tag on this blog.

The primary point of this post is to show that benchmarking is tedious, time-consuming, difficult, and requires a lot of attention to details. I spent two full days doing all the following work, not including the time spent writing this post. It's easy to get things wrong in the benchmarking itself, and it's easy to get things wrong when reporting the results. Transparency, honest dealing, and a commitment to intellectual integrity (what Feynman called "a principle of scientific thought that corresponds to a kind of utter honesty--a kind of leaning over backwards") -- these things are key.

The secondary point of this post is to show that Solar is in fact more-responsive than Symfony 2 when they are treated alike, even under Fabien's test conditions.

Note that this benchmarking series uses the codebase for Fabien's Symfony 2 benchmarks; this is not part of the official web-framework-benchmarks series, as the tested conditions in Fabien's code are somehwat different.

These are the major portions of this post:

1. We run Fabien's benchmarks using Siege 2.66 on an Amazon ec2 "c1.xlarge" instance using his instructions and codebase. Since Fabien left out the static HTML baseline target, we will add it ourselves for comparison. We will find that his initial report is inaccurate; two frameworks are non-responsive, and the difference between Solar and Symfony is much less than reported.

2. We will attempt to run the same series using Siege 2.69. We will find that it fails because of socket unavailability.

3. We backtrack a bit and run the benchmarks on a "m1.large" instance, using Siege 2.66 to ascertain the original scenario.

4. We run the same series using Siege 2.69, and find the same relative performance ranking as with 2.66, but with lower percent-of-PHP numbers, because Siege 2.69 reports higher (and more believable) baseline numbers.

5. Finally, we show that Symfony 2 uses a preloaded classes file. When we do the same for Solar and re-run the benchmarks, we find that Solar is more responsive than Symfony 2 by roughly 28%.

Fabien's inital benchmark report is at http://symfony-reloaded.org/fast.

The code he benchmarked against is at http://github.com/fabpot/framework-benchs.

His instructions for reproducing his results are at http://github.com/fabpot/framework-benchs/blob/master/replicating.markdown.

For reference, here are the numbers Fabien initially reported (alphabetized by framework):

framework            |      rel |      avg |        1 |        2 |        3 |        4 |        5
-------------------- | -------- | -------- | -------- | -------- | -------- | -------- | --------
baseline-php         |   1.0000 |  5465.30 |  4602.06 |  5509.34 |  5694.15 |  6232.73 |  5288.23
cakephp-1.2.6        |   0.0513 |   280.43 |   255.91 |   279.50 |   291.80 |   291.13 |   283.83
flow3-1.0.0alpha7    |   0.0048 |    26.29 |    23.87 |    26.97 |    26.67 |    26.93 |    27.02
lithium-0.6          |   0.2128 |  1163.27 |  1059.44 |  1179.42 |  1180.52 |  1197.73 |  1199.25
solar-1.0.0beta3     |   0.2825 |  1544.14 |  1293.81 |  1596.28 |  1601.55 |  1613.20 |  1615.86
symfony-1.4.2        |   0.1737 |   949.59 |   916.84 |   944.49 |   953.88 |   967.52 |   965.24
symfony-2.0.0alpha1  |   0.3312 |  1810.07 |  1693.15 |  1846.41 |  1827.51 |  1856.98 |  1826.30
yii-1.1.1            |   0.1901 |  1038.77 |  1033.20 |  1037.60 |  1038.47 |  1041.57 |  1043.01
zend-1.10            |   0.0906 |   494.90 |   320.74 |   519.74 |   537.15 |   546.11 |   550.76

Finally, for those who wish to follow along, the scripts for each of the major sections of this post are available here: http://paul-m-jones/public/fabiens-benches.sh

Part 1

We set up a "c1.xlarge" instance per the instructions from Fabien, and run siege.php against the targets file.

Flow3 had an exception:

<h1>500 Internal Server Error</h1>
<p>FLOW3 experienced an internal error (uncaught exception):</p>
<p>PDOException</p>

Looks like PDO has to be loaded for Flow3.

Lithium had an error too:

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /lithium-0.6/app/webroot/hello/Fabien was not found on this server.</p>
<hr>
<address>Apache/2.2.9 (Debian) PHP/5.3.1-0.dotdeb.1 with Suhosin-Patch Server at localhost Port 80</address>
</body></html>

This is an Apache 404 error; the target line for Lithium appears to be wrong.

Therefore, we can ignore those two frameworks in our results. The report looks like this:

framework                |      rel |      avg |        1 |        2 |        3 |        4 |        5
------------------------ | -------- | -------- | -------- | -------- | -------- | -------- | --------
baseline-html            |   0.9002 |  5239.91 |  4438.38 |  5458.72 |  5822.69 |  5230.76 |  5249.02
baseline-php             |   1.0000 |  5821.08 |  4950.66 |  5763.70 |  5729.34 |  5939.63 |  6722.07
cakephp-1.2.6            |   0.1022 |   594.87 |   568.08 |   597.78 |   603.13 |   603.88 |   601.48
flow3-1.0.0alpha7 *      |   0.0008 |     4.40 |     3.41 |     4.55 |     4.64 |     4.72 |     4.70
lithium-0.6 *            |   1.0119 |  5890.55 |  5436.60 |  5326.39 |  6037.85 |  5996.23 |  6655.69
solar-1.0.0beta3         |   0.2441 |  1420.88 |  1323.49 |  1404.17 |  1416.07 |  1483.37 |  1477.30
symfony-1.4.2            |   0.0876 |   509.73 |   493.31 |   506.76 |   517.65 |   514.61 |   516.33
symfony-2.0.0alpha1      |   0.2573 |  1497.54 |  1242.21 |  1433.91 |  1607.85 |  1626.23 |  1577.50
yii-1.1.1                |   0.1360 |   791.69 |   790.61 |   787.22 |   795.12 |   793.43 |   792.08
zend-1.10                |   0.0769 |   447.62 |   382.17 |   448.55 |   473.18 |   471.33 |   462.88

(* ignore)

Contrary to Fabien's report, we see that Symfony 2 is not "20% faster" than Solar. Symfony 2 at .2573, and Solar at .2441, is more like a 5% difference, with Symfony 2 in the lead.

However, the baseline PHP response was 10% faster than a static HTML page (where the PHP engine is not invoked at all). This indicates something is wrong with the benchmarking environment. I saw similar behavior when using ab (the Apache benchmark tool) and it looks like Siege 2.66 has the same erroneous behavior. Let's switch to the more-recent Siege version 2.69 and see if we can eliminate that.

Part 2

We remove Siege 2.66, install Siege 2.69, and attempt to run the benchmarks again.

The problem is, even with the ulimit set as high as it is, Siege 2.69 floods the server and we get tons of "error: socket: 1168148816 address is unavailable.: Cannot assign requested address" errors.

We can set 'failures' => 1048576 in siege.php (i.e., the same as the ulimit value) to try and ignore the socket errors. However, even at that high value, we still can't get through the baseline html response; socket availability is still too low.

As such, we will step down to an "m1.large" instance for the remainder of the process. I know from previous experience that Siege will not exceed the socket availability on this kind of instance.

Part 3

We terminate the "c1.xlarge" instance and run an "m1.large" instance in its place. Now that we're on a new instance, we need to re-run Fabien's benchmark series using Siege 2.66 again to make sure the errors we received before are not instance-type specific. When we do, we get these results (Flow3 and Lithium show the same errors as before):

framework                |      rel |      avg |        1 |        2 |        3 |        4 |        5
------------------------ | -------- | -------- | -------- | -------- | -------- | -------- | --------
baseline-html            |   0.9658 |  2424.05 |  2430.61 |  2452.35 |  2357.62 |  2431.77 |  2447.89
baseline-php             |   1.0000 |  2509.98 |  2548.19 |  2517.85 |  2509.79 |  2439.35 |  2534.70
cakephp-1.2.6            |   0.0748 |   187.65 |   187.34 |   188.88 |   187.83 |   187.62 |   186.57
flow3-1.0.0alpha7 *      |   0.0004 |     1.07 |     1.12 |     1.02 |     1.04 |     1.11 |     1.06
lithium-0.6 *            |   1.0639 |  2670.45 |  2653.04 |  2714.18 |  2659.09 |  2662.56 |  2663.39
solar-1.0.0beta3         |   0.1944 |   487.88 |   486.83 |   478.97 |   493.04 |   489.55 |   491.03
symfony-1.4.2            |   0.0810 |   203.22 |   204.22 |   204.79 |   204.08 |   201.49 |   201.53
symfony-2.0.0alpha1      |   0.2330 |   584.73 |   582.53 |   578.17 |   589.57 |   588.38 |   584.98
yii-1.1.1                |   0.1463 |   367.16 |   362.83 |   373.34 |   362.67 |   377.57 |   359.41
zend-1.10                |   0.0542 |   135.99 |   135.06 |   135.28 |   137.09 |   136.67 |   135.87

(* ignore)

Now we see a difference in the Solar and Symfony 2 numbers that looks like Fabien's original reporting; Symfony 2 at 0.2330 is about 20% faster than Solar at 0.1944.

But we still see the same error condition of PHP looking like it runs faster than static HTML. Let's move away from Siege 2.66 and try Siege 2.69 on this smaller instance.

Part 4

We remove Siege 2.66, install Siege 2.69, and re-run the new Siege against the same targets on the same "m1.large" instance.

framework                |      rel |      avg |        1 |        2 |        3 |        4 |        5
------------------------ | -------- | -------- | -------- | -------- | -------- | -------- | --------
baseline-html            |   1.1710 |  5594.40 |  5732.63 |  5610.27 |  5769.64 |  5663.68 |  5195.80
baseline-php             |   1.0000 |  4777.65 |  4853.58 |  4729.50 |  4721.51 |  4772.37 |  4811.31
cakephp-1.2.6            |   0.0401 |   191.71 |   186.32 |   192.19 |   190.43 |   194.06 |   195.53
flow3-1.0.0alpha7 *      |   0.0000 |     0.00 |     0.00 |     0.00 |     0.00 |     0.00 |     0.00
lithium-0.6 *            |   1.0689 |  5107.01 |  5196.32 |  5158.93 |  4888.00 |  5092.78 |  5199.03
solar-1.0.0beta3         |   0.1126 |   537.85 |   541.37 |   537.44 |   537.34 |   536.59 |   536.52
symfony-1.4.2            |   0.0443 |   211.73 |   212.29 |   211.68 |   213.56 |   210.47 |   210.64
symfony-2.0.0alpha1      |   0.1370 |   654.65 |   655.15 |   654.62 |   653.44 |   651.52 |   658.52
yii-1.1.1                |   0.0819 |   391.44 |   396.89 |   389.75 |   390.60 |   396.30 |   383.68
zend-1.10                |   0.0293 |   140.19 |   139.83 |   138.85 |   141.18 |   140.87 |   140.24

(* ignore)

This looks more like what we should be seeing: HTML is now faster than PHP. The rankings and relative ratings appear similar to the Siege 2.66 run; Symfony 2 at .1370 is about 20% faster than Solar at .1126.

Part 5

I spent some time picking apart Symfony 2 to see what it might be doing that Solar could use for improvement. One reason for Symfony's performance is that (in the benchmarked code) all the Symfony 2 foundation classes are concatenated into a single "bootstrap.php" file. Similarly, Symfony 2 caches its application classes into another single file (hello/cache/prod/classes.php). From what I can tell, none of the other frameworks are doing anything like that; they are reading class files individually as needed.

Let's see if we can even out the Solar vs. Symfony 2 playing field. In this final benchmarking pass, we get the latest trunk code of Solar, compile the Solar foundation classes into a preload file just like Symfony's, and use that preload file in the Solar bootstrap. Then we'll target just Symfony 2, Solar beta 3 (non-preload), and the Solar trunk preload for comparison to each other. The results on the "m1.large" instance are:

framework                |      rel |      avg |        1 |        2 |        3 |        4 |        5
------------------------ | -------- | -------- | -------- | -------- | -------- | -------- | --------
baseline-html            |   1.1884 |  5611.92 |  5428.74 |  5700.62 |  5654.88 |  5622.77 |  5652.61
baseline-php             |   1.0000 |  4722.21 |  4646.16 |  4698.98 |  4780.01 |  4753.59 |  4732.29
solar-1.0.0beta3         |   0.1142 |   539.31 |   542.25 |   543.45 |   538.97 |   537.07 |   534.82
solar-preload            |   0.1780 |   840.38 |   841.38 |   846.63 |   838.32 |   833.13 |   842.44
symfony-2.0.0alpha1      |   0.1384 |   653.52 |   658.03 |   658.61 |   654.93 |   649.51 |   646.50

It appears that when we treat Solar and Symfony 2 the same way, by preloading the foundation classes, we find that Solar is about 28% faster than Symfony 2 (and about 55% faster than the non-preload Solar beta 3 with no code changes at all). Perhaps it would be wise for Solar to provide a something like a preload.php file of its own as part of the system distribution.

Conclusion

1. Fabien's benchmark report, as shown at http://symfony-reloaded.org/fast, is inaccurate for the setup he describes. Lithium and Flow3 do not work in Fabien's benchmark codebase at Github. Also, Symfony 2 is faster than Solar beta 3 by 5%, not 20%, on a "c1.xlarge" instance; to get a relative difference like Fabien describes, one has to use an "m1.large" instance. (It is entirely possible that the process Fabien used for benchmarking is incompletely described, and that the codebase is not fully updated, thus contributing to this disparity in results.)

2. We should use Siege 2.69, not 2.66, for more accurate benchmarking of baseline responsiveness. If we notice that HTML is slower than PHP, it's a sign that something is wrong.

3. Symfony 2 preloads its foundation and application classes, something no other framework does in the benchmarked code. When we treat Solar and Symfony 2 the same way, by preloading the foundation classes for each, we find that Solar is roughly 28% faster than Symfony 2.

Who is the worst killer in the long, ugly history of war and extermination? Hitler? Stalin? Pol Pot? Not even close. A single book called Silent Spring killed far more people than all those fiends put together.

...

The motivation behind Silent Spring, the suppression of nuclear power, the global-warming scam, and other outbreaks of environmentalist lunacy is the worship of centralized power and authority. The author, Rachel Carson, didn’t set out to kill sixty million people – she was a fanatical believer in the newly formed religion of radical environmentalism, whose body count comes from callousness, rather than blood thirst. The core belief of the environmental religion is the fundamental uncleanliness of human beings. All forms of human activity are bad for the environment…

via The Greenroom » Forum Archive » The Green Death.

In my regular calls with my dad, he mentioned that it seems Baby Boomers, when unhappy with their jobs, tend to do little but whine about it, whereas Gen-Xers tend to quit. I thought that observation meshed well with this insight from Peter Drucker ("The Essential Drucker", p. 314):

But increasingly also, knowledge workers, and especially people of advanced knowledge, see the organization as the tool for the accomplishment of their own purposes and, therefore, resent ... any attempt to subject them to the organization as a community, that is, to the control of the organization; to the demand of the organization that they commit themselves to lifetime membership and to the demand that they subordinate their own aspirations to the goals and values of the organization. This is inevitable because the possessor of knowledge ... owns his or her "tools of production" and has the freedom to move wherever opportunities for effectiveness, for accomplishment, and for advancement seem greatest.

Managers in IT organizations might do well to remember this.

Yesterday I released the beta4 version of the Solar Framework for PHP. You can see the change notes here.

We have also added three new chapters to the manual:

• Working with Models
• Working with Related Models
• Views and Layouts by Jon Elofson
• And some some appendices for more in-depth information about Solar's underpinnings.

Take Solar for a quick test-drive with the blog demo and see if it suits you.

(Cross-posted from the Solar blog.)

Many of those who had no problem with government control or power growing under President Clinton were horrified to see control or power growing under President George W. Bush. Yet, as vociferously as they objected to what they saw as abuse of power by Bush, many of these same people welcome growing government control and power under President Obama. And those who supported Bush as he expanded government, even domestically and even outside of areas related to national security, were then horrified at the growth of government under Obama.

It’s time to stop cheering for a team, to put down the political pom-poms and to allow the government, when it is your party in office, to only have as much power as you would be comfortable with it having if the other side were in power. Because at some point it will be the other side in power. And if you were cheering for one side to have that power, most likely you won’t be happy about how that power is used when the other side has it (even when both parties take us in the same direction).

via When Falls the Coliseum » Put down the political pom-poms.

I feel I’ve been had.

One thing I could not have known in 1996 was that the IPCC’s warming predictions would be wrong. Mean global surface temperatures have not risen since 1998, and, by some measures, have dropped since 2001. The CRU e-mails show scientists trying to hide this decline, to give one detail--I don’t have room in this column to detail the extent of CRU’s shenanigans, nor could I tell the story as well as others, so please read this “Editor’s Page” online for links (see below).

This doesn’t necessarily mean manmade global warming is disproven. But it does deflate the certainty and moral righteousness of the Al Gores and the IPCCs of the world.

via Editor's Page: May Cooler Heads Prevail - Honolulu Magazine - February 2010 - Hawaii.

The IRS is soliciting quotes for its purchase of 60 shotguns.

Maybe you ought to have one too. Via TaxProf Blog: IRS to Buy 60 Shotguns.

Yesterday, I released Solar 1.0.0beta3. You can see the very short change log here. Special thanks to “kalkin” for committing a series of fixes and improvements.

These days I’m concentrating more on documentation. Previously, we had only the ubiquitous quick-start blog demo. This release includes a new chapter on Solar’s dynamic dispatch cycle; i.e., the bootstrap, front controller, page controller, action method, and view. (I’m starting a chapter on the DataMapper-style model system today.)

Incidentally, I’ve been writing the docs in DocBook 5 and building them with xsltproc. You can see the whole build system in the Solar docs repository. I tried using the new PhD build system, but it required more effort and energy than I could spare. PhD is screamingly fast, but it’s not doing everything that xsltproc does just yet, and I barely have enough time to work on my own projects as it is.

(This announcement is cross-posted from the Solar framework blog).

Here’s an idea. Why not let people of who want to borrow money to buy a house convince the people who lend the money that there’s a good chance that the money will get paid back. Creative, no? It’s called voluntary exchange. Or a market. Or normal. Fannie and Freddie were abnormal. We don’t need a “new system.” The old system didn’t work because you, Barney Frank, and others, tried to steer it. Let’s have NO SYSTEM that you figure out and control and steer. Let’s let the housing market emerge that has the built-in feedback loops of profit and loss.

via Frank on Fannie and Freddie.

So our society has created a mess where men are vilified in the classroom, fed PC rhetoric, told their life's goal is to make women happy and do anything that assists her with her goals, while simultaneously told that he is a dope, idiot and unable to care properly for children, and now people are questioning where Mr. Right went?

via Dr. Helen: Mr. Right has left the building.....