Paul M. Jones

When you even contemplate bringing an old legacy system into a large-scale web project, you should do load testing on that system as part of the feasibility process before you ever write a line of production code, because if those old servers can’t handle the load, your whole project is dead in the water if you are forced to rely on them. There are no easy fixes for the fact that a 30 year old mainframe can not handle thousands of simultaneous queries. And upgrading all the back-end systems is a bigger job than the web site itself. Some of those systems are still there because attempts to upgrade them failed in the past. Too much legacy software, too many other co-reliant systems, etc. So if they aren’t going to handle the job, you need a completely different design for your public portal.

A lot of focus has been on the front-end code, because that’s the code that we can inspect, and it’s the code that lots of amateur web programmers are familiar with, so everyone’s got an opinion. And sure, it’s horribly written in many places. But in systems like this the problems that keep you up at night are almost always in the back-end integration.

The root problem was horrific management. The end result is a system built incorrectly and shipped without doing the kind of testing that sound engineering practices call for. These aren’t ‘mistakes’, they are the result of gross negligence, ignorance, and the violation of engineering best practices at just about every step of the way.

The title is adapted from Peter Drucker, "All business failures are management failures." The quoted piece is from a comment that I can't link to directly, so go to Arnold Kling on the problems with the health insurance exchanges and look for Dan Hanson on October 25, 2013 at 2:13pm.

The blog post Is ORM Abstraction A Pipe Dream? is one of the best technical articles I’ve read in maybe a year. Not because it describes a successful technical solution, but because it describes an attempt and a failure to achieve a goal. It reminds me of this quote from Feynman in Cargo Cult Science:

If you’ve made up your mind to test a theory, or you want to explain some
idea, you should always decide to publish it whichever way it comes out.
If we only publish results of a certain kind, we can make the argument
look good. We must publish both kinds of results.

(Cargo Cult Science is one of those essays that everyone should read, especially programmers who think of themselves as scientists or science-minded.)

What Feynman is saying here is that, when it comes to expanding a body of knowledge, the failures are just as important as the successes, perhaps more so in some cases. (Be careful here; they have to be “honest” failures, where you had some reason to believe in advance that it had a good chance of working.)

So what is it about the “Pipe Dream” article that impressed me? It is that the the author first signals his tribe membership by mentioning his “framework of choice”, then proceeds to try to do some work outside of that tribe. In doing so, he is expanding his knowledge and skills so that he can be useful outside the norms of that particular tribe. He realizes that there are other ways to do things than the way his tribe does them, he does some work with those things, he tries to unify them, and he fails at the unification. In the end, he has no technical artifact to show for his exploratory work, but he has now expanded his skills as a programmer beyond his “framework of choice.”

Your Framework Will Fail You

And being able to operate outside a framework is really important, because the framework you choose today, even if it’s the self-described “greatest PHP framework of all time”, is going to fail you in some non-trivial way in 3 years or so, even if it has a “long-term support” version. (After all, how long is long-term in the web world?)

Take a look at some points from this article by Yannick Mahe titled The consequences of living with a legacy PHP framework (note that “legacy” in this case is only 4-5 years):

You can no longer rely on the community. …

Documentation can be hard to find now. …

I can effectively no longer use plugins. …

[This framework] used Prototype (scriptaculous) as its Javascript framework … UX advances which could require just a simple jQuery plugin have to be written from scratch. …

This framework is not compatible with newer versions of PHP. It was written for PHP 5.2, and is compatible with PHP 5.3 but no longer works on PHP 5.4. …

Basically I’m one third less productive when programming than I could be.

Is that a slam against the framework or its defenders? No more than saying “the sky is blue” is a slam against the physics of light diffusion in atmosphere. “The framework is going to fail you” is a truism regardless of the framework.

And everybody uses a framework, whether they recognize it or not. I have said at other times that a “framework” can be defined as “that collection of tools and techniques you reuse on a regular basis from project to project”. So everyone uses a framework one way or another, whether formal or informal, well-architected or not, designed with intent or evolved through use.

How To Handle Failure

So if all frameworks are going to fail you, and you can’t avoid using one, what are your options?

All too often, software is like a complex system in that requires a constant input of energy, and complex systems fail. So the key point here is not “the framework is going to fail you.” The key point is “how easy will it be to handle that failure when it happens?”

A developer twin of Nassim Taleb might argue that whatever framework you have needs to be antifragile. That is, when one subsystem fails or becomes obsolete, it should not render the rest of the system unusable. Indeed, the failure of a subsystem should lead to a replacement that makes the system stronger in the future.

Frameworks in general are especially susceptible to their subsystem failures: they are usually developed as a thing of whole cloth (yes, even the ones that advertise they are “component based” — the components might be OK, but the framework often is not). The different parts depend on each other, and if one fails, you need to dive into the guts of the framework proper to fix it in place, which frequently leaves you worse off than before when it comes to maintainability.

Instead, for a robust or antifragile system, you should be able to swap out the different subsystems when they no longer meet your needs. As each piece fails (and each one will eventually fail to meet your needs), you’ll swap out each of the pieces at different times. Eventually the system will be composed entirely of replacements, but the whole thing will have kept running the whole time.

And that’s where we come back to the “Pipe Dream” article. By taking some steps outside his “framework of choice” the author is beginning to get an idea of how to integrate independent and disparate systems into a whole. In the long run, that ability is going to serve him better as a programmer than his knowledge of any particular framework.

Afterword: Aura for PHP

Of course, I’m going to mention Aura here. The ideals illuminated by article are the ideals that drive the development of the Aura, even to the point of prejudicially declining certain feature requests.

The Aura project is a collection of truly independent and fully decoupled libraries. None of the libraries has any external dependencies. The project is “library-first” and not “framework-first”, even though we do offer a framework built of the combined packages. The libraries evolve independently of the framework and can be incorporated individually or in any combination you like into any project.

If you liked the ideas behind this article, then the Aura project is for you. Download a single package and start using it in your project today, with no added dependencies.

UPDATE (30 Oct 2013): The title on the final section has been prefixed with the indicator "Afterword" since at least one person mistook it for a summary.

... it turns out that merely extracting the SQL parts was not good enough for some of our audience. They asked, “Can we download just the SQL connection classes, without the query objects or the mapper?” As with Solar, we had to answer, “No; although the Aura.Sql package is independent from every other Aura package, the various SQL tools all have be taken together as a whole.”

As a result, we have done even more decoupling with the Aura.Sql-v2 packages. We have managed to make these packages independent from each other:

Aura.Sql-v2 is composed primarily of an extended PDO implementation, along with a connection manager and a bare-bones profiler. Its only dependency is the native PDO; there are no userland dependencies. As an added bonus, we have taken pains to make it PHP 5.3 compatible.

Aura.Sql_Query is a package of SELECT, INSERT, UPDATE, and DELETE query objects for MySQL, PostgreSQL, SQLite, and Microsoft SQL Server. It has no dependencies at all, not even on PDO. You build your query with the object, then pass it along to the database abstraction layer of your choice. Because it uses traits, it is for PHP 5.4 and up.

Aura.Sql_Schema comprises “list tables” and “list columns” functionality for MySQL, PostgreSQL, SQLite, and Microsoft. As with Aura.Sql-v2, its only dependency is the native PDO (note, not the Aura.Sql-v2 ExtendedPdo). Also as with Aura.Sql-v2, it is PHP 5.3 compatible.

With this dedicated effort at decoupling and true indpedendence, you can now use just the extended PDO connection object, or the query objects, or the schema discovery tools, without having to download the others. Of course, you can use them all in concert if you like.

via Aura for PHP.

From the book Lost Moon, retitled Apollo 13 after the movie was made:

Apollo was downright dangerous. Earlier in the development and testing of the craft, the nozzle of the ship’s giant engine…shattered like a teacup when engineers tried to fire it. During a splashdown test, the heat shield of the craft had split open, causing the command module to sink like a $35 million anvil to the bottom of a factory test pool. The environmental control system had already logged 200 individual failures the spacecraft as a whole had accumulated roughly 20,000.

In January 1967, one of the first Apollo spacecraft caught fire during an on-the-ground test, killing astronauts Gus Grissom, Ed White, and Roger Chaffee. At that point, NASA decided that quality was more important than schedule and they overhauled the Apollo project (although they still managed a moon landing 2-1/2 years later).

Far too many software projects are like that. It usually takes a serious failure, one where blame cannot be shifted, for managers to realize that their schedule is unrealistic. Via Quality, Features, and Schedule | askblog.

From an intellectual perspective gay men allow the rest of us a “control group” of male sexuality, containing within a wide range of masculine expressions, one which refutes or repudiates a lot of the feminist sexual ideology.  After all, you can’t say “society makes straight men do this” without checking to see if society (which has mostly ignored gay men, culturally speaking, except as two-dimensional stock characters to be trotted out for laughs) also makes gay men do it, too.

Case in point: the oft-touted feminist maxim that “men only go after hot young girls because our rape-culture tells them that’s the ideal they should be shooting for.”  Feminism has always taken issue with the masculine preference for youth – and I think we all know why – but blamed it squarely on “the Patriarchy’s” efforts to take power away from older women.  If it wasn’t for our screwed-up youth-worshiping culture, the feminist myth goes, men would get just as hard over saggy tits and cottage cheese thighs as they do perky tits and a tight ass.

But if “the Patriarchy” is the one dictating what men should and shouldn't be attracted to, culturally speaking, then one must also assume that a group like Gay Men, who “the Patriarchy” has traditionally had an antipathy toward, would therefore not be subject to the same “artificial rules” that straight men are.

But it turns out gay dudes like young stuff, too.  A lot.

While the post might not be safe for work, depending on where you work, it is still very much worth reading in its entirety. Via The Red Pill Room: The Lambda Factor.

I don’t hold it against the contractors that they had prior government experience. I don’t hold it against them that they lobby or contribute to campaigns.

To me, the scandal is that there are 47 different organizations involved in building the site. I cannot imagine that any sane project executive would want it that way. I am just guessing, but it seems more likely to me that this many contractors were imposed on the project executive because there was a requirement to “spread the work out” to keep all these companies in the politicians’ pockets.

In any case, if you are trying to fix something that was assembled by 47 different organizations….good luck with that.

Remember this story any time you think government actors (at any level!) are somehow immune to bad management practices and skewed incentives. They are no smarter, and in aggregate often dumber, than non-government actors. Via Pinpoint the Scandal | askblog.

A while ago I tweeted

ContainerAware is the new Singleton.

While many people agreed by retweeting and faving. I feel the need to elaborate some more on this statement and safe the explaination for the future.

TL;DR: No class of your application (except for factories) should know about the Dependency Injection Container (DIC).

The ContainerAware interface (actually ContainerAwareInterface, ContainerAware is a basic implementation of it) is part of the Symfony2 API, but a similar concept is known from many other frameworks and many applications rely on it. It defines only the one method setContainer(), which allows to inject the DIC into into an object so that it can directly retrieve services from it.

I wouldn't call it "the new Singleton", I'd call it "Service Location." If you use a dependency injection container inside your class to bring dependencies into the object, you are using Service Locator, not Dependency Injection. Solar used SL, but when we started Aura (a collection of truly decoupled library packages, each with no dependencies) we switched over to DI proper. Real DI has been a huge win.

Via ContainerAware Considered Harmful - Qafoo GmbH - passion for software quality.

What the heck could be going on? My friend stated the obvious: “It's clear that they're getting more traffic than they can handle. The question is why they can't handle the traffic they're getting.” Load problems could explain servers hanging in California and New York … but the drop-downs? The standard explanation for this is “high load,” but high server loads don’t cause your security dropboxes to empty out.

“The drop-down thing is mystifying,” he told me. If federal exchanges decided to populate the security question fields by calling up a list of possible questions from another server -- one that didn’t have a lot of capacity -- then that might be causing the sign-up process to stall at that step. For an application that expects a lot of traffic, this is a very bad idea.

“Just cache them on the front ends, for heaven's sake, so you only need to ask once,” he said. “A database call to get questions shouldn't be in the critical serving path. If you're hitting the database just to load the security questions, then just serving individual pages is going to be expensive.”

The various glitches, he pointed out, “could very easily be because deadline pressure caused them to take some shortcuts that impacted their ability to scale.”

Such as?

“The aforementioned let's-hit-the-database-for-security-questions thing.”

Why would they use such a seemingly obvious poor design?

“It can be easier to make a call to another server to get something when you need it than to implement a cache that you prepopulate either from static files or from the database on startup. Making a call to another server is also something you'd naturally think to do if you hadn't had to focus on scalability before. The security question page is probably not the thing you're most concerned about, so you give it to the new hire to do as their starter project. They don't know what they're doing, so they implement it the straightforward way … and since you're under unbelievable deadline pressure to get something working now nobody reviews it in detail.”

Obviously, we don’t know if this theory is correct -- but it does fit the particulars.

Government programmers are subject to the same development pressures as the rest of us. Via Untangling Obamacare's Web Glitches - Bloomberg.

“Libraries first” is the way to go; it imposes good discipline

Extracting is detailed work, but you get testable units

DependencyInjection is awesome …

… but in-library ServiceLocators can be useful

There are still lots of PHP 5.3 users

If you have server or session vars in your HTTP request object, it’s not an HTTP request object

Some packages are still too broad in scope

From PEAR to Composer

People love ORMs

New packages: Aura.Dispatcher, Aura.Includer, Aura.Sql_Query, and others

PHP 5.3 support in some new packages

via Aura for PHP.

The Aura “system” package, which combines a system skeleton and the Aura libraries into a full-stack framework, was released today at 1.0.0 stable. This makes it formally ready for production use (although it has been in production use informally for the past year or so).

You can now start an Aura project with a single Composer command:

composer create-project aura/system /path/to/your/project

Take a look at the installation instructions for more information.

It feels great to have both stable, decoupled, truly independent package libraries and a system that integrates them into a framework. Many, many thanks to everyone involved who helped get us to this point.

Aura for PHP.