Paul M. Jones

(Heavily condensed from the original, which you should read in its entirety.)

More Isn’t Always Better: Higher code coverage was not the best measure of post-release failures in the field. If 99 percent of the code has been tested, but the 1 percent that did not get tested is what customers use the most, then there is a clear mismatch between usage and testing. It is more beneficial to achieve higher code coverage of more complex code than to test less complex code at an equivalent level. Those are the kinds of tradeoffs that development managers need to keep in mind.

Write Test Code First: TDD teams produced code that was 60 to 90 percent better in terms of defect density than non-TDD teams. They also discovered that TDD teams took longer to complete their projects--15 to 35 percent longer. “Over a development cycle of 12 months, 35 percent is another four months, which is huge. However, the tradeoff is that you reduce post-release maintenance costs significantly, since code quality is so much better. Again, these are decisions that managers have to make--where should they take the hit?”

Proving the Utility of Assertions: More assertions and code verifications means fewer bugs. Looking behind the straight statistical evidence, they also found a contextual variable: experience. Software engineers who were able to make productive use of assertions in their code base tended to be well-trained and experienced, a factor that contributed to the end results. What kind of action should development managers take based on these findings? The research team believes that enforcing the use of assertions would not work well; rather, there needs to be a culture of using assertions in order to produce the desired results.

Organizational Structure Does Matter – a Lot: Organizational metrics, which are not related to the code, can predict software failure-proneness with a precision and recall of 85 percent. This is a significantly higher precision than traditional metrics such as churn, complexity, or coverage that have been used until now to predict failure-proneness.

Geographical Distance Doesn’t Matter – Much: Most people preferred to talk to someone from their own organization 4,000 miles away rather than someone only five doors down the hall but from a different organization. Organizational cohesiveness played a bigger role than geographical distance.

Source: Exploding Software-Engineering Myths - Microsoft Research

(Greatly condensed from the original, which you should read in its entirety; all emphasis in original.)

I like to think about media in terms of questions answered.

Like other media (the web) has a question that it answers better than any other. That question is:

Why wasn't I consulted?

“Why wasn't I consulted,” which I abbreviate as WWIC, is the fundamental question of the web. It is the rule from which other rules are derived. Humans have a fundamental need to be consulted, engaged, to exercise their knowledge (and thus power), and no other medium that came before has been able to tap into that as effectively.

WWIC is the thing people talk about when they talk about nicer-sounding things like “the wisdom of crowds” or “cognitive surplus.” It has become the first thing I think about when I think about the web. I start by asking: “How do we deal with the WWIC problem?” Everything else comes after.

The obvious example of WWIC at work is Wikipedia, created for free by unpaid labor. It tapped into the basic human need to be consulted and never looked back.

Once you see [thumbs-up/thumbs-down icons so that you can rank the comments], a website is complete. You're down to the bedrock. A boolean or integer value is the digital equivalent of a grunt. You can't get any more basic than a like, or a thumbs-up, or a favorite.

Source: The Web Is a Customer Service Medium (Ftrain.com)

One of the great things about PHP is its session handling capabilities. One call to session_start() and a huge amount of heavy lifting is done for you. It’s a great aid when writing page scripts.

However, as you start to need finer control over the HTTP response in your project, some of the automatic session behaviors begin to get in the way. In particular, when you are using PSR-7 to build your HTTP response, you realize that session_start() and session_regenerate_id() both automatically do the equivalent of calling setcookie() to write headers directly to the output. This means you cannot buffer those calls into the Response object for later sending.

How then can we use PHP’s session handling, when we want finer control over when and how cookies get sent?

The first trick is to tell PHP not to send a cookie when it does session work. This is accomplished with three ini_set() calls:

ini_set('session.use_trans_sid', false);
ini_set('session.use_cookies', false);
ini_set('session.use_only_cookies', true);

These direct PHP not to use transparent session IDs, not to use cookies, and (counterintuitively) to use only cookies. If I understand correctly, the combination of the last two means that PHP will read only from the cookies, and from nowhere else, to find the session ID value.

With those settings, a call to session_start() will cause PHP to read from the cookie values for the session ID, but it will not cause PHP to set any cookies for the session.

The second trick is to compare the session ID in the incoming request, to the session_id() value at the time you want to send the response. If they are different, that means a session has been started or regenerated, at which point you can send the session cookie manually. The following is an example Relay-compatible middleware that puts session cookie handling logic into effect:

SessionHeadersHandler.php

When you examine the class, note that the cookie-creation code is intended to be the same as in the PHP session handling code itself. Note also that you can extract the relevant logic (“compare the Request session ID to the current one, and send a cookie if they’re different”) and use it in a non-middleware-based application.

With SessionHeadersHandler in place, subsequent middleware decorators can call session_start() and session_regenerate_id(), and PHP will no longer automatically write out a session cookie on its own. The handler will set the cookie into a PSR-7 Response object for later sending.

Unfortunately, this is only a partial solution for session headers. The handler does not deal with things like session cache expire and limiter headers. However, it does give you control over when session cookie itself get sent, and that’s a great aid when you want to work with PSR-7 Response objects.

Read the Reddit discussions about this post here and here.

What we are seeing worldwide, from India to the UK to the US, is the rebellion against the inner circle of no-skin-in-the-game policymaking "clerks" and journalists-insiders, that class of paternalistic semi-intellectual experts with some Ivy league, Oxford-Cambridge, or similar label-driven education who are telling the rest of us 1) what to do, 2) what to eat, 3) how to speak, 4) how to think... and 5) who to vote for.

With psychology papers replicating less than 40%, dietary advice reversing after 30y of fatphobia, macroeconomic analysis working worse than astrology, microeconomic papers wrong 40% of the time, the appointment of Bernanke who was less than clueless of the risks, and pharmaceutical trials replicating only 1/5th of the time, people are perfectly entitled to rely on their own ancestral instinct and listen to their grandmothers with a better track record than these policymaking goons.

Indeed one can see that these academico-bureaucrats wanting to run our lives aren't even rigorous, whether in medical statistics or policymaking. I have shown that most of what Cass-Sunstein-Richard Thaler types call "rational" or "irrational" comes from misunderstanding of probability theory.

Source: Nassim Nicholas Taleb

The victim narrative of the Left is very infectious. You are always the victim and you are always owed something. The wealthy are always evil, while you are always good and wholesome. Converts are often more intense than those born into it. My father, raised a leftist, eventually mellowed and began to question some leftist beliefs. My mother, not raised a leftist, but having become one, never mellowed.

The victim narrative was in every conversation.

The class struggle/oppressed victim narrative is part of daily life on the Left. As a child, I would listen to adults talking. With friends and co-workers, with mothers chatting over tea, it was part of every conversation. They would talk about the weather, their kids, television, but before parting, one of them would always say something relating to the greedy oppression of the rich -- and the other had to agree. To not agree was social suicide.

While there were differences between working-class and middle-class leftists, certain attitudes were universal:When a leftist has never worked, they feel very generous toward anyone who claims to need help, who fits the narrative. They are generous with their emotions.

Emphasis in original. Source: The Mind of the Left From an Insider | Frontpage Mag

Muhammad's message featured violence with increasing intensity, culminating in surah 9, chronologically the last major chapter of the Quran, and its most expansively violent teaching. Throughout history, Muslim theologians have understood and taught this progression, that the message of the Quran culminates in its ninth chapter.

Surah 9 is a command to disavow all treaties with polytheists and to subjugate Jews and Christians (9.29) so that Islam may “prevail over all religions” (9.33). It is fair to wonder whether any non-Muslims in the world are immune from being attacked, subdued or assimilated under this command. Muslims must fight, according to this final chapter of the Quran, and if they do not, then their faith is called into question and they are counted among the hypocrites (9.44-45). If they do fight, they are promised one of two rewards, either spoils of war or heaven through martyrdom. Allah has made a bargain with the mujahid who obeys: Kill or be killed in battle, and paradise awaits (9.111).

Muslim thought leaders agree that the Quran promotes such violence. Maajid Nawaz, co-founder of the Quilliam Foundation in the United Kingdom, has said, “We Muslims must admit there are challenging Koranic passages that require reinterpretation today. ... Only by rejecting vacuous literalism are we able to condemn, in principle, ISIS-style slavery, beheading, lashing, amputation & other medieval practices forever (all of which are in the Quran). … Reformers either win, and get religion-neutral politics, or lose, and get ISIL-style theocracy.” In other words, Muslims must depart from the literal reading of the Quran in order to create a jihad-free Islamic world.

This is not at all to say that most Muslims are violent. The vast majority of Muslims do not live their lives based on chapter 9 of the Quran or on the books of jihad in the hadith. My point is not to question the faith of such Muslims nor to imply that radical Muslims are the true Muslims. Rather, I simply want to make clear that while ISIL may lure youth through a variety of methods, it radicalizes them primarily by urging them to follow the literal teachings of the Quran and the hadith, interpreted consistently and in light of the violent trajectory of early Islam. As long as the Islamic world focuses on its foundational texts, we will continue to see violent jihadi movements.

Source: The Quran's deadly role in inspiring Belgian slaughter: Column

tl;dr: Producer will look over your Composer-based library package just before you are ready to tag it for release, make sure it appears ready-to-go, and then do the release for you through Github, Gitlab, or Bitbucket. Producer works with both Git and Mercurial.

I. History

Back when I was working on Solar, we needed a process to package up each release of the entire framework and make sure I hadn’t forgotten anything. Thanks to the magic of the Internet Archive, you can see it here. You can read more about the 10 year old (!!!) process here; the script is of course PEAR-centric, since PEAR was the main packaging system available in PHP at that time.

After Solar was done, we began extracting its individual components as 30 or so separate packages in Aura. As before, I needed a process to make sure each package release was actually ready-to-go, so that earlier PEAR-centric release script evolved into a collection of Composer-centric commands. These release and management tools are specifically for Aura, with its particular conventions and expectations in mind, and have served well for 3 major versions of the project over the last 5+ years.

But now I have started some non-Aura projects: Relay, Radar, Arbiter, Bookdown, and most recently Atlas. These projects do not have the benefit of the automated release process, with all of its checks and validation, that Aura does.

With that in mind, then, I have extracted a substantial amount of the Aura package release process into a new project, Producer, so that I can use it with any non-Aura library package. That means you can use it with your library package, too.

II. Why Use Producer?

When you think your Git or Mercurial library package repository is ready for a tagged version release, Producer help to validate that it is actually in a high-quality releasable state. Then, if Producer thinks everything looks good, it will release your library package through its remote origin API (i.e., through Github, Gitlab, or Bitbucket).

(Note that Producer is not for regular daily development work. It is specifically for the day you want to release the package. At that time, it can be easy to forget steps in a release process; Producer manages those steps for you.)

III. Validating A Package For Release

Most of the things Producer checks for are there because I forgot to check them myself at some point in the past, and it was embarassing for one reason or another.

For example, you don’t want to make a release from a local copy that has not been updated to match the remote copy, or when the local copy has some modified or uncommitted files. So the very first thing Producer does is to pull down changes from the remote origin, push up local changes to the remote, and then check the local status to see if there are uncommitted files. If the local status check fails, Producer won’t release the package.

After it’s sure the local copy is in a clean state, Producer will run composer validate to make sure it has no obvious errors. Obviously, if composer.json is not valid, the package is not in a releasable state.

Next, Producer looks to see if you have a particular set of non-empty informational files in place. These are administrative, but necessary: README, LICENSE, CONTRIBUTING, and CHANGES (not CHANGELOG yet; more on that later). The files may or may not have .md extensions. These files ought to be present in any packaged release, so Producer will fail if they are not present, or if they are empty.

Of course, Producer can’t tell if their contents are sensible or not, though it can tell if the LICENSE notes the current year. If it does not, Producer will tell you to update the copyright year in the LICENSE. You don’t want to release a package with an outdated copyright year!

After that, Producer will run your test suite with PHPUnit. Producer expects that you have a phpunit.xml.dist file at the root of your package, so if that’s missing, the release will fail. Likewise, if the tests fail, Producer will not release the package.

As part of the test-running, Producer will issue composer update to make sure all the require-dev packages are in place. After the tests pass, Producer will check the local copy status yet again, to make sure the tests have cleaned up after themselves properly. This also has the benefit of checking that your “ignore” files are set up properly; at the very least, composer.lock and vendor/ should be ignored, so their presence in the status check will cause Producer to stop its release process.

We want to have well-documented code, so Producer runs PHPDocumentor to check all the docblocks in the package src/ directory. (Yes, this presumes that you have a src/ directory in your library package.) If any PHP docblocks are missing or malformed, Producer will stop the release. Similarly, if the @package tags are missing or incorrect, Producer will tell you which files and stop the release. (In this case, “correct” means “the same as the Composer package name”.)

As the next-to-last step in the validation process, Producer examines your CHANGES file commit timestamp. If the CHANGES file is not part of the very last commit, Producer will balk, and tell you to update your CHANGES so that all changes have been mentioned in the release notes. (Producer uses CHANGES because that’s what I use in Aura; at some point in the future, this may become CHANGELOG, but for now just the release-specific change listing seems enough.)

Finally, Producer will go to your remote origin API and retrieve a list of open issues for the repository. This is not technically a validation step, only a reminder in case there issues you have forgotten to address. If there are open issues, Producer will not stop the release process.

IV. Releasing A Package

The above validations can be run on their own using the producer validate command; they are useful as a pre-flight or pre-check to the actual release process. When the validate command finishes successfully, you can move on to the release command; the release process is exactly the same as the validation process, with the added step of actually releasing the package when validation passes.

At this point, Producer has already looked at your .git or .hg configuration to determine the remote origin API; this can be Github, Gitlab, or Bitbucket. It then prepares and sends a release through that API, using the CHANGES file for the release notes.

Afterwards, it will sync the local copy with the remote origin to pull down any newly-created tags.

That’s it; you’re done! Producer has now run a series of “final checks” on your repository and released it with a new version number.

V. Questions

When I announced Producer over the weekend, it got posted to Reddit by somone other than me (for once ;-). I hope I have answered the “why is Producer useful?” question with this blog post. Two others remain:

1. Q: “Why does Producer run composer update as part of validation? It should only look at the current state of the repo, not modify the repo.”

   A: Perhaps “validation” is not the right word to use. It’s intended as a pre-flight or preparatory step towards releasing, to make sure that the package will actually install what it requires through Composer. In addition, for my own projects at least, the tests use the Composer autoloader for the src/ files, so composer update is a necessary preliminary to running the tests.

2. Q: “Why require PHPDocumentor and PHPUnit as part of Producer? What if I have those already installed somewhere else?”

   A: For myself, global installs of these kinds of ecosystem-level tools seem reasonable, but that may be a function of the fact that I mostly manage library packages; having 30+ installs of PHPUnit and PHPDocumentor is just not to my liking. Having said that, I can see how some folks would have different requirements, so I’ll see if I can modify that requirement in a future version of Producer.

Finally, if you have questions/comments/critique, please raise them as issues over at Github. Thanks, and I hope Producer is as useful to you as its earlier versions in Aura have been to me!

Read the Reddit discussion about this post here.

I want the clever, hard-working children of those in the bottom half of income distribution to move up, and the less able children of those in the top half to move down.

In other words, I think the answer is more meritocracy. I approve of the principle for the same reason my father disapproved of it, because it helps to secure people’s consent to the inequalities that are the inevitable consequence of limited government. It does this by (a) allocating wealth and prestige in a way that appears to be fair; and (b) creating opportunities for those born on the wrong side of the tracks, so if you start with very little that doesn’t mean you’ll end up with very little, or that your children will. If you think a free society is preferable to one dominated by the state, and the unequal distribution of wealth is an inevitable consequence of reining in state power, then you should embrace the principle of meritocracy for making limited government sustainable.

This sentence also appears: "If the history of the twentieth century teaches us anything, it is that the dream of creating a socialist utopia often leads to the suppression of free speech, the imprisonment of a significant percentage of the population and, in some extreme cases, state-organised mass murder."

Source: The Fall of the Meritocracy -- Quadrant Online

I’ve pointed out before that Model-View-Controller is a user interface pattern, not an application architecture. But why would PHP developers get the idea that MVC is an application architecture in the first place? (This may apply to all server-side developers, not just PHP folks.)

I used to think that MVC was an application architecture. Even after reading Fowler’s POEAA and seeing that MVC was for the user interface, I figured that meant I was doing “user interface applications.” But that was not quite right; it would have been more accurate to say that I had been mixing the concerns of user interface with the underlying core application.

Based on that experience, I think the reason MVC is mistaken for an application architecture is because PHP developers generally start with page scripts. In our first page scripts, we combine all the concerns in a single ball of mud: SQL queries are intermingled with HTML, and the business logic is scattered throughout. As far as we are concerned, the page script itself is “the application.” Later, as more functionality is required, we add another page script, and another, and another. We continue to see that collection of page scripts as “the application.”

Then, as we progress in our profession, and begin to learn how to better organize our work, we start separating the different concerns of “the application.” When we begin separating concerns, we separate them out of the page script, which we see as “the application.” Extracting the view concerns from the page script means extracting them from “the application.” Separating the model from the page script into its own layer means separating it from “the application.” Pulling the controller logic out of the page script means pulling it out of “the application.” So of course Model-View-Controller is seen as an application architecture – we separated the concerns of our application according to that pattern, right?

Except, in retrospect, it’s not. One of the big leaps we have to make is to realize that MVC is for the user interface portion of our systems, just like Fowler notes. We on the server side think the user interface is HTML, CSS, and Javascript, but it’s not. Instead, the user interface is the HTTP Request and Response. In other words, the template is not the View.

Once we make that conceptual leap, we begin to realize that the Model layer is the entry point to “the application”. That is where “the application” should live, or at least where it should look like it lives. The Controller should have little to do but take input from the Request and pass it to the Model; likewise, the View should do nothing but put together the Response using the output from the Model.

With that idea of server-side MVC, we then begin to see that a lot of what’s in server-side MVC frameworks is “too much.” Framework functionality that is not related to merely taking input from a Request and presenting output through a Response becomes entirely secondary to, perhaps even actively harmful to, building well-structured applications – applications that are independent of any particular user interface.

Afterword

The server-side MVC pattern, as descended through Sun’s Model 2 architecture, is so distinctive from the client-side MVC pattern as to be a completely different thing. I realized this as I was writing my book on modernizing legacy applications, and further research led me to write up Action-Domain-Responder as a web-specific alternative to MVC. Action-Domain-Responder places a much stronger emphasis on the HTTP Request and Response as the user interface elements. If you are interested in building better applications, you may wish to read the ADR essay, and try it out in your next project.

Read the Reddit discussions about this post here and here.

Some quick notes from the past few days about various projects:

• The core Action-Domain-Responder implementation package for Radar has had its first beta release. (I’ll be able to release a stable version when the various pacakges it requires go stable as well, Aura.Di 3.x being the most relevant.)

• The core implementation package of Bookdown, the DocBook-like documentation generator that uses Markdown and JSON instead of XML, has had its first beta release as well.

• We released Aura.Filter 2.1.0 with new isNotBlank() functionality. Have I mentioned that it has what is probably the most thorough email validation in PHP-land? (And really good UTF-8 support as well.)

• We’re starting on the 3.x version of Aura.Sql, which provides a bunch of convenience functionality around PDO. This new version starts off by providing yield*()generator methods; you can use those instead of fetch*() to conserve memory when working with larger data sets.

Finally, I have added a ton of content and resource links to this blog:

• under appearances you will find lots of podcasts I’ve been invited to;

• code lists all my current and past projects;

• community lists some of my community work;

• talks reveals almost every presentation I’ve ever given, with slides and recordings when available;

• and writing collects my various technical publications into one place.

Enjoy!

Read the Reddit discussions about this post here and here.