Stubs Aren't Mocks; BDD Isn't TDD; Which Side(s) Are You On?

I just finished re-reading Martin Fowler's Mocks Aren't Stubs from 2007. I wasn't as experienced then in the various forms of agile development as I am now, so couldn't quite appreciate his perspective until somebody (and I'm sorry I can't find whom) brought up the paper again in a tweet a month or two ago. (Yes, that's how far behind I am; how do you do when you're working 15- to 18-hour days, 6 or 7 days a week for 6 months?)

In particular, the distinctions he draws between "classical" and "mockist" test-driven development (TDD), and then between mockist TDD and behaviour-driven development (BDD) are particularly useful given the successes and challenges of the last dozen or so projects I've been involved with. I wouldn't quite say that many teams are doing it wrong. They/we have been, however, operating on intuition, local folklore and nebulously-understood principles gained through trial-and-error experience. Having a systematic, non-evangelistic, nuts-and-bolts differentiation and exploration of various techniques and processes is (and should be) a basic building block in any practitioner's understanding of his craft.

Put (perhaps too simply), the major distinction between classic and mockist TDD is that one focuses on state while the other focuses on specific, per-entity function; projects that mix the two too freely often come to grief. I believe that projects, especially midsize, greenfield development projects by small or inexperienced teams should pick one approach (classic or mockist TDD, or BDD) and stick with it throughout a single major-release cycle. You may credibly say "we made the wrong choice for this product" after getting an initial, complete version out the door, and you should be able to switch the next full release cycle to a different approach. But if you don't know why you're doing what you're doing, and what the coarse- and fine-grained alternatives are to your current approach, you can't benefit from having made a conscious, rational decision and your project thus can't benefit from that choice.

Anything that gives your team better understanding of what you're doing, why and how will enhance the likelihood of successfully delivering your project and delighting, or at least satisfying, your customers. Even on a hobby project where your customer is…you yourself. Because, after all, your time is worth something to you, isn't it?

It's Not Amtrak: Enjoy Riding the Rails

Historical note: Most of this post was written in the first week of October, 2011.

---

I've spent much of the last three weeks wrapping my head around Rails and Ruby after several years of mostly PHP with a couple of Groovy/Java side trips. It's been interesting, and a lot of fun so far.

I'm new to Ruby on Rails, but not to Ruby itself — though I took a long breather when Rails first became trendy. It was not at all clear to me at the time whether the (then-)popular trend among both the media and technology-involved people who Should Know Better to treat Ruby as essentially a synonym for "Ruby on Rails" was going to kill the language for other uses. It was quite clear to me that a huge chunk of early Rails culture was built around a cult-of-personality dedicated to Rails' primary creator, David Heinemeier Hansson. I try very hard to avoid cults; they almost invariably either spectacularly self-destruct (see Jonestown, Guyana) or get co-opted to serve unrelated or even opposing purposes (see State religions).

In about the last year or two, Rails has visibly grown up, both in capability and in outlook. There are other people who've stepped up and delivered great, widely-used code (does anybody know what happened to _why?), and Rails is now used in lots of systems both interesting and serious. I'd been thinking that maybe I ought to start getting myself up to scratch, particularly as I've been increasingly disenchanted with both the technical and sociopolitical aspects of PHP. The feelings of excitement, progress and everybody-pitch-in community that sustained PHP through its phenomenal growth in the last decade are, at best, flickering.

The final kick in the shorts was an interview cycle for new work with a great group that does, among other things, quite a bit of Rails work. Thus motivated, I grabbed copies of Agile Web Development with Rails, Fourth Edition and The RSpec Book and started going through them.

And promptly went out in search of better books. As of April, 2012, the ones I refer to most often are:

• Russ Olsen, Eloquent Ruby. Addison-Wesley Professional, 2011. Nearly 400 excellent pages that will help your Ruby code be more effective, expressive, pleasant to work with and easier to maintain; pick any five.
• Obie Fernandez, The Rails 3 Way; 2nd Edition. Addison-Wesley Professional, 2011. An encyclopedic yet guided introduction and continually-usable reference to Rails 3. Does not cover everything in 3.1/3.2, but will leave you comfortably able to adapt to the new changes.

One of the things I appreciate about Rails is the size and openness of the communities that have grown up around it. Everywhere I look, I find new code and tools to do useful things. I'm still on the upward slope of the "each Neat New Thing points to n other Neat New Things which point to…" curve, and I expect to be for quite some time. There are useful, useable if not always perfectly ideal tools to do things, and those tend to get adopted by (large chunks of) the community as de facto working standards. The whole "gem" ecosystem is chalk-and-Friday different (and dramatically better) than its PHP analogue; it's almost as though the "gems" folks looked at nearly everything PEAR gets wrong and made the conscious, deliberate decision to do the opposite. From the perspective of a newbie who's had painful experience with other systems, it Just (Seems Like It) Works.

Now, I'm sure that the rose-coloured spectacles will eventually pop out of their frames and get ground into dust. I'll find things I'm doing or using that just seem so bone-headed that I'll think, "I'm obviously well outside the ideal target audience for this". I seriously doubt that I'd think "This part of the system is so badly broken that it can't possibly be fixed"; again, this is a relatively new stack of tools with large, creative communities built around them. There will be alternatives to the particular whatever-it-is that's causing me pain, and there is a virtual certainty that I will be able to pull the offending piece out of my stack, put the new thing in, and continue working forward more effectively. I'm also convinced that, in the unlikely event that I'm such a black-swan corner case that I get motivated to write my own replacement, that it'll get visibility and feedback from others (if only to say "Hey, you just duplicated quteobscurename; go have a look at www.github.com/JoePalooka/quteobscurename — but hey, nicely done"). That would be more useful — and far more encouraging — than the usual PEAR mishmash of components that don't even all really work with the current language, written by people who believe that terse, sloppy, untested code is self-documenting.

---

Addition: April, 2012

And so it has come to pass; over the course of developing what will very soon be a commercial Web-based service, at least three initially-chosen components have been replaced by other, better and/or more suitable components. In all cases, the time involved in flailing around and deciding "this really has to go now" was several times greater than the hour or two it took to actually replace the old code with the new. That's been one of the big wins promised by object-oriented software development for roughly half a century. (Depending on who you ask, the first object-oriented language was either ALGOL X, in 1965, or Simula 67, in 1967. C++? Well over a decade later, and that's still older than most of its coders today.)

DRY May Not Be Wet, But It Sure Is Cool

As any developer who values his time, sanity, or amicable relations with teammates who have to maintain his code knows, one of the cardinal rules of modern programming is "Don't Repeat Yourself", or DRY. This is obviously something that should make reading code easier. Obviously, therefore, it should be applied except where the lengths you go to to not repeat yourself make your code harder for someone else to read. The idioms of some languages help this more than those of others.

I was recently browsing through a (rather poor) Ruby programming book when this hit me between the eyes. Consider this ERB example from the book.

Pretty ugly, yes? Part of that ugliness is intrinsic to ERB; which is why people are leaving it in droves.

Compare this exact translation into Haml:

= form_for @product do |f|
  - if @product.errors.any?
    .error_explanation
      %h2= "#{pluralize(@product.errors.count, "error")} prohibited this product from being saved:"

      %ul
        - @product.errors.full_messages.each do |msg|
          %li= msg

  .field
    = f.label :title
    %br/
    = f.text_field :title

  .field
    = f.label :description
    %br/
    = f.text_area :description

  .field
    = f.label :image_url
    %br/
    = f.text_field :image_url

  .field
    = f.label :price
    %br/
    = f.text_field :price

  .actions
    = f.submit

A bit easier to understand, now that we've gotten most of the clutter out of the way, yes? Now, the repeating field definitions stick out like a sore thumb; they're almost the same, but not exactly. We've got three text_fields and one text_area, so a simple loop that just plugs in values won't quite cut it, will it? How about this:

= form_for @product do |f|
  - if @product.errors.any?
    .error_explanation
      %h2= "#{pluralize(@product.errors.count, "error")} prohibited this product from being saved:"

      %ul
        - @product.errors.full_messages.each do |msg|
          %li= msg

  - [:title, :description, :image_url, :price] do |name|
    .field
      = eval("f.label :#{name}")
      %br/
      = eval("f.text_field :#{name}") unless name == :description
      = eval("f.text_area  :#{name}") if     name == :description

  .actions
    = f.submit

If you're coming to Ruby from a PHP background, you've been conditioned not to use eval; there's all sorts of nastiness that could lurk there, especially when code is sloppy. (And PHP code is notorious for its sloppiness.) But in Ruby and Haml, this lets us solve a problem more eloquently than the two or three other possibilities that come to mind. Further, the idiom of the repeated-but-not-repeated text_field/text_area fragment

      = eval("f.text_field :#{name}") unless name == :description
      = eval("f.text_area  :#{name}") if     name == :description

should make it clear to even the most hurried reader that we're dealing with a special case. This is one of those situations where a helper method would be nuclear overkill. (If we were scattering dozens of these all-but-one-fields-is-the-same forms through an app, I'd revisit that stance.)

---

OK, but why blog about this in the first place? Well, the company I'm working with now is looking for a senior Ruby/Rails dev. Most of the CVs we're getting are from mid-level guys (if you're a Rails doyenne (i.e., female), please email!) who have done some Ruby and one or two other languages, generally PHP and Java. What I've learned, on both sides of the table, is that if you've got some experience in several languages that aren't all closely-related to each other, it's a lot easier for you to ramp up on any language you need. Any language will eventually go out of favour. Being able to code idiomatically in anything you need to, won't.

Automating So You Don't Forget

This is a bit of an introductory-level post/rant/tutorial, but I've been peppered by enough "why on earth would you do this?" questions by various (seemingly experienced) project team members and on various mailing lists that I thought I'd just write my own take on this and point people to it when useful.

---

I'm pulling my (semihemidemiexistent) hair out on four different PHP projects at the moment. Not because they take all my time (they don't, unfortunately), but because three of them are in "maintenance" mode and express that in different ways. Take version control: one uses Mercurial (my favourite DVCS package); Subversion (once "subversive;" now the "safe" non-DVCS choice); and (tragically) git. Each project has different coding standards. One of those is actually widely-enough used that PHP CodeSniffer comes with support for it right in the tin. The others are relatively easy to code "sniffs" for. (Do remember that none of the three "maintenance" projects were using CodeSniffer (or equivalent), and all three have very sporadic use of their main VCS repositories.)

Wait a minute...now I've got to remember which standards go with which projects? And oh, yeah, it would be Really Nice™ to have any changes automatically saved in version control... if they're worthy.

What do I mean by "worthy?" Well, before I worry overmuch about how code is formatted, I should be able to prove that it works properly. After all, the most beautifully-formatted code that doesn't work is still (essentially) useless. This, of course, is where a tool like PHPUnit comes in; once you have sufficient coverage of your code with automatable tests, especially if you write the tests before you write (new) code, you can make changes confidently and quickly, because a) your tests prove that the code works as expected, and b) you're making sensible use of a (D)VCS, so that when your wonderful new code goes south and doesn't come back, you can follow your virtual-breadcrumb trail back up the face of the cliff. Only after PHPUnit blesses the code should CodeSniffer get a crack at it.

The new folks are scribbling away: "first test everything, then comply with standards, and then update version control." The rest of you are saying "hang on a minute; that problem's been sorted any of several different ways."

Precisely. If you're developing in the Java world, you're spoilt for choice: you can do perfectly reasonable build/test/deploy automation using Ant, or if you want to keep a large number of people (allegedly) gainfully employed managing a J2EE-on-steroids project, you can go for Maven.

In the PHP world, we've got a nice "little" analogue to Ant called Phing. It will quickly become "dead-finger" technology; you'll wonder how (or why) you ever did a reasonably "serious" project without it. And yet, most of the open-source PHP projects I've seen (on Sourceforge and elsewhere) don't use such a tool; they rely on error-prone, manual steps. This manual process, with steps easily forgotten or mangled, is the source of many bugs in released software — in any language.

Enter Phing (or equivalent). You set up the moral equivalent of a makefile with the steps you want to have performed the same way in the same order, every time. Phing supports properties, which can be stored separately from the "master" build file that references them. This allows you to set up consistent process and policy (defined in the build file) and plug in the values for a specific project using the separate properties file.

So how much difference does all this make? Let's take an example set of steps, some variation of which I follow in my build files:

1. First, clean out all the files created by steps that come later (like test reports);
2. Then, run unit tests, displaying the output as they run. If tests fail, stop;
3. Verify compliance with your chosen coding standards; if a problem is found, stop. Either fix the problem if it's in a file you've touched or add the file to the ignore list if it's a legacy file;
4. I like to run PHPDocumentor to automatically generate developer documentation, from comments left in the code. CodeSniffer will check these, too, so by the time phpdoc gets its grubby virtual paws on your code, it shouldn't find any problems;
5. If all is well, then it's on to version control. I have Phing show a "diff" report of what's changed since the last checkin, and then prompt me for a checkin comment. If I want to run the whole process but not check in to VCS (maybe I'm coming back to a project after a while away and just want to see the earlier steps run), I can hit the Return key, and my build file will skip the VCS checkin because I've supplied an empty comment (which it checks for).

Great, so (since I've followed a few conventions), all I need to do is type ''phing'' at the command line and it's off to the races. Trivially easy to use and, much more importantly, proof against a very high level of idiocy.

What's that? You in the back... I'm putting the cart before the horse, you say? I shouldn't do a process that drives VCS checkin, but a VCS checkin "hook" that does the validation and so on instead?

To some degree, that's a matter of taste. From a very practical perspective, though, having your build-and-test automation drive VCS instead of the other way 'round means that you can use any VCS operable from a command line, with minimal pain moving between projects. Not every VCS implements a pre-commit hook in the same way; some apparently don't implement them at all. (Yes, we know they're toys, but they're "enterprisey" big-ticket toys. Some managers will buy anything.) So, by having a single-command process execution/enforcement tool, you'll generally find that the internal and external quality of your project improves considerably and quickly; you'll also find that the risk involved with sweeping changes or audacious new features drops to a more comfortably survivable level.

And that's why I always answer the question "What tools should I be using for my PHP development?" to include at least:

• Your project's version control tool of choice (again, I recommend Mercurial);
• Phing;
• PHPUnit;
• PHP CodeSniffer; and
• PHPDocumentor.

Once we get people used to a core set of tools and practices, we can then go on to the thorny religious issues like, "which PHP framework should I use?"

Next question?

I Thought Standard Libraries Were Supposed to be Better...

...than hand coding. Either the PHP folks never got that memo, or I'm seriously misconceptualising here.

Case in point: I was reading through Somebody Else's Code™, and I saw a sequence of "hand-coded" assignments of an empty string to several array entries, similar to:

    $items[ 'key2' ] = '';
    $items[ 'key1' ] = '';
    $items[ 'key6' ] = '';
    $items[ 'key3' ] = '';
    $items[ 'key8' ] = '';
    $items[ 'key5' ] = '';
    $items[ 'key4' ] = '';
    $items[ 'key7' ] = '';

I thought, "hey, hang on; there's a function to do easy array merges in the standard library (array_merge); surely it'd be faster/easier/more reliable to just define a (quasi-)constant array and merge that in every time through the loop?"

Fortunately, I didn't take my assumption on blind faith; I wrote a quick little bit to test the hypothesis:

$count = 1e5;
$data = array(
        'key2' => '',
        'key1' => '',
        'key6' => '',
        'key3' => '',
        'key8' => '',
        'key5' => '',
        'key4' => '',
        'key7' => '',
        );
$realdata = array();

$start = microtime( true );
for ( $loop = 0; $loop < $count; $loop++ )
{
    $realdata = array_merge( $realdata, $data );
};
$elapsed = microtime( true ) - $start;
printf( "%ld iterations with array_merge took %7.5f seconds.\n", $count, $elapsed );

$start = microtime( true );
for ( $loop = 0; $loop < $count; $loop++ )
{
    $data[ 'key2' ] = '';
    $data[ 'key1' ] = '';
    $data[ 'key6' ] = '';
    $data[ 'key3' ] = '';
    $data[ 'key8' ] = '';
    $data[ 'key5' ] = '';
    $data[ 'key4' ] = '';
    $data[ 'key7' ] = '';
};
$elapsed = microtime( true ) - $start;
printf( "%ld iterations with direct assignment took %7.5f seconds.\n", $count, $elapsed );

I ran the tests on a nearly two-year-old iMac with a 3.06 GHz Intel Core 2 Duo processor, 4 GB of RAM, OS X 10.6.4 and PHP 5.3.1 (with Zend Engine 2.3.0). Your results may vary on different kit, but I would be surprised if the basic results were significantly re-proportioned. The median times from running this test program 20 times came out as:

Assignment process	Time (seconds) for 100,000 iterations
array_merge	0.41995
Hand assignment	0.15569

So, the "obvious," "more readable" code runs nearly three times slower than the existing, potentially error-prone during maintenance, "hand assignment." Hang on, if we used numeric indexes on our array, we could use the array_fill function instead; how would that do?

Adding the code:

    $data2 = array();
    $data2[ 0 ] = '';
    $data2[ 1 ] = '';
    $data2[ 2 ] = '';
    $data2[ 3 ] = '';
    $data2[ 4 ] = '';
    $data2[ 5 ] = '';
    $data2[ 6 ] = '';
    $data2[ 7 ] = '';
$start = microtime( true );
for ( $loop = 0; $loop < $count; $loop++ )
{
    $data2 = array_fill( 0, 8, '' );
};
$elapsed = microtime( true ) - $start;
printf( "%ld iterations with array_fill took %7.5f seconds.\n", $count, $elapsed );

produced a median time of 0.21475 seconds, or some 37.9% slower than the original hand-coding.

For folks coming from other, compiled languages, such as C, C++, Ada or what-have-you, this makes no sense whatsoever; those languages have standard libraries that are not only intended to produce efficiently-maintainable code, but (given reasonably mature libraries) efficiently-executing code as well. PHP, at least in this instance, is completely counterintuitive (read: counterproductive): if you're in a loop that will be executed an arbitrary (and arbitrarily large) number of times, as the original code was intended to be, you're encouraged to write code that invites typos, omissions and other errors creeping in during maintenance. That's a pretty damning indictment for a language that's supposedly at its fifth major revision.

If anybody knows a better way of attacking this, I'd love to read about it in the comments, by email or IM.

Those were the days, my friend...but they're OVER.

No, I'm not talking about Facebook and the idea that people should have some control over how their own information makes money for people they never imagined. That's another post. This is a shout out to the software folk out there, and the wannabe software folk, who wonder why nobody outside their own self-selecting circles seems to get excited about software anymore.

Back in the early days of personal computers, from 1977 to roughly 2000, hardware capabilities imposed a real upper limit on software performance and capabilities. More and more people bought more and more new computers so they could do more and more stuff that simply wasn't practical on the earlier models. Likewise, whenever any software package managed something revolutionary, or did something in a new way that goosed performance beyond what any competitor could deliver, that got attention...until the next generation of hardware meant that even middle-of-the-road software was "better" (by whatever measure) than the best of what could happen before.

After about 2000, the speed curve that the CPUs had been climbing flattened out quite a lot. CPU vendors like Intel and AMD changed the competition to more efficient designs and multi-core processors. Operating systems – Linux, Mac OS X and even Microsoft Windows – found ways to at least begin to take advantage of the new architectures by farming tasks out to different cores. Development tools went through changes as well; new or "rediscovered" languages like Python and Forth that could handle multiprocessing (usually by multithreading) or parallel processing became popular, and "legacy" languages like C++ and even COBOL now have multithreading libraries and/or extensions.

With a new (or at least revamped) set of tools in the toolbox, we software developers were ready to go forth and deliver great new multi-everything applications. Except, as we found out, there was one stubborn problem.

Most people (and firms) involved in software application development don't have the foggiest idea how to do multithreading efficiently, and even fewer can really wrap their heads around parallel processing. So, to draw an analogy, we have these nice, shiny new Porsche 997 GT3s being driven all over, but very few people know how to get the vehicle out of first gear.

I started thinking about this earlier this evening, as I read an article on Lloyd Chambers' Mac Performance Guide for Digital Photographers & Performance Addicts (pointed to in turn by Jason O'Grady and David Morgenstern's Speedtest article on ZDNet's Apple blog. One bit in Chambers' article particularly caught my eye:

With Photoshop CS4/CS5, there is also increased overhead with 8 cores due to CS5 implementation weakness; it’s just not very smart about knowing how many threads are useful, so it wastes time and memory allocating too many threads for tasks that won’t even benefit from them.

In case you've been stuck in Microsoft Office for Windows for the last 15 years, I'll translate for you: The crown-jewels application for Adobe, one of the oldest and largest non-mainframe software companies in history, has an architecture on its largest-selling target hardware platform that is either too primitive or defective to make efficient use of that platform. This is on a newly released version of a very mature product, that probably sells a similar proportion of Macs to the proportion of Windows PCs justified by Microsoft Office (which is even better on the Mac, by the way). Adobe are also infamous among Mac users and developers for dragging their feet horribly in supporting current technologies; CS5 is the first version of Adobe Creative Suite that even begins to support OS X natively – an OS which has been around since 2001.

I've known some of the guys developing for Adobe, and they're not stupid...even if they believe that their management could give codinghorror.com enough material to take it "all the way to CS50." But I don't recall even Dilbert's Pointy Haired Boss never actually told his team to code stupid, even if he did resort to bribery (PHB announces he'll pay $10 for every bug fix. Wally says, "Hooray! I'm gonna code me a minivan!"...i.e., several thousand [fixed] bugs).

No, Adobe folk aren't stupid, per se; they're just held back by the same forces that hold back too many in our craft...chiefly inertia and consistently being pushed to cut corners. Even (especially?) if you fire the existing team and ship the work off halfway around the world to a bunch of cheaper guys, that's a non-solution to only part of the problem. Consider, as a counterexample, medicine; in particular, open-heart surgery. Let's say that Dr. X develops a refinement to standard technique that dramatically improves success rates and survival lifetimes for heart patients he operates on. He, or a colleague under his direct instruction, will write up a paper or six that get published in professional journals – that nearly everybody connected with his profession reads. More research gets done that confirms the efficacy of his technique, and within a fairly short period of time, that knowledge becomes widespread among heart surgeons. Not (only) because the new kids coming up learned it in school, but because the experienced professionals got trained up as part of their continuing education, required to keep their certification and so on.

What does software development have in common with that – or in fact, with any profession? To qualify as a profession, on the level of the law, accounting, medicine, the military, or so on, a profession is generally seen as

• Individuals who maintain high standards of ethics and practice;

• who collectively act as gatekeepers by disciplining or disqualifying individuals who fail to maintain those standards;

• who have been found through impartial examination to hold sufficient mastery of a defined body of knowledge common to all qualified practitioners, including recent advances;

• that mastery being retained and renewed by regular continuing education of at least a minimum duration and scope throughout each segment of their careers;

• the cost of that continuing education being knowingly directly or indirectly subsidised by the professional's clients and/or employer;

• such that these professionals are recognised by the public at large as being exclusively capable of performing their duties with an assuredly high level of competence, diligence and care;

• and, in exchange for that exclusivity, give solemn, binding assurances that their professional activities will not in any way contravene the public good.

Whether every doctor (or accountant, or military officer, or lawyer) actually functions at that level at all times is less of a point than that that is what is expected and required; what defines them as being "a breed apart" from the casual hobbyist. For instance, in the US and most First World countries, an electrician or gas-fitter is required to be properly educated and licensed. This didn't happen just because some "activists" decided it was a good idea; it was a direct response to events like the New London School explosion. By contrast, in many other countries (such as Singapore, where I live now), there is no such requirement; it seems that anybody can go buy himself the equipment, buy a business license (so that the government is reassured they'll get their taxes), and he can start plastering stickers all over town with his name, cell phone number and "Electrical Repair" or whatever, and wait for customers to call.

Bringing this back to software, my point is this: there is currently no method to ensure that new knowledge is reliably disseminated among practitioners of what is now the craft of software development. And while true professionals have the legal right to refuse to do something in an unsafe or unethical manner (a civil engineer, for example, cannot sign off on a design that he has reason to believe would endanger public users/bystanders, and without properly-executed declarations by such an engineer, the project may not legally be built). Software developers have no such right. Even as software has a greater role in people's life from one day to the next, the process by which that software is designed, developed and maintained is just as ad hoc as the guy walking around Tampines stuffing advertisements for his brother's aircon-repair service under every door. (We get an amazing amount of such litter here.)

Adobe may eventually get Creative Suite fixed. Perhaps CS10 or even CS7 will deliver twice the performance on an 8-core processor than on a 4-core one. But they won't do it unless they can cost-justify the improvement, and the costs are going to be much higher than they might be in a world where efficient, modern software-development techniques were a professionally-enforced norm.

That world is not the one we live in, at least not today. My recurring fear is that it will take at least one New London fire with loss of life, or a software-induced BP Deep Horizon-class environmental catastrophe, for that to start to happen. And, as always, the first (several) iterations won't get it right – they'll be driven by politics and public opinion, two sources of unshakable opinion which have been proven time and again to be hostile to professional realities.

We need a better way. Now, if not sooner.

Let's Do The Time Warp Agai-i-i-i-n!! (Please, $DEITY, no...)

For those who may somehow not be aware of it, LinkedIn is a (generally quite good) professionally-oriented social-networking site. This is not Facebook, fortunately. It's not geared towards teenagers raving about the latest corporate boy band du jour. It often can be, however, a great place to network with people from a variety of vocational, industry and/or functional backgrounds to get in contact with people, share information, and so on.

One of the essential features of LinkedIn is its groups, which are primarily used for discussions and job postings. In the venerable Usenet tradition, these discussions can have varying levels of insightful back-and-forth, or they can degenerate into a high-fidelity emulation of the "Animal House" food fight. As with Usenet, they can often give the appearance of doing both at the same time. Unlike Usenet, one has to be a member of LinkedIn to participate.

One of the (several) groups I follow is LinkedPHPers, which bills itself as "The Largest PHP Group" on LinkedIn. Discussions generally fall into at least one of a very few categories:

• How do I write code to solve "this" problem? (the 'professional' version of "Help me do my homework");

• What do people know/think about "this" practice or concept?

• I'm looking for work, or people to do work; does anybody have any leads?

As veterans of this sort of discussion would expect, the second type of discussion can lead to long and passionate exchanges with varying levels of useful content (what became known on Usenet as a "flame war.") The likelihood of such devolution seems to be inversely proportional to its specificity and proportionally to the degree which the concept in question is disregarded/unfamiliar/unknown to those with an arguable grasp of their Craft.

It should thus be no surprise that a discussion on the LinkedPHPers group of "Procedural vs Object Oriented PHP Programming" would start a flame war for both of the above reasons. With 58 responses over the past month as I write this, there are informational gems of crystal clarity buried in the thick, gruesome muck of proud ignorance. As Abraham Lincoln is reported to have said, "Better to remain silent and be thought a fool than to speak out and remove all doubt."

What's my beef here? Simply that this discussion thread is re-fighting a war that was fought and settled over a quarter-century ago by programming in general. The reality is that any language that has a reasonable implementation of OOP (with encapsulation/access control, polymorphism and inheritance, in that order by my reckoning) should be used in that way.

Several of the posts trot out the old canard about a performance 'penalty' when using OOP. In practice, that's true of only the sharpest edge cases – simple, tiny, standalone classes that should never have been developed that way because they don't provide a useful abstraction of a concept within the solution space, generally by developers who are not professionally knowledgeable of the concepts involved and quite often by those copying and pasting code they don't understand into their own projects (which they also don't understand). That bunch sharply limited the potential evolution and adoption of C++ in the '80s and '90s, and many of their ideological brethren have made their home in Web development using PHP.

Yes, I know that "real" OOP in PHP is a set of tacked-on features, late to the party; first seriously attempted in PHP 4, with successively evolving implementations in 5.0, 5.2 and 5.3, with the semi-mythological future PHP 6 adding many new features. I know that some language features are horribly unwieldy (which is why I won't use PHP namespaces in my own code; proven idea, poor implementation). But taken as a whole, it's increasingly hard to take the Other Side ("we don' need no steeeenkin' objects") at all seriously.

The main argument for ignoring the "ignore OOP" crowd is simply this: competent, thoughtful design using OOP gives you the ability to know and prove that your code works as expected, and data is accessed or modified only in the places and ways that are intended. OOP makes "software-as-building-blocks" practical, a term that first gained currency with the Simula language in the mid-1960s. OOP enables modern software proto-engineering practices such as iterative development, continuous integration and other "best practices" that have been proven in the field to increase quality and decrease risk, cost and complexity.

The 'ignore OOP in PHP' crowd like to point to popular software that was done in a non-OOP style, such as Drupal, a popular open-source Web CMS. But Drupal is a very mature project, by PHP standards; the open-source project seems to have originated in mid-2000, and it was apparently derived from code written for a project earlier still. So the Drupal code significantly predates PHP 5, if not PHP 4 (remember, the first real whack at OOP in PHP). Perusing the Drupal sources reveals an architecture initially developed by some highly experienced structured-programming developers (a precursor discipline to OOP); their code essentially builds a series of objects by convention, not depending on support in the underlying language. It is a wonder as it stands – but I would bet heavily that the original development team, if tasked with re-implementing a Web CMS in PHP from a blank screen, would use modern OO principles and the underlying language features which support them.

And why would such "underlying language features" exist and evolve, especially in an open-source project like PHP, if there was not a real, demonstrable need for them? Saying you're not going to do OOP when using PHP is metaphorically akin to saying you intend to win a Formula One race without using any gear higher than second in the race.

Good luck with that. You might want to take a good, hard look at what your (more successful) colleagues are doing, adopt what works, and help innovate your Craft further. If you don't, you'll continue to be a drag on progress, a dilettante intent upon somehow using a buggy whip to accelerate your car.

It doesn't work that way anymore.

Process: Still 'garbage in, garbage out,', but...

...you can protect yourself and your team. Even if we're talking about topics that everybody's rehashed since the Pleistocene (or at least since the UNIVAC I).

Traditional, command-and-control, bureaucratic/structured/waterfall development process managed to get (quite?) a few things right (especially given the circumstances). One of these was code review.

Done right, a formal code review process can help the team improve a software project more quickly and effectively than ad-hoc "exploration and discovery" by individual team members. Many projects, including essentially all continuing open-source projects that I've seen, use review as a tool to (among other things) help new teammates get up to speed with the project. While it can certainly be argued that pair programming provides a more effective means to that particular end, they (and honestly, most agile processes) tend to focus on the immediate, detail-level view of a project. Good reviews (including but not limited to group code reviews) can identify and evaluate issues that are not as visibly obvious "down on the ground." (Cédric Beust, of TestNG and Android fame, has a nice discussion on his blog about why code reviews are good for you.

Done wrong, and 'wrong' here often means "as a means of control by non-technical managers, either so that they can honour arbitrary standards in the breach or so that they can call out and publicly humiliate selected victims," code reviews are nearly Pure Evil™, good mostly for causing incalculable harm and driving sentient developers in search of more humane tools – which tend (nowadays) to be identified with agile development. Many individuals prominent in developer punditry regularly badmouth reviews altogether, declaring that if you adopt the currently-trendy process, you won't ever have to do those eeeeeeeeevil code reviews ever again. Honest. Well, unless.... (check the fine print carefully, friends!)

Which brings us to the point of why I'm bloviating today:

1. Code reviews, done right, are quite useful;

2. Traditional, "camp-out-in-the-conference-room" code reviews are impractical in today's distributed, virtual-team environment (as well as being spectacularly inefficient), and

3. That latter problem has been sorted, in several different ways.

This topic came up after some tortuous spelunking following an essentially unrelated tweet, eventually leading me to Marc Hedlund's Code Review Redux... post on O'Reilly Radar (and then to his earlier review of Review Board and to numerous other similar projects.

The thinking goes something like, Hey, we've got all these "dashboards" for CRM, ERP, LSMFT and the like; why not build a workflow around one that's actually useful to project teams. And these tools fit the bill – helping teams integrate a managed approach to (any of several different flavours of) code review into their development workflow. This generally gets placed either immediately before or immediately after a new, or newly-modified, project artifact is checked into the project's SCM. Many people, including Beust in the link above, prefer to review code after it's been checked in; others, including me, prefer reviews to take place before checkin, so as to not risk breaking any builds that pull directly from the SCM.

We've been using collaborative tools like Wikis for enough years now that any self-respecting project has one. They've proven very useful for capturing and organising collective knowledge, but they are not at their best for tracking changes to external resources, like files in an SCM. (Trac mostly finesses this, by blurring the lines between a wiki, an SCM and an issue tracker.) So, a consensus seems to be forming, across several different projects, that argues for

• a "review dashboard," showing a drillable snapshot of the project's code, including completed, in-process and pending reviews;

• a discussion system, supporting topics related to individual reviews, groups of reviews based on topics such as features, or the project as a whole; these discussions can be searched and referenced/linked to; and

• integration support for widely-used SCM and issue-tracking systems like Subversion and Mantis.

Effective use of such a tool, whatever your process, will help you create better software by tying reviews into the collaborative process. The Web-based versions in particular remove physical location as a condition for review. Having such a tool that works together with your existing (you do have these, yes?) source-code management and issue-tracking systems makes it much harder to have code in an unknown, unknowable state in your project. In an agile development group, this will be one of the first places you look for insight into the cause of problems discovered during automated build or testing, along with your SCM history.

And if you're in a shop that doesn't use these processes, why not?

---

On a personal note, this represents my return to blogging after far, far too long buried under Other Stuff™. The spectacularly imminent crises are now (mostly, hopefully) put out of our misery now; you should see me posting more regularly here for a while. As always, your comments are most welcome; this should be a discussion, not a broadcast!

NIH v. An Embarrassment of Riches

One thing most good developers learn early on is not to "reinvent" basic technology for each new project they work on, The common, often corporate, antithesis to this is NIH, or "Not Invented Here." But sometimes, it's hard to decide which "giants" one wants to "stand on the shoulders of."

I've recently done a couple of mid-sized Web projects using PHP and the Kohana framework. A framework, as most readers know, is useful a) by helping you work faster b) by including a lot of usually-good code you don't have to write and maintain (but you should understand!). Good frameworks encourage you to write your own code in a style that encourages reuse by other projects that use the same framework.

One task supported by many frameworks is logging. There have also been many "standalone" (i.e., not integrated into larger systems) logging packages. The most well-known of these, and the source of many derivatives, is the Apache log4j package for Java. This has been ported, also as an Apache project, is log4php.

Log4php has saved me countless hours of exploratory debugging. I stand firmly with the growing group of serious developers who assert that if you use a reasonably agile process (with iterative, red, green, refactor unit testing) and make good use of logging, you'll very rarely, if ever, need a traditional debugger.

What does this have to do with Kohana? Well, Kohana includes its own relatively minimalist, straightforward logging facility (implemented as static methods in the core class, grumble, grumble). There's a standard place for such logs to be written to disk, and a nice little 'debug toolbar' add-on module that lets you see logging output while you're viewing the page that generated it.

So I ought to just ditch log4php in favor of the inbuilt logging system when I'm developing Kohana apps, right? Not so fast...

Log4php, as does log4j, has far more flexibility. I can log output from different sources to different places (file, system log, console, database, etc.), have messages written to more than one place (e.g., console and file), and so on. Kohana's logging API is too simple for that.

With log4php, I have total control over the logging output based on configuration information stored in an external file, not in the code itself. That means I can fiddle with the configuration during development, even deploy the application, without having to make any code changes to control logging output. The fewer times I have to touch my code, the less likely I am to inadvertently break something. Kohana? I only have one logging stream that has to be controlled within my code, by making Kohana method calls.

Many experienced developers of object-oriented software are uncomfortable with putting more than one logical feature into a class (or closely-related set of classes). Why carry around overhead you don't use, especially when your framework offers a nice extension capability via "modules" and "helpers"?. While there may sometimes be arguments for doing so (the PHP interpreter is notoriously slow, especially using dynamic features like reflection), I have always failed to understand how aggregating large chunks of your omniverse into a Grand Unified God Object™ pays dividends over the life of the project.

So, for now, I'll continue using log4php as a standalone tool in my various PHP development projects (including those based on Kohana). One thing that just went onto my "nice to do when I get around to it" list is to implement a module or similar add-on that would more cleanly integrate log4php into the surrounding Kohana framework.

This whole episode has raised my metaphorical eyebrow a bit. There are "best practices" for developing in OO (object-oriented) languages; PHP borrows many of these from Java (along with tools like log4php and PHPUnit, the de facto standard unit-test framework). I did a fairly exhaustive survey of the available PHP frameworks before starting to use Kohana. I chose it because it wasn't a "everything including several kitchen sinks" tool like Zend, it wasn't bending over backwards to support obsolete language misfeatures left over from PHP 4, and it has what looks to be a pretty healthy "community" ecosystem (unlike some once-heavily-flogged "small" frameworks like Ulysses). I'm not likely to stop using Kohana very soon. I may well have to make time to participate in that community I mentioned earlier, if for no other reason that to better understand why things are the way they are.

But that's the beauty of open source, community-driven development, surely?

Fixing A Tool That's Supposed To Help Me Fix My Code; or, Shaving a Small Yak

Well, that's a couple of hours I'd really like to have back.

One of the oldest, simplest, most generally effective debugging tools is some form of logging. Writing output to a file, to a database, or to the console gives the developer a window into the workings of code, as it's being executed.

The long-time "gold standard" for such tools has been the Apache Foundation's log4j package, which supports Java. As with many tools (e.g., PHPUnit), a good Java tool was ported to, or reimplemented in, PHP. log4php is uses the same configuration files and (as much as possible) the same APIs as log4j. However, as PHP and Java are (significantly) different languages, liberties had to be taken in various places. Add to this the fact that PHP has been undergoing significant change in the last couple of years (moving from version 5.2 to the significantly different 5.3 as we wait for the overdue, even more different, 6.0), and a famous warning comes to mind when attempting to use the tool.

Here be dragons.

The devil, the saying goes, is in the details, and several details of log4php are broken in various ways. Of course, not all the breakage gives immediately useful information on how to repair it.

Take, as an example, the helper class method LoggerPatternConverter::spacePad(), reproduced here:

    /**
     * Fast space padding method.
     *
     * @param string    $sbuf      string buffer
     * @param integer   $length    pad length
     *
     * @todo reimplement using PHP string functions
     */
    public function spacePad($sbuf, $length) {
        while($length >= 32) {
          $sbuf .= $GLOBALS['log4php.LoggerPatternConverter.spaces'][5];
          $length -= 32;
        }
        
        for($i = 4; $i >= 0; $i--) {    
            if(($length & (1<<$i)) != 0) {
                $sbuf .= $GLOBALS['log4php.LoggerPatternConverter.spaces'][$i];
            }
        }

        // $sbuf = str_pad($sbuf, $length);
    }

Several serious issues are obvious here, the most egregious of which is acknowledged in the @todo note: "reimplement using PHP string functions." The $GLOBALS item being referenced is initialized at the top of the source file:

$GLOBALS['log4php.LoggerPatternConverter.spaces'] = array(
    " ", // 1 space
    "  ", // 2 spaces
    "    ", // 4 spaces
    "        ", // 8 spaces
    "                ", // 16 spaces
    "                                " ); // 32 spaces

If you feel yourself wanting to vomit upon and/or punch out some spectacularly clueless coder, you have my sympathies.

The crux of the problem is that the function contains seriously invalid code, at least as of PHP 5.3. Worse, the error messages that are given when the bug is exercised are extremely opaque, and a Google search produces absolutely no useful information.

The key, as usual, is in the stack trace emitted after PHP hits a fatal error.

To make a long story short(er), the global can be completely eliminated (it's no longer legal anyway), and the code can be refactored so:

    public function spacePad(&$sbuf, $length) {
        $sbuf .= str_repeat( " ", $length );
    }

Of course, this makes the entire method redundant; the built-in str_repeat() function should be called wherever the method is called.

I'll make that change and submit a patch upstream tomorrow... errr, later today; it's coming up on 1 AM here now.

But at least now I can use log4php to help me trace through the code I was trying to fix when this whole thing blew up in my face.

At least it's open source under a free license.

You want to start a tutorial; well, you know...

Not as catchy as the Beatles' Revolution, even if the meter works.... oh well....

Continuing from the first post in this tutorial. What do I think is important when starting to demonstrate some code? As with most writing, it depends on the audience. For the purpose of this series of posts, I'm assuming that you fit comfortably in or near the following:

• You're comfortable with HTML and XML doesn't make you run screaming from the room;
• You have a basic understanding of databases; you've run across SQL before and understand the basic concepts;
• You understand PHP; you've written some code before;
• You understand the concepts of "object-oriented development", "patterns", "best practices" and ideally "test-driven development" (usually abbreviated as "TDD"), even though you may not have loads of experience (yet) with them; and crucially
• You want to improve your ability to write code that you can refine and possibly reuse over time.

The assumption that you know or at least are interested in PHP is a given, since that's the language we'll be using here.

What will you need to have installed and available to follow along?

1. Access to a system with PHP 5.2 or higher, available both from the command line and the Web server (via a module or CGI);
2. The PHPUnit and MDB2_Driver_mysql modules installed and available;
3. A text editor of your choice;
4. The ability to create PHP scripts and HTML files and have those accessible from the Web server as well as the command line.

These should all be pretty obvious to more experienced PHP developers, but making sure that we're both operating from the same set of assumptions — and no others — greatly reduces the likelihood of confusion and breakage along the way. Many of you haven't yet dealt much with unit tests using PHPUnit or similar systems; that's going to be a starting point for us.

Remember to test your testing tools!

I've been doing some PHP development lately that involves a lot of SPL, or Standard PHP Library exceptions. I do test-driven development for all the usual reasons, and so make heavy use of the PHPUnit framework. One great idea that the developer of PHPUnit had was to add a test-case method called setExpectedException(), which should eliminate the need for you (the person writing the test code) to do an explicit try/catch block yourself. Tell PHPUnit what you expect to see thrown in the very near future, and it will handle the details.

But, as the saying says, every blessing comes with a curse (and vice versa). The architecture of PHPUnit pretty well seems to dictate that there can only be one such caught exception in a test method. In other words, you can't set up a loop that will repeatedly call a method and pass it parameters that you expect it to throw on; the first time PHPUnit's behind-the-scenes exception-catcher catches the exception you told it was coming, it terminates the test case.

Oops. But if you think about it, pretty expectable (pardon the pun). For PHPUnit to catch the exception, the exception has to get thrown and unwind the call stack past your test-case method. That makes it very difficult (read: probably impossible to do reliably inside PHPUnit's current architecture) to resume your test-case code after the call that caused the exception to be thrown — which is what you'd want if you were looping through these things.

This leaves you, of course, with the option of writing try/catch blocks yourself — which you were hoping to avoid but which still works precisely as expected.

Moral of the story: Beware magic bullets. They tend to blow up in your face when you least expect it.

Maybe not eating 'crow', specifically, but..... DUCK!!!

As in, "bend over, here it comes again..."

One of the things I have greatly appreciated about the Mac, especially with OS X, is how simple and straightforward software management is, compared to Linux and especially Windows (where every system change is a death-defying adventure against great odds). Operating system or Apple-supplied apps need an update? Software Update is as painless as it gets: the defaults Just Work in proper Mac fashion, but you can set your own schedule, along with a few other options. There is a well-established convention for third-party apps to check for updates via a Web service "phoning home" at app startup; this has been very easy to deal with. Application and file layout is regular and sensible; libraries and resources are generally grouped in bundles at the system or user level. After a few years of DLL hell in Windows and library mix-and-match in Linux, this was shaping up to be a real pleasure.

Then, as some of you know, I updated Mac OS X on my iMac from 10.5.5 to 10.5.6. As expected, that apparently went as smooth as glass. I even blogged about it. XCode worked; MS Office 2008 for the Mac worked; Komodo Edit worked; all my IM clients worked; all seemed customarily wonderful in the omniverse. I even started up Mail; it opened normally and happily downloaded my regular mail and Google mail, just as it had done every day for months. (I didn't actually open any messages then; that will turn out to be important.) Satisfied that everything Just Worked as always, I went back to working on a project for a few hours before turning in for the night.

Next morning, I went through the usual routine. Awake the Mac from hibernation; log in; start Yahoo, MSN and Skype; start Mail; open Komodo; open Web browsers (Safari, Opera and Camino) and I'm ready to get started. First thing...here's an interesting-sounding email message; let's open that up and... *POOF* — Mail crashes.

WTF? It started up just fine; I even got the "Message for you, Sir" Monty Python WAV I'd set Mail to use as my new-mail-received notification. I start Mail again. Picking a different message, I double-click it in the inbox. A window frame opens with the message title, sits empty for a few hundred milliseconds, then Mail goes away again. Absolutely, totally repeatable. Reboot changes nothing. Safe Boot (the Mac equivalent of Windows' "safe mode") changes nothing. The cold fingers of panic stroke my ribs like Glenn Gould at the piano. On a bad-karma scale of 0 to 10, initial reaction is an "O my God"; we're not dead, but we're hurt bad; the karma has definitely run over the dogma. 

The next couple of days are spent using my ISP's Webmail service, and a set of Python scripts I'd previously written to search mailbox contents — Apple Mail, like any sensible email program, adheres to established standard formats. If I'd been using Microsoft Lookout! in a similar situation, I'd have been up the creek.

Finally, I come across some Web-forum items that indicate that GPGMail needs to be updated; if it's not, Mail will crash under OS X 10.5.6 — which is exactly what was happening. (If you're not using GPGMail, GNU Privacy Guard, or any of the various GPG interfaces for Windows such as Enigmail for Mozilla Thunderbird, you don't know how many people are recording and/or reading your email — but if it transits a server in the US or UK, it's guaranteed that it will be.

Installing the upgraded GPGMail bundle was the work of less than two minutes (hint: remove or rename the old bundle before copying the new one over. You probably don't need the insurance, but consider how we got here...). Then start up Mail as usual. It should, once again, Just Work — complete with being able to read and reply to messages, with or without GPG signatures.

OK, so what lessons can we take away from this experience, both as users and developers?

Time Machine may well be the single most rave-worthy piece of software I've touched in 30 years, but it can't (obviously, easily) do everything, and in a crisis, even experienced users may well not want to risk bringing too much (or too little) "back from history". There's definitely a market for addons to TM to do things like "look in my user and system library directories, the Application directory structure, MacPorts, etc., and bring application Foo back to the state it was in last Tuesday morning, but leave my data files as they are." I almost certainly could do that with the bare interface -- but, especially since it was "broken" as part of an OS upgrade, and (with the Windows/Linux experience fresh in mind) not comfortable exploring hidden dependencies... I was without my main email system for three days. Sure, I had workarounds -- that I wouldn't have had if I'd been in a stock Windows situation -- but that's not really the point, is it?

Also, app developers (Mac or other), add this to your "best practices" list: If your software uses any sort of plug-in/add-on architecture, where modules are developed independently of the main app, then you can have dependency issues. The API you make available to your plugin developers will change over time (or your application will stagnate); if you make it easy for them (and your users) to deal with your latest update, you'll be more successful. There's (at least) two ways to go about doing this:

The traditional "brute force" approach. Have a call you can use to tell plugins what version of the app is running, and allow them to declare whether or not they're compatible with that version. Notify the user about any that don't like your new version. For examples of this, see the way Firefox and friends deal with their plugins. Yes, it works, but it's not very flexible; a new version may come out that doesn't in fact modify any of the APIs you care about — which means that the plugin should work even though it was developed against version 2.4 of your app and you're now on 4.2.

Alternatively, a more fine-grained approach. Group your API into smaller, functional service areas (such as, say, address-book interface or encryption services for an email program). Have your plug-in API support a conversational approach.

1. The app calls into the plugin, asking it which services it needs and what versions of each it supports.
2. The app parses the list it gets back from the plugin. If the app version is later than the supported range for a specific feature identified by the plugin, add that to a "possibly unsupported" list. (If the app version is earlier than the range supported by the plugin, assume that it's not supported and go on to check the next one.)
3. If the "possibly unsupported" plugin list is empty, go ahead and continue bringing up the app, loading the plugins normally; you're done with this checklist.
4. For each item in the "possibly unsupported" list, determine whether the API for each feature required for the plugin has changed since the plugin was explicitly supported. (This is how a plugin for an earlier release, say 2.4, could work just fine with a later version, like 4.2.) If there's no change in the APIs of each feature required by the plugin, remove that plugin from the "possibly unsupported" list.
5. If any plugins remain in the list, check if there's an updated version of that plugin on the Net. This might be done using a simple web-service-to-database-query on your Web server. If your Web server knows of an update, ask the user for permission to install it. If the user declines, or no upgrade is available, unload the plugin. (You'll check again next time the app is started; maybe there's an update by then.)
6. Once the status of each plugin has been established, and compatible plugins loaded, finish starting up your app.

Of course, there are various obvious optimisations and convenience features that can be built into this. Any presentation to the user can and likely should be aggregated; "here's a list of the plugins that I wasn't able to load and couldn't find updates for." Firefox and friends are a good open-source example of this. The checks for plugin updates can also be scheduled, so as not to slow down every app startup. This might be daily, weekly, twice a month, whatever; the important thing is to let the user configure that schedule and view a list of plugins that are installed but not active.

As I started this post by saying, I've been very favorably impressed by Mac apps' ease of use (including installation and maintenance). Mail fell down and couldn't get up again without outside assistance; this is unusual. The fact that this was caused by a plugin and that Mail could not detect and work around the conflict just amazes me; I expect more from Apple. I'm not ready to decrease my use of the Mac because this happened — but I am going to pay more attention to how things work under the hood. The fact that I have to even be aware of this -- which is one of the features that hitherto distinguished the Mac from the grubbier Windows and Linux alternatives -- is worrisome.

Again, your comments are welcome.

Things that make you go 'Hmmmm', continued

Very much picking up from the mindset expressed in my earlier post... with the knowledge that this could (and probably should) be broken up into at least three different rants...

I've been working heavily in Python for the past couple of months, regrettably letting some other projects slide a bit. Now done with that, I spent yesterday picking up where I'd left off in a moderately-sized, reasonably well-designed PHP 5.2 project. (Bear in mind that my PHP experience is easily 5 or 6 times as much as my Python.)

And... while I'm not ready to jump on the "PHP sucks" bandwagon, it does feel clunky. Occasionally obscure (though never up to the standards of obscurity a good Perl hacker deals with every day).

Why is this? Three years ago, Joel Spolsky wrote an excellent rant on The Perils of JavaSchools. His point essentially boils down to that how you're trained (or "educated") as a developer shapes the way you look at problems; if all you know is a "Hammer", you try to visualize every problem as a "Nail" (even when it's a "Glass Figurine"). You may well have less-than-satisfying success with that view.

More importantly, the tools and techniques you know shape whether you can properly understand a problem at all. Not having certain features in a language (or not being knowledgeable in their use) means that you'll write clunky, hard-to-understand (and therefore -maintain) code to achieve the desired result...and wind up with (an attempt at) calculus using Roman numerals. Just as having a positional numeric system (e.g. Arabic or "modern" numerals) makes whole classes of problems possible that weren't otherwise, languages and their features make programming problems practical or more efficient.

What we don't want is one language that tries to do everything, in every way possible. We already have that; it's called Perl, and one ramification of its overriding philosophy, "There's more than one way to do it", is that there's always a better way to do it; the search for same can and often does suck in resources faster than a Sol-sized black hole. This also illustrates the failing of most of the more recent practitioners of software development that I've worked with. While those of us who've been working since before oh, about 1988 or 1990 generally make a point of reading at least one new technical book a month, I recently led a group of about 20 young (less than 5 years experience as of 2006) developers where not a single one admitted to reading more than one technical book a year since graduation. These people didn't know how to solve the problem we were working on because the two or three tools which they were familiar with encourage their users to think in ways that do not lead to effective solutions for this problem.

A language, any language, can really only do a limited number of things well. If you are fluent in more than one human language, say, English and Mandarin Chinese, think for a moment about concepts and sayings that are natural in one language but just don't work well in the other. Computer languages are like that, too, which is one reason why your computer's operating system is much less likely to be written in COBOL than your company's accounting program is (with benefits for all concerned).

Getting back to what started this rant....coming back to PHP after a sojourn in Python....

PHP gets the job done. Recent versions, particularly the current 5.2, are much more pleasant to work in than previous versions were for those of us who "think in objects". But it has taken years to get here.

As I look at one class in particular in this PHP project's code base, part of my mind is working on "if this were Python code, I'd write it like..." — and a two-hundred-line unit of code would be about 60 or 80, and much cleaner and easier to understand to boot. Why is that? Think "original intent".

PHP was originally developed as an adjunct to HTML for Web pages, to provide some simple dynamic content. It then "just growed", adding new features and capabilities (database access, object orientation) as it became used in a wider variety of problems. It is, quite simply, a tool that grew into a reasonably useful — if not quite general-purpose — language. There are a number of things it does quite well, especially since it can be used to do useful work without requiring a steep learning curve beforehand.

Python is different. A general-purpose language, with functional-programming features, it is useful for Web application development (e.g., with mod_python for the Apache Web server). Whereas Perl has the idea that "There's more than one way to do it" — and therefore neither a best way, nor strictly an advance certainty that anything in particular will work — Python argues that "there should be one — and preferably only one — obvious way to do it", whatever "it" is.

So am I suggesting that PHP developers ditch everything and go with Python? Of course not. What I am arguing is the seemingly quaint notion that developers, especially those who aspire to work in the craft for a living, should continually strive to learn new tools, techniques, languages and processes. Your abilities are like every other living thing; they're either growing, or they're dying.

Things that make you go 'Hmmmm'

...or 'Blechhhh', as the case may be... I've been using PHP since the relative Pleistocene (I recently found a PHP3 script I wrote in '99). I've been using and evangelising test-driven development (TDD) for about the last five years, usually with most such work being done in C++, Java, Python or other traditionally non-Web languages (with PHP really only being amenable to that since PHP 5 in 2004). So here I am, puttering away on a smallish PHP project that I've decided to TDD from the very beginning. For one of the classes, I throw together a couple of simple constructor tests in PHPUnit, to start, such as:

require_once( 'PHPUnit/Framework.php' );

require_once( '../scripts/foo.php' );

class FooTest extends PHPUnit_Framework_TestCase
    public function testCanConstructBasic();
    {
        $Foo = new Foo( 'index.php' );
    }
    
    public function testCanConstructBasicWildcard()
    {
        $Foo = new Foo( '*.php' );
    }
};

And, as is right and proper, I code the minimal class necessary to make that pass:

class Foo
{
};

That's it. That's really it. No declaration whatever for the constructor or any other methods in the class. Since it doesn't subclass something else, we can't just say "oh, there might be a constructor up the tree that matches the call semantics."  PHPUnit will take these two files and happily pass the tests. I understand what's really going on here - since the class is empty, you've just defined a name without defining any usage semantics (including construction). I would say fine; not a problem. But I would think that PHPUnit should, if not give an error, then at least have some sort of diagnostic saying "Hey, you're constructing this object, but there are no ctor semantics defined for the class." I can see people new to PHP and/or TDD, who are maybe just working through and mentally adapting an xUnit tutorial from somewhere, getting really confused by this. I know I did a double-take when I opened the source file to add a new method (to pass a test not shown above) and saw nothing between the curly braces. On one level, very cool stuff. On another, equally but not always obviously important level, more than enough rope for you to shoot yourself in the foot. Or, to put it another way, even though I've been writing in dynamic languages off and on for ages, I still tend to think in incompletely dynamic ways. Sometimes this comes back and bites me.  Beware: here be (reasonably friendly, under the circumstances) dragons.

Best Practices Alleged; Your Mileage May Vary

Yahoo! quite often releases interesting/useful/thought-provoking tools for people doing "serious" Web development. I add the modifier to specify that we're usually not talking about the Joe Leet three-page magnum oopus; a lot of what they do and talk about really only pays huge returns when you work with a site as large and complex as, well, Yahoo!.

Recently, they brought out a couple of nifty tools that integrate into the Firefox browser's Firebug Web-developer-Swiss-Army-knife extension. One of these, YSlow ("why [my site] slow?") does some interesting evaluations and calculations against whatever page (with secondary requests) you throw it at. Its "Performance" tab shows how your page matches up against Yahoo!'s new "Best Practices for Speeding Up Your Web Site." At first blush, a lot of these make perfect sense; "Avoid Redirects", "No 404s", and so on. YSlow, on the other hand, evaluates against a slightly different set of guidelines to those on the Best Practices Page:

1. Make fewer HTTP requests
2. Use a CDN
3. Add an Expires header
4. Gzip components
5. Put CSS at the top
6. Put JS at the bottom
7. Avoid CSS expressions
8. Make JS and CSS external
9. Reduce DNS lookups
10. Minify JS
11. Avoid redirects
12. Remove duplicate scripts
13. Configure ETags

"Huh?", our hypothetical Web pseudogod Mr Leet might well ask. "What the heck is an 'ETag'? Or a 'CDN'? Does any of this even apply to me?" Well, Joe, yes and no. For instance, content-delivery networks like Akamai or ATDN, as you might well know by hearing the names, scatter servers at strategic places around the planet with the aim of reducing the time it takes to get data from huge, media-content-heavy sites like CNN.com or the like, down to your browser at the end of a surprisingly long chain. Does everybody who puts a site up need something like this? Does the average small-to-midsize business? Usually not, unless you really are a Web Hype-Dot-Oh site that shoves exabytes out every day to wow the yokels or the investors. For the local pizza joint with a site containing maybe forty files, tops, with a couple of megabytes of images, a CDN is thermonuclear overkill. As many Web-development sites have pointed out for the last decade, there's quite a bit you can do to speed things up and lower bandwidth usage without spending the big bucks on this.

Why do I blather on about this when I started talking about best practices and YSlow? Because for practices to be "best", they first and foremost have to be appropriate for the use at hand. Buying a Lamborghini Countach to go down to the corner store for some sodas will quite likely get you yelled at by the Significant Other (followed by your bank). But if Lewis Hamilton showed up at pole position in a '72 Ford Pinto... you'd hear the laughter from St Paul to São Paolo.

Use the tools and techniques appropriate to the task at hand. There's a lot that small Website developers can learn from Google and the tools they publish. Getting an "A" score has a certain karmic appeal, and most of the optimizations required are straightforward anyway (tweaking how your Web server serves your data, for the most part). But is this worth all the geek love it's been getting?

Until someone with the developer credibility and experience of a Yahoo! stands up and explains a better set of practices for the SMB developer, the answer seems to be "yeah, probably". We who make our living (or our diversion) from the creation, care and feeding of Web sites are, for the most part, artisans posing as engineers, with inconsistent knowledge or practice of our craft; we dream of building the online equivalent of the Empire State Building but wind up with the Cologne Cathedral; a wonder, yes, but surely 600 years was well beyond the original estimated schedule! Agreed-upon standards (so that, say, a page appears identical in different browsers),; a shared, common body of knowledge; even (gasp!) widespread, vendor-neutral certifications of professional competence will eventually become common in software (including Web) development for the same reasons as, say, in architecture. The artifacts involved (skyscrapers, Web sites) have important social and policy implications, and inconsistent competence in practice poses a real and serious danger to the public at large. Sooner or later, it's going to be uneconomic for the present ad hoc system to advance the state of the art, or to meet the needs placed upon its products.

Best practices are good; best practices that actually work for the stated purposes in a broad variety of praxis are much better. But to get there, we're going to need to collaborate and communicate effectively, and to do that, we're going to have to make sure everybody involved is speaking the same language to describe the same things. If we don't, we'll continue to be stuck in pretty much the same place we are now — with a bunch of shade-tree mechanics running around in the pits at the Monaco Grand Prix...only doing a lot more damage.

Comments are welcomed, as always.