Rules can be broken, but there are Consequences. Beware.

If you're in an agile shop (of whatever persuasion), you're quite familiar with a basic, easily justifiable rule:

No line of production code shall be created, modified or deleted in the absence of a failing test case, for which the change is required to make the test case pass.

Various people add extra conditionals to the dictum (my personal favourite is "…to make the test case pass with the minimal, most domain-consistent code changes currently practicable") but, if your shop is (striving towards being) agile, it's very hard to imagine that you're not honouring that basic dictum. Usually in the breach at first, but everybody starts in perfect ignorance, yes?

I (very) recently was working on a bit of code that, for nearly its entire existence over several major iterations, had 100% test coverage (technically, C0 or line coverage) and no known defects in implemented code. It then underwent a short (less than one man-week) burst of "rush" coding aimed at demonstrating a new feature, without the supporting tests having been done beforehand. It then was to be used as the basis for implementing a related set of new features, that would affect and be affected by the state of several software components.

That induced some serious second-guessing. Do we continue mad-hatter hacking, trusting experience and heroic effort to somehow save the day? Do we go back and backfill test coverage, to prove that we understand exactly what we're dealing with and that it works as intended before jumping off into the new features (with or without proper specs/tests up front)? Or do we try to take a middle route, marking the missing test coverage as tech debt that will have to be paid off sometime in The Glorious Future To Come™? The most masochistic of cultists (or perhaps the most serenely confident of infinite schedule and other resources) would pick the first; the "agile" cargo-cultist with an irate manager breathing fire down his neck the second; but the third is the only pragmatic hope for a way forward… as long as development has and trusts in assurances that the debt will be paid in full immediately after the current project-delivery arc (at which time increased revenue should be coming in to pay for the developer time).

The moral of the story is well-known, and has been summed up as "Murphy " (of Murphy's Law fame) "always gets his cut", "payback's a b*tch" and other, more "colourful" metaphors. I prefer the dictum that started this post, perhaps alternately summed up as

Don't point firearms at bits of anatomy that you (or their owner) would mind losing. And, especially, never, ever do it more than once".

Because while even the most dilettante of managers is likely to have heard Fred Brooks' famous "adding people to a late project only makes it later" or his rephrasing as "nine women cannot have a baby in one month", too few know of Steve McConnell's observation (from 1995!) that aggressive schedules are at least equally delay-inducing. If your technical people say that, with the scope currently defined, that something will take N man-weeks, pushing for N/4 is an absolutely fantastic way to turn delivery in N*4 into a major challenge.

Remember, "close" only counts in horseshoes and hand grenades and, even then, only if it's close enough to have the intended effect. Software, like the computers it runs on, is a binary affair; either it works or it doesn't.

In Praise of Robust Tools and of The Future™

One of the best points about doing Web development in PHP is that it's so widely used; several respectable estimates by organizations that get paid to find these things out say that some varying number north of 50% of all sites on the World Wide Web use PHP as their implementation language. This includes numerous content management systems, or CMS, such as WordPress, Joomla! and Drupal.

One of the far-less-good points about doing Web development in PHP is that it's so widely used; the (evolving) "standard" set of tools, in true open-source fashion, are assembled and maintained by an ad-hoc band of individual and small-corporate luminaries, on infrastructure that worked just fine back when any given server was hit a couple of thousand times a year; by continuing to rely on (what to the outside Net appear to be) individual servers without terribly huge pipes connecting them to the larger Internet, the infrastructure completely fails when that's scaled up by a factor of a thousand or ten.

Web developers using PHP are highly dependent on an extension architecture/platform called PEAR, the "PHP Extension and Application Repository." Singular. Well, for any given extension or application, it's singular. So, as the user base scales upwards, popular tools that go through a vigorous release cycle, the servers they're hosted on (and network choke points between large subsets of users and those servers) start having reliability problems; transfers slow to a crawl, or fizzle out entirely. (One such transfer this evening proceeded at a sedate 600 bytes per second. Not kilobytes: bytes. That's 1980s dial-up speed.)

Developers in other languages have similar tools; Rubyists have gems available from sites like rubygems.org; Pythonistas have eggs. But somehow I hear a lot less grumbling, and do less of my own, when the subjects of gems or eggs come up; they seem to Just Work™ – which indicates that the network infrastructure is a lot more robust; either larger "pipes" and/or more, distributed servers such as with a CDN.

Trying to access servers like this from Second World cities like Singapore that apparently devote more resources to content filtering/monitoring than easing congestion doesn't help either. Hey, SingNet and M"D"A, how come it takes ten hops to get off an island that's barely that many miles across? If we want to plug Singapore into the "new global economy," having grotesquely under-resourced connections is not helpful.

But back to the main subject: If PHP is going to continue its phenomenally successful growth, then the infrastructure is going to have to decentralise, aggressively. Having essential tools like PEAR and the PHPUnit repository available only from a single point ensures that single points of failure will continue to seriously compromise the PHP ecosystem. No access to servers, no access to tools. No access to tools, then development of PHP artifacts with reasonable efficiency and economy is severely degraded.

In other words, the status quo is a limiting factor on future growth and success. However, as many will be quick to point out, building infrastructure costs money – for hardware, for connectivity, for the paid labour of skilled craftspeople and professionals to create, install, operate and maintain this enhanced infrastructure. No explicit means of funding such an endeavour presently exists, at least not to my knowledge.

So where do we go from here, PHP community?>

Companies Lose their Minds, Then their Partners, Then their Customers, Then...

Following is a comment which I posted to Jason Kincaid's article on TechCrunch, "Why Apple's New Ban Against Sexy Apps is Scary". I don't know why Apple seem to be deliberately shooting themselves in so many ways on the iPhone recently; I am sure that they are leaving golden opportunities for Palm, Android and anybody else who isn't Apple or Microsoft.

Even if you're not developing for the iPhone or even for the Mac, this whole drift should concern you — because its most likely effect is going to be that you have fewer choices in what should be a rapidly-expanding marketplace.

Thanks for reading.

---

Exactly; they're pulling a Singapore here. They're saying "we're better than anybody else" because they've got this App Store with hundreds of thousands of titles, and hundreds of useful apps. Then they turn around and say "we're the only game in town; we're not going to let you sell your apps to customers any way except through us — and oh, yeah, we can be completely arbitrary and capricious before, during and after the fact."

Let me be clear: up until very recently, I've been an unalloyed Apple fan; the only sensible response to 25+ years of Windows development and user support and 10 years hitting similar but different walls in Linux. I'm typing this on one of the two Macs sitting on my desk. I've got logs and statistics that prove I'm far more productive on my worst Mac days than I ever was on my best Windows days. And I've had several Switcher clients over the past few years who say the same thing.

I can write and sell any app I want on the Mac; Apple even give me all the (quite good) tools I need right in the box. I can use any app I want to on my Mac; the average quality level is so far above Windows and Linux apps it's not even funny. In neither of those do I need the permission of Apple or anyone else outside the parties to the transaction involved. Apple do have good support for publicising Mac apps; browse http://www.apple.com/downloads/ to see a (far earlier) way they've done it right. But developers don't have to use their advertising platform.

With the iPhone, and soon the iPad, they're doing things in a very untraditionally-Apple way: they're going far out of their way to alienate and harm developers. You know, those people who create the things that make people want to use the iPhone in the first place. And a lot of us are either leaving the platform or never getting into iPhone development in the first place.

And that can't be healthy for the long-term success of the Apple mobile platform (iPhone/iPad/iWhatComesNext). As a user, as a developer, as a shareholder, that disturbs me, as I believe it should disturb anyone who cares about this industry.

What's wrong with this picture?

Apple chose the only name that ever made sense for their new change-the-world device: the iPad.

So what's not to like about it? Well, that depends on who you are and what you really wanted this to be. No surprise there.

My biggest problem with it was summed up well by James Kendrick in his post, Thoughts on the iPad — Just Push the Buy Button, says Apple: it really is primarily about media consumption - i.e., paying big corporations for temporary, passive access to data. Something to switch your creative intellect off, not on.

And those of us who are often called the "Apple faithful" were hoping, praying if you will, for something so much more. Which in fact this could still be. But for the first time in Apple history, a product with serious computing power is being positioned in the marketplace such that creative, collaborative expression by individuals is almost a subversive act.

We've fallen a long, long way from "Think Different". And it's now much, much harder to argue that Apple is as much a computer company dedicated to creativity and the pursuit of excellence, than to argue that Apple has become "just another" corporate big-media company, helping to turn the Internet from what was once called "the greatest opportunity for expressive, collaborative democratic action in human history" into just another television set. Active creation becomes passive "consumption." All hail the primacy of content, and don't even think about upsetting the applecart. That would be Thinking (too) Differently.

Fixing A Tool That's Supposed To Help Me Fix My Code; or, Shaving a Small Yak

Well, that's a couple of hours I'd really like to have back.

One of the oldest, simplest, most generally effective debugging tools is some form of logging. Writing output to a file, to a database, or to the console gives the developer a window into the workings of code, as it's being executed.

The long-time "gold standard" for such tools has been the Apache Foundation's log4j package, which supports Java. As with many tools (e.g., PHPUnit), a good Java tool was ported to, or reimplemented in, PHP. log4php is uses the same configuration files and (as much as possible) the same APIs as log4j. However, as PHP and Java are (significantly) different languages, liberties had to be taken in various places. Add to this the fact that PHP has been undergoing significant change in the last couple of years (moving from version 5.2 to the significantly different 5.3 as we wait for the overdue, even more different, 6.0), and a famous warning comes to mind when attempting to use the tool.

Here be dragons.

The devil, the saying goes, is in the details, and several details of log4php are broken in various ways. Of course, not all the breakage gives immediately useful information on how to repair it.

Take, as an example, the helper class method LoggerPatternConverter::spacePad(), reproduced here:

    /**
     * Fast space padding method.
     *
     * @param string    $sbuf      string buffer
     * @param integer   $length    pad length
     *
     * @todo reimplement using PHP string functions
     */
    public function spacePad($sbuf, $length) {
        while($length >= 32) {
          $sbuf .= $GLOBALS['log4php.LoggerPatternConverter.spaces'][5];
          $length -= 32;
        }
        
        for($i = 4; $i >= 0; $i--) {    
            if(($length & (1<<$i)) != 0) {
                $sbuf .= $GLOBALS['log4php.LoggerPatternConverter.spaces'][$i];
            }
        }

        // $sbuf = str_pad($sbuf, $length);
    }

Several serious issues are obvious here, the most egregious of which is acknowledged in the @todo note: "reimplement using PHP string functions." The $GLOBALS item being referenced is initialized at the top of the source file:

$GLOBALS['log4php.LoggerPatternConverter.spaces'] = array(
    " ", // 1 space
    "  ", // 2 spaces
    "    ", // 4 spaces
    "        ", // 8 spaces
    "                ", // 16 spaces
    "                                " ); // 32 spaces

If you feel yourself wanting to vomit upon and/or punch out some spectacularly clueless coder, you have my sympathies.

The crux of the problem is that the function contains seriously invalid code, at least as of PHP 5.3. Worse, the error messages that are given when the bug is exercised are extremely opaque, and a Google search produces absolutely no useful information.

The key, as usual, is in the stack trace emitted after PHP hits a fatal error.

To make a long story short(er), the global can be completely eliminated (it's no longer legal anyway), and the code can be refactored so:

    public function spacePad(&$sbuf, $length) {
        $sbuf .= str_repeat( " ", $length );
    }

Of course, this makes the entire method redundant; the built-in str_repeat() function should be called wherever the method is called.

I'll make that change and submit a patch upstream tomorrow... errr, later today; it's coming up on 1 AM here now.

But at least now I can use log4php to help me trace through the code I was trying to fix when this whole thing blew up in my face.

At least it's open source under a free license.

Time v. Money v. Risk v. Frustration

I just spent three hours beating my head against The Joys of Wi-Fi Networking. No doubt because it saved $50 or so, the HDB public-housing flat I live in here in Singapore (which are "affectionately" known as 'chicken coops' for their quality and structural integrity) didn't put telephone jacks in every room. Nor did they run conduit between rooms so that suchlike could be added later. The upshot of this is that my DSL modem, in the living/dining room, is 15 meters and two concrete-slab walls away from my study/office, in a bedroom (with no phone jack and two power outlets).

Up to now, this hasn't been an insurmountable problem, because the DSL modem/802.11g router the government-linked telco sells you when you open a DSL subscription could punch a (barely) usable signal through those two concrete walls; desktop and laptop computers strewn around the study were on the LAN, with quite serviceable WPA2 encryption to keep private bits private. The router (in the living room) claimed to have a 14 Mbps connection to The Net; the aggregate total bandwidth available in the study ranged from 3 to 5 Mbps; not fantastic but usable. The world was in a survivable state of chaotic flux.

Then, yesterday morning, FedEx delivered a bright, shiny (actually used, in a Target bag rather than original box, no longer in the Polycom US catalog but still serviceable) Polycom SoundPoint 501 SIP phone, courtesy of my newest client (thanks, Nathan, Matt and Margaret - I'm not kvetching, honest!). An hour of work with jackhammer and flamethrower cleared sufficient space on the desk for the new trophy. Now to plug it in and fire it up.

Plug it in? To what? My ear?

Just to make sure I had my head straight, I looked at what it would take to run line from point P to R. Two solid concrete/rebar walls, routing around various doors/windows, across the span of the living room.... no, that wasn't going to happen, certainly not by Thursday. Put the phone in the living room? Not a chance. Wait a minute - they have these things called wireless Ethernet bridges; I should be able to get one of those, stick it in the study, connect the phone to it and it to the existing wireless LAN, and I'm good to go.

I went out to the local "IT super-mall", Funan. It's the best place in Singapore to buy electronic gear of whatever variety from reasonably reputable dealers. After visiting some 25 different shops, large and small, I was firmly reacquainted with one of the basic facts of Singapore life.

This is a firmly Stalinist economy in the areas that matter. All imports (which is to say, anything of value) are brought in through a small number of generally well-connected exclusive distributors. When one shop says "finished, already!" (the de facto motto of the city), everybody is going to say the same thing. That is, if you're fortunate enough to even run into some salesperson who actually understands what you're looking for. I could find things easier (and often cheaper!) in Vietnam.

Then I happened to pop into one hole-in-the-wall one-man shop, Crystal Systems. The "one man" said "say, I understand what you're trying to do; I did exactly that for a firm here recently; all you need are two routers." The archetypal light-bulb moment. (Remember, I'm a software craftsman, not a hardware/network engineer.) This is where the "time v. money v. ..." of the title comes in. What I should have done was to buy two identical brand-new 802.11n routers. That would have cost me about S$190 (~US$136 or so), and would have given me a perfect excuse to replace my aging 2Wire 2700HGV-2, But this has already been a bit of an expensive month - new hard drive, various software - so after some consideration I decided to try to salvage the existing router (apparent POS though it may be) and just spend the minimum necessary. So I went home with one S$69 TP-LINK TL-WR340G router.

It features the now-customary browser-based setup - just remember to use a hard-wired link rather than wireless (d'oh!), and it's almost as easy as falling off your seat. Bridging mode is pretty obvious; the TP-LINK wants to know the MAC address of the router you're connecting to, along with its encryption setup.

In bridge mode, the TL-WR340G only supports the obsolescent WEP encryption, not the current and generally superior WPA2. The 2Wire - along with every device currently connected to it, obviously - uses WPA2, and does not support a mixed WPA2/WEP environment.

Back to the store I go tomorrow; the only question now is: one new Netgear N router (which can fall back to 802.11g) or two?