Again, Standards Matter

People who I've worked with, or worked for, or read my writing here and elsewhere, have probably figured out that I'm a huge fan of standards just about everywhere they make sense: data formats, user interfaces, and so on. After all, why should we have to relearn how to drive a car simply because we buy a new Ford in place of a Toyota that the Government doesn't want us driving anymore? (You see very few old — or even fully-paid-for — cars in Singapore.) The steering wheel, pedals, and other controls are in a familiar layout; any slight differences are quickly adapted to.

Not so with the Western world's most widely-sold word processing software (for instance); when Microsoft Word 2007 for Windows shipped with a different, unique ('innovative', whether or not you find it debatable, is beside the current point) interface. Bloggers bloviated, many users were deeply confused, and corporate help-desk calls (and support/training costs) spiked. People running Windows PCs were very rarely neutral about the change.

A year later, Microsoft shipped Word 2008 for the Mac. Although there were some interface changes, the points of loudest discussion in the Word:Mac user community seemed to be

• the omission of Visual Basic for Applications as an attempted cross-platform macro language; and
• the new semi-proprietary document format, which allowed flawless interchange with Windows users (VBA notwithstanding).

Interface changes, per se, didn't spark nearly as much angst as had the Windows version of the year before. While a certain amount of this should no doubt be attributed to Microsoft's experience with the earlier release, the main reason was both different and obvious.

When developing Mac applications, it's much easier and better to follow the Apple Human Interface Guidelines than to "roll your own" interface. Developers, including Microsoft, are well aware of the ways in which the Mac development tools make your life easier if you structure and style your app to meet the Guidelines (and user expectations), as opposed to how much scutwork needs to be reinvented from scratch to do things differently. Users benefit even more, as the amount of learning needed to use a new app, or a new version of an existing app, are much less than is the average under Windows or Linux. And, unlike far too many Windows programs, Mac programs are usually highly discoverable; the user may not know how to accomplish a desired action, but there is one (and preferably only one) obvious path to follow, and mis-steps are generally not heavily penalised.

Right, "everybody" knows this, so why did I spend five paragraphs restating the reasonably obvious? Because the real intent of this post is to draw your attention to a phenomenon which is a necessary outcome of that standardisation and discovery: it is much easier to switch from one Mac app that performs a given task to another than it is on Windows. Most Mac users periodically switch between different applications for a given purpose, even keeping two or three installed on their systems. When you ask them why, they don't (generally) point to bugs or deficiencies in one product over another; they merely switch between them as their use cases change. For example, though I have both Microsoft Office and Apple iWork on this Mac, I will often create or open smaller Word documents in a simpler application such as AbiWord instead. It doesn't have all the features of Word or Pages, but it has the basics, and it loads more quickly and uses fewer resources than its two "big brothers."

The average Mac user is also generally more willing to try new applications, and generally owns more applications, than is the average for Windows. Since she is confident in her ability to pick up and use a new program, generally without resorting to a manual or even online help, there is a much more open discussion between users and developers, since both have seen a good bit of "the competition" and know what they like and don't like.

More rarely than is the case elsewhere, but not rarely enough, this easy migration from one app to another is due to real or perceived defects in a previously-used program. This happened to me recently; the program I had been using for a few months as my main Twitter client was not showing me all the tweets of people I was following in the "mainline" stream that I would see when I looked at each person's stream individually. Once you start following more than about two or three people, the mainline becomes absolutely indispensable; you simply don't want to have to take the time to look at each stream in isolation. So, I moved to another client, Nambu (now in "private beta" for a new release; version 1.2.1 can be found via web search).

Two immediate observations: I already know how to use this, even though Nambu has a far more dense presentation than my earlier client. And, because of that "dense presentation", it now takes me about a fifth as much time to get through my morning and afternoon Twitter catchups as it did previously. (Multi-column view is the killer feature, guys; there's only one little thing I'd like to see different...)

Again, why make noise about this? Simple: I've been a Windows user (usee?) and developer quite literally as long as there's been a "Windows"; I ran across my 1.0-beta developer kit floppies (5-1/4", of course) a couple of weeks ago (thinking about having them bronzed...or mounted on a dartboard. Maybe both.) But the nasty truth is, I very rarely change applications that perform a given task in Windows. The pain level and the (hopefully temporary) hit on my productivity aren't worth it until the pain becomes absolutely excruciating. I don't have that problem with the Mac, at all. I can try out new applications at will, even daily-workflow applications, secure in the knowledge that

• I already know how to use this, at least well enough to get started, and
• I can go back — or on to another candidate — any time I want to.

There's a word for the feeling that having that kind of freedom, that control over your computing experience gives you:

Empowerment.

Companies Lose their Minds, Then their Partners, Then their Customers, Then...

Following is a comment which I posted to Jason Kincaid's article on TechCrunch, "Why Apple's New Ban Against Sexy Apps is Scary". I don't know why Apple seem to be deliberately shooting themselves in so many ways on the iPhone recently; I am sure that they are leaving golden opportunities for Palm, Android and anybody else who isn't Apple or Microsoft.

Even if you're not developing for the iPhone or even for the Mac, this whole drift should concern you — because its most likely effect is going to be that you have fewer choices in what should be a rapidly-expanding marketplace.

Thanks for reading.

---

Exactly; they're pulling a Singapore here. They're saying "we're better than anybody else" because they've got this App Store with hundreds of thousands of titles, and hundreds of useful apps. Then they turn around and say "we're the only game in town; we're not going to let you sell your apps to customers any way except through us — and oh, yeah, we can be completely arbitrary and capricious before, during and after the fact."

Let me be clear: up until very recently, I've been an unalloyed Apple fan; the only sensible response to 25+ years of Windows development and user support and 10 years hitting similar but different walls in Linux. I'm typing this on one of the two Macs sitting on my desk. I've got logs and statistics that prove I'm far more productive on my worst Mac days than I ever was on my best Windows days. And I've had several Switcher clients over the past few years who say the same thing.

I can write and sell any app I want on the Mac; Apple even give me all the (quite good) tools I need right in the box. I can use any app I want to on my Mac; the average quality level is so far above Windows and Linux apps it's not even funny. In neither of those do I need the permission of Apple or anyone else outside the parties to the transaction involved. Apple do have good support for publicising Mac apps; browse http://www.apple.com/downloads/ to see a (far earlier) way they've done it right. But developers don't have to use their advertising platform.

With the iPhone, and soon the iPad, they're doing things in a very untraditionally-Apple way: they're going far out of their way to alienate and harm developers. You know, those people who create the things that make people want to use the iPhone in the first place. And a lot of us are either leaving the platform or never getting into iPhone development in the first place.

And that can't be healthy for the long-term success of the Apple mobile platform (iPhone/iPad/iWhatComesNext). As a user, as a developer, as a shareholder, that disturbs me, as I believe it should disturb anyone who cares about this industry.

Simple Changes Rarely Are

I want to say "thank you" to the anonymous commenter to my post, XHTML is (Nearly) Useless, who said "Paragraphs would be useful."

Google's Blogger software, by default, separates paragraphs by an "HTML break tag" (br); this preserves the visible separation as intended, but destroys the structural, semantic flow. I recently discovered this, found the feature to turn it off ("don't put "break" tags in unless I bloody tell you to!), and set it. I naively thought that this would apply only to new posts, and posts that had previously been saved in the old, "broken", format would stay as is.

Oops.

I must now go back and hand-edit every post over the last nearly five years so that their paragraph flow is properly marked up. Since I do in fact have other demands on my time, this is not likely to be completed immediately; it is, however, a very high priority.

It's enough to make you seriously consider migrating to WordPress — which is apparently trivially easy.

NIH v. An Embarrassment of Riches

One thing most good developers learn early on is not to "reinvent" basic technology for each new project they work on, The common, often corporate, antithesis to this is NIH, or "Not Invented Here." But sometimes, it's hard to decide which "giants" one wants to "stand on the shoulders of."

I've recently done a couple of mid-sized Web projects using PHP and the Kohana framework. A framework, as most readers know, is useful a) by helping you work faster b) by including a lot of usually-good code you don't have to write and maintain (but you should understand!). Good frameworks encourage you to write your own code in a style that encourages reuse by other projects that use the same framework.

One task supported by many frameworks is logging. There have also been many "standalone" (i.e., not integrated into larger systems) logging packages. The most well-known of these, and the source of many derivatives, is the Apache log4j package for Java. This has been ported, also as an Apache project, is log4php.

Log4php has saved me countless hours of exploratory debugging. I stand firmly with the growing group of serious developers who assert that if you use a reasonably agile process (with iterative, red, green, refactor unit testing) and make good use of logging, you'll very rarely, if ever, need a traditional debugger.

What does this have to do with Kohana? Well, Kohana includes its own relatively minimalist, straightforward logging facility (implemented as static methods in the core class, grumble, grumble). There's a standard place for such logs to be written to disk, and a nice little 'debug toolbar' add-on module that lets you see logging output while you're viewing the page that generated it.

So I ought to just ditch log4php in favor of the inbuilt logging system when I'm developing Kohana apps, right? Not so fast...

Log4php, as does log4j, has far more flexibility. I can log output from different sources to different places (file, system log, console, database, etc.), have messages written to more than one place (e.g., console and file), and so on. Kohana's logging API is too simple for that.

With log4php, I have total control over the logging output based on configuration information stored in an external file, not in the code itself. That means I can fiddle with the configuration during development, even deploy the application, without having to make any code changes to control logging output. The fewer times I have to touch my code, the less likely I am to inadvertently break something. Kohana? I only have one logging stream that has to be controlled within my code, by making Kohana method calls.

Many experienced developers of object-oriented software are uncomfortable with putting more than one logical feature into a class (or closely-related set of classes). Why carry around overhead you don't use, especially when your framework offers a nice extension capability via "modules" and "helpers"?. While there may sometimes be arguments for doing so (the PHP interpreter is notoriously slow, especially using dynamic features like reflection), I have always failed to understand how aggregating large chunks of your omniverse into a Grand Unified God Object™ pays dividends over the life of the project.

So, for now, I'll continue using log4php as a standalone tool in my various PHP development projects (including those based on Kohana). One thing that just went onto my "nice to do when I get around to it" list is to implement a module or similar add-on that would more cleanly integrate log4php into the surrounding Kohana framework.

This whole episode has raised my metaphorical eyebrow a bit. There are "best practices" for developing in OO (object-oriented) languages; PHP borrows many of these from Java (along with tools like log4php and PHPUnit, the de facto standard unit-test framework). I did a fairly exhaustive survey of the available PHP frameworks before starting to use Kohana. I chose it because it wasn't a "everything including several kitchen sinks" tool like Zend, it wasn't bending over backwards to support obsolete language misfeatures left over from PHP 4, and it has what looks to be a pretty healthy "community" ecosystem (unlike some once-heavily-flogged "small" frameworks like Ulysses). I'm not likely to stop using Kohana very soon. I may well have to make time to participate in that community I mentioned earlier, if for no other reason that to better understand why things are the way they are.

But that's the beauty of open source, community-driven development, surely?

I thought OSS was supposed to break down walls, not build them.

Silly me. I've only been using, evangelizing and otherwise involved in open source software for 15 or 20 years, so what do I know?

In reaction to the latest feature article in DistroWatch Weekly, I'm angry. I'm sad. Most of all, I feel sorry for those losers who would rather keep their pristine little world free of outside involvement, especially when that involvement is with the express intent of not making it easy for non-übergeeks to use open source software — in this case, OpenBSD. OpenBSD, like its cousins FreeBSD and NetBSD, is one of the (current) "major" base versions of BSD Unix. While the latter two have had numerous other projects that have taken their software as a base, fewer have branched from "Open" BSD, as can be seen in this comparison on Wikipedia. Recently, two OpenBSD enthusiasts have attempted to address this, only to be flamed mercilessly for their trouble.

The DistroWatch feature previously mentioned concerns GNOBSD, a project created by Stefan Rinkes, whose goal plainly was to make this highly stable and secure operating system, with a lineage that long predates Linux, accessible to a wider audience (who don't use Mac OS X - based (indirectly) on both FreeBSD and NetBSD).

For his troubles, Mr. Rinkes was the subject of repeated, extreme, egregiously elitist flaming, this thread being but one example. He was eventually forced to withdraw public access to the GNOBSD disc image, and adding a post stating that he did not "want to be an enemy of the OpenBSD Project."

Reading along the thread on the openbsd-misc mailing list brings one to a post by one "FRLinux", linking to another screaming flamewar summarized here, with the most directly relevant thread starting with a message titled "ComixWall terminated. Another hard worker with the intent of creating an OpenBSD-based operating system that was easy for "ordinary people" to use, promptly incurred the wrath of the leader of the OpenBSD effort, Theo de Raadt, with numerous other "worthies" piling on.

WTF?!?

OK, I get it. The OpenBSD folks don't want anyone playing in their sandbox providing access to the OpenBSD software (itself supposedly completely open source, free software) that might in any way compete with "official" OpenBSD downloads or DVD/CD sales. They especially don't want any possibility of the Great Unwashed Masses™ — i.e., anyone other than self-professed, officially-blessed übergeeks — from playing with "their" toys.

OK, fine. It has been a large part of my business and passion to evangelize open source and otherwise free software as an enabler for the use of open data standards by individuals and businesses. I have supported and/or led the migration, in whole or in part, of several dozen small to midsize businesses from a completely closed, proprietary software stack (usually Microsoft Windows and Microsoft Office for Windows) to a mix of other (open and proprietary) operating systems and applications. OpenBSD has historically been high on my list of candidates for clients' server OS needs; if you can manage a Windows Server cluster with acceptable performance and stability, your life will be easier managing almost anything else — provided that you're not locked in to proprietary Windows-only applications and services. In these days of "cloud" computing and "software as a service", the arguments against Ye Olde Standarde are becoming much more compelling.

I just won't be making those arguments using OpenBSD anymore. And to my two clients who've expressed interest in BSD on the desktop for their "power" developers over the next couple of years, I'll be pitching other solutions to them...and explaining precisely why.

Because, out here in the real world, people don't like dealing with self-indulgent assholes. That's a lesson that took this recovering geek (in the vein of "recovering alcoholic") far too many years to learn. And people especially don't like trusting their business or their personal data to such people if they have any choice at all.

And, last I checked, that was the whole point of open standards, open source, and free software: giving people choices that allow them to exercise whatever degree of control they wish over their computing activities. I'd really hate to think that I've spent roughly half my adult life chasing a myth. Too many people have put too much hard work into far too many projects to let things like this get in our way.

Changing Infrastructure - and "Mea maxima culpa *thwack*"

(which I remember from a Monty Python sketch but can't find attribution. Oh well; please feel free to comment and enlighten me.)

About a week ago, I switched my domain hosting for my Web/mail domain, seven-sigma.com, from Simplehost Limited of New Zealand to WebFaction, based in the UK. Now, Simplehost are a good bunch of guys; their customer service is very responsive and you get knowledgeable, thoughtful answers to questions. On most things, they're very willing to work with people to help them resolve issues. If you're looking for a host in this general slice of the planet, put them on your list of providers to have a good look at.

The deal-breaker for me, though, after two years or so with them, was that I need to be able to develop and demonstrate Web sites using the latest and greatest variety of tools, in particular PHP 5.3.x, which introduces important new features (in addition to fixing numerous bugs).

WebFaction, on the other hand, offer PHP 5.3 support (as well as numerous other tools, such as Ruby on Rails, have good pricing on their shared-hosting packages (as does Simplehost, honestly), and fantastic self-help and support options - video tutorials, Twitter feeds, and so on. Also, doing the obligatory Googling for dissatisfied-customer reactions mostly brings up hits like this one, talking about how (except for a couple of brief periods), the only hits were from reviews pointing out how few hits there were. Chatting on IM and IRC with a few customers also helped.

So, a process that I'd been poking at for a couple of months, with three weeks of serious effort, is done with, hopefully for several years to come.

However, there was a silly-me postscript to all of this: For several days after I made the switch, I just wasn't able to get email working. I went onto the WebFaction email-support forums, flipped through a few messages that described solutions to problems other users had had, tried them, and no luck. Then, tonight, I came across the answer that had been staring me plain in the face from the very first message telling me that my account had been set up.

You folks who have your own domains can probably imagine what went wrong; suffice it to say that the difference between what I was reading and what I was thinking was sufficient that over 1,300 email messages have downloaded in less time than it took me to write this post. So if you've been emailing me and wondering why I haven't answered, I apologize. It's unlikely to happen again — hopefully for several years.

And to the Simplehost support guys, thanks very much for your help. This was not in any way at all Simplehost's fault.

So why would I write something like this, showing a fairly major screwup? Because... I'd rather deal with people who admit their faults and publicly commit to do better, than with people who are infallible legends in their own mind. This industry has far too many of the latter sort. I'm hoping that enough other people feel the way I do that I can continue to do great work for my clients.

Thanks for reading.

EDIT:Years on, never mind how many, I discovered that I'd mistyped "WebFaction" when I meant "Simplehost" three paragraphs earlier. Heartfelt apologies to both. Mea maxima culpa *thwack!*

Piling On: More Kibitzing about the iPad

What do I think about the new iPad? Glad you asked. What's that? You didn't, really – or not many of you did. But that's OK, really; at least two other writers argue that we're going from an era of "nearly universal literacy" to "nearly universal authorship", so here's my two rupiah worth.

John Gruber, the blogger behind Daring Fireball, is a justly respected voice on a variety of topics, notably all things Apple. Two of his posts on the iPad, The iPad Big Picture and Various and Assorted Thoughts... are a good starting point if you've somehow just managed to crawl out of a bubble that protected you from hearing anything about it for the past, oh, six months or so, or more crucially the last three days. I also enjoyed reading Stephen Fry's thoughts on the device.

But one bit I would recommend above all to anybody who is thinking about "what does this iPad thing/phenomenon really mean?", or anyone who just cares about free expression, open societies, and all the other progress that humanity has made these past few centuries, should seriously ponder Alex Payne's thoughts On the iPad. Al3x makes some very good, if deeply disturbing, points.

As many people have pointed out, the iPad differs in one critical respect from every personal computing device as computing device that Apple have ever built, from the Apple I on up to the iMac that I am typing this on. The iPad is a device, first, foremost and specifically, for the consumption of digital "content". As James Kendrick says, Apple just want us to push the 'Buy' button, in an endless, mindless Pavlovian dystopia.

"Hang on," you say, "it can't really be as bad as all that. You're just pushing histrionics!" I truly hope so. But consider: Apple has sold every Mac, as well as all their earlier models, on the basis of a small group of generally positive, empowering ideas:

• "It just works better."
• "The power to be your best."
• And, of course, "Think different".

One of the (many) things that make the Mac special is the fact that every single Mac sold comes with a copy of the full development toolset for Mac OS X, right in the box. Anybody who wants to invest the time and effort can become a Mac developer. (The money involved, of course, is in buying a Mac in the first place, but you were going to do that anyway, right?) You. The smart-aleck kid down the street. Your Aunt Tillie. Anyone. That has been one of the core strengths of the Apple Mac platform and product; the ability for anyone, without genuflecting before any sort of gatekeeper, to write anything they can conceive of, using some pretty great tools. And thousands, dozens of thousands have. You don't need to be a big corporation. You don't even need to ask Apple's permission, or use Apple's site to market your creation. All you need is an idea, some persistence, and the willingness to learn.

Apple made a radically different statement with the iPhone. If you want to write a "real" app for the iPhone, you need to submit your bits to the App Store, a process that is the seeming antithesis of open, collaborative or fair. As a practical matter, you need to join the iPhone Developer Program, and pay a fee. Doing so will subject you to various license agreements, limitations of what can be developed, and so on.

Apple justified this by saying "hey, the iPhone is a phone, an appliance. We have to keep some control over things, to make sure that our (non-technical) customers have the best possible experience — and incidentally to ensure that we continue to honor agreements we've made with carriers like AT&T." And the developer community grumbled and moaned, but large numbers went along. And the App Store, by almost any measure, is a raging success in the aggregate — even though most individual developers aren't making much at all.

Now comes the iPad, which has been widely, often dismissively, described as a "super-sized iPod Touch." Which, in several senses, it is. But whereas the "iTouch" is an accessory in one's life, being a combination PDA, music player, and (ostensibly) simple application platform, the iPad promises to be all that and more. Specifically, it's being pitched as this "magical" piece of technology which you'll "always" have at hand. Why? Well, you can listen to music on it, or watch videos, or run the same apps you ran on your iTouch or iPhone along with a new generation of "super-sized" apps. Oh, and Apple did announce that their office-software packages would be available on the iPad — so you can use it for "work stuff."

What about the kind of creative outlets that have been highlighted at each Mac introduction since the 1980s? Well, um, good luck with that. Because, even though this is a "real" computer, it's "really not"; it's an up-sized iPod Touch, which is (officially, mostly) a passive device.

The iPad is being pitched as a "magical", but inherently passive, device that consumers will use to buy (or, usually, rent) "content". There'll be "social media" like Facebook, Twitter and so on; the iPhone already supports those. Anything that can be done entirely over the Web, without the use of plugins like Java or Flash (like blogger.com, is kosher, too. But anything truly creative, "revolutionary", "game-changing", is going to have to survive the App Store gauntlet — which means that it's going to have to be consistent with Apple's view of how the iPad "should" be used. As a passive device which consumers use to buy access to content.

Think "57 Channels and Nothing On," a million times over. Perfect for the top-down, don't-make-me-think society. (I'm sure it will be very popular here in Singapore, for precisely that reason.)

Apple, you can do better. We know, because you've mostly done much better before, and encouraged us, developers and users, to do great things with what you've built for us. The iPad isn't so much a step in the wrong direction, it's a leap of faith worthy of Wile E. Coyote — but we know how far off the edge of the cliff we've gone. And the only way to go from here, is down.

What's wrong with this picture?

Apple chose the only name that ever made sense for their new change-the-world device: the iPad.

So what's not to like about it? Well, that depends on who you are and what you really wanted this to be. No surprise there.

My biggest problem with it was summed up well by James Kendrick in his post, Thoughts on the iPad — Just Push the Buy Button, says Apple: it really is primarily about media consumption - i.e., paying big corporations for temporary, passive access to data. Something to switch your creative intellect off, not on.

And those of us who are often called the "Apple faithful" were hoping, praying if you will, for something so much more. Which in fact this could still be. But for the first time in Apple history, a product with serious computing power is being positioned in the marketplace such that creative, collaborative expression by individuals is almost a subversive act.

We've fallen a long, long way from "Think Different". And it's now much, much harder to argue that Apple is as much a computer company dedicated to creativity and the pursuit of excellence, than to argue that Apple has become "just another" corporate big-media company, helping to turn the Internet from what was once called "the greatest opportunity for expressive, collaborative democratic action in human history" into just another television set. Active creation becomes passive "consumption." All hail the primacy of content, and don't even think about upsetting the applecart. That would be Thinking (too) Differently.

ANFSD: Newman and Redford, Not Streisand and Farrow

The leads in the classic 1973 movie, "The Sting", of course. Both leads were male, playing roles that were prototypically male in our social order. Why is that?

Clay Shirkey has some ideas about that, which he wrote up as "A Rant About Women". Not "A Rant Against Women", mind you. Some did not see that as a difference that makes a difference, notably Zo at this post on her (misnamed) humorlessbitch.com blog. This post, beyond this explanatory header, is my comment to her blog-post-as-comment, the 190th comment in 30 hours. I have slightly reformatted it, as the original was only able to use CAPS for emphasis.

A tip of the hat to Venessa Miemis (@venessamiemis on Twitter) for tweeting about this.

---

Maybe this is because I have the ability to look at this as a Shirkeyesque male who “only told lies I could live up to, and I knew when to stop.” The sorry fact that he’s pointing out – too obliquely – is that many/most arenas of professional and creative endeavour have been created, led and/or taken over by men who fit the portrait that Shirkey is painting. Risk-taking, self-promotion and so on are seen as predominantly “male” character aspects/flaws, in large part because society has been and is fundamentally sexist – and is likely to remain so unless and until women can function within the existing framework successfully enough to alter it.

That’s what I get from reading Shirkey’s piece; not a snarky bit of braggadocio that “men are on top because we know how to game the system,” so much as “until increasing numbers of women in a variety of fields – professional, academic, artistic, and so on – can work within this aspect of the system, it won’t be changed.” Not because change wouldn’t benefit men as well as women. It would. But rather, because those “gatekeepers” who control the system, having got to where they are now by gaming the system, lack the will if not the means to change it.

We are at a turning point in human history, of a kind that hasn’t been seen in well over a thousand years. Institutions and conventions are changing all around us, and, not knowing how to change and survive, they change and die. Others watch the process attentively, determined against reality to avoid the same fate. But if we do NOT come up with a way to fundamentally change our society, to make it more equitable, transparent and open, we risk a new Dark Ages that will compare to the European mediaeval one as a broken fingernail compares to a petrol bomb.

I believe that is what Shirkey fears, as do I.

---

Comments, please.

Fixing A Tool That's Supposed To Help Me Fix My Code; or, Shaving a Small Yak

Well, that's a couple of hours I'd really like to have back.

One of the oldest, simplest, most generally effective debugging tools is some form of logging. Writing output to a file, to a database, or to the console gives the developer a window into the workings of code, as it's being executed.

The long-time "gold standard" for such tools has been the Apache Foundation's log4j package, which supports Java. As with many tools (e.g., PHPUnit), a good Java tool was ported to, or reimplemented in, PHP. log4php is uses the same configuration files and (as much as possible) the same APIs as log4j. However, as PHP and Java are (significantly) different languages, liberties had to be taken in various places. Add to this the fact that PHP has been undergoing significant change in the last couple of years (moving from version 5.2 to the significantly different 5.3 as we wait for the overdue, even more different, 6.0), and a famous warning comes to mind when attempting to use the tool.

Here be dragons.

The devil, the saying goes, is in the details, and several details of log4php are broken in various ways. Of course, not all the breakage gives immediately useful information on how to repair it.

Take, as an example, the helper class method LoggerPatternConverter::spacePad(), reproduced here:

    /**
     * Fast space padding method.
     *
     * @param string    $sbuf      string buffer
     * @param integer   $length    pad length
     *
     * @todo reimplement using PHP string functions
     */
    public function spacePad($sbuf, $length) {
        while($length >= 32) {
          $sbuf .= $GLOBALS['log4php.LoggerPatternConverter.spaces'][5];
          $length -= 32;
        }
        
        for($i = 4; $i >= 0; $i--) {    
            if(($length & (1<<$i)) != 0) {
                $sbuf .= $GLOBALS['log4php.LoggerPatternConverter.spaces'][$i];
            }
        }

        // $sbuf = str_pad($sbuf, $length);
    }

Several serious issues are obvious here, the most egregious of which is acknowledged in the @todo note: "reimplement using PHP string functions." The $GLOBALS item being referenced is initialized at the top of the source file:

$GLOBALS['log4php.LoggerPatternConverter.spaces'] = array(
    " ", // 1 space
    "  ", // 2 spaces
    "    ", // 4 spaces
    "        ", // 8 spaces
    "                ", // 16 spaces
    "                                " ); // 32 spaces

If you feel yourself wanting to vomit upon and/or punch out some spectacularly clueless coder, you have my sympathies.

The crux of the problem is that the function contains seriously invalid code, at least as of PHP 5.3. Worse, the error messages that are given when the bug is exercised are extremely opaque, and a Google search produces absolutely no useful information.

The key, as usual, is in the stack trace emitted after PHP hits a fatal error.

To make a long story short(er), the global can be completely eliminated (it's no longer legal anyway), and the code can be refactored so:

    public function spacePad(&$sbuf, $length) {
        $sbuf .= str_repeat( " ", $length );
    }

Of course, this makes the entire method redundant; the built-in str_repeat() function should be called wherever the method is called.

I'll make that change and submit a patch upstream tomorrow... errr, later today; it's coming up on 1 AM here now.

But at least now I can use log4php to help me trace through the code I was trying to fix when this whole thing blew up in my face.

At least it's open source under a free license.

Virtual Real Work

(As posted in a comment on a very detailed Ars Technica review:

After giving PD5 and Fusion 3 full evaluations, I'm going back to Fusion for another cycle.

A lot of what I do is testing software in odd developer builds of various BSDs, Linuxes and OpenSolaris; Fusion gives me the least trouble when venturing away from the Microsoft megalith. I have XP and Win7 VMs too; I just don't fire up either more than once or twice a month. I typically have 2-3 other VMs up and on my network at any given time.

I also did take a good look at OpenBox; it's the best of the bunch for running OpenSolaris (wonder why?), but for anything else, particularly the oddball Linuxes, it gave me as much trouble as PD 5. Where Parallels would crash like a drunken teenager in a Maserati, OpenBox just would not install several important systems. So much for that.

VMWare isn't as slick in some areas as PD5 (LOVE that one-click 'exclude from TM' feature, and the VM list is nice), but it DOES have one feature that's absolutely non-negotiable: It tends to work better/more often. That's worth the extra US$10 or so for the upgrade. (with an additional new license for my new MBP).

I've put about 20 hours over the past 2 months into formal evaluation of the hypervisors mentioned (along with a couple of hundred hours of just using them to run VMs with which I was working, and not trying to think "is this better than...?"). The pain level with Parallels has been fairly consistent; VMWare only surprised me a couple of times. Parallels also gives you only half the review time that VMWare does before you have to pony up. That tells me that they aren't as confident in their product (and rightfully so); there are problems that merely suggest themselves in 15 calendar days of varied use that you could really get a handle on in 30. Not having confidence borne of experience in whether I can solve subtle issues as they arise wasn't the only reason for rejecting Parallels Desktop 5, but it certainly didn't help.

The impression I get from using both products, from reading what has been written and from chatting with other users, is very consistent. VMWare portray themselves as a company that tries, and often succeeds, to do things well - starting with a truly usable, stable, versatile virtualisation platform. Parallels, on the other hand... by bringing out releases well before they're really ready, and with a shorter evaluation time, give the impression that they're desperate for your money. Give it to the company that actually earn it.

Address Book him, Danno!

This is a follow-up to a tweet I left yesterday, where I was praising a great little app called ABMenu. This little guy just sits in the system menu bar and gives you easy, two-click access to any entry in your Address Book. I'd installed it earlier on my main iMac. This morning, I noticed that it wasn't installed on the new, two-week-old MacBook Pro, and fixed that.

Then of course I noticed that the new system didn't have all the entries that the old one did. The customary way of syncing data like this between Macs is to use Apple's MobileMe service; the Mac can also use LDAP or other enterprise-style directory services. "Fine, " I thought, "this definitely sells a lot of MobileMe subscriptions." (In fact, getting one is on my New Year's resolution list; the reasons for the delay have been Complicated™.) Continuing the thought process, " There's gotta be a dedicated app out there that will let me sync just Address Book."

A couple of minutes browsing VersionTracker brought me to address-o-sync, which does just what it says on the tin. Obviously reasonable but slightly irritating irritation: the app has to be installed and running on both Macs in order to sync; address-o-sync can't reach across your local net and pull "raw" Address Book data. (It compensates nicely, with a couple of features that would be impractical otherwise.)

So, I've updated my "how to bring up a new Mac" checklist and starred both ABMenu and address-o-sync on my installed-app lists. Take a look, and see if you agree.

(Now if I could only get the Hawaii Five-O theme song out of my head.)

The Audacity of Hype

This is the content of a comment which I posted to a TechCrunch article on The World Doesn't Need Someone Telling Us What We Don't Need In Tech. For what it's worth, I hope the iSlate/iPad/iTablet/iWhatever is real this time; if we have another Charlie Brown, Lucy and the Football moment, I think the market will see this as one of Apple's rare blown opportunities - to the tune of maybe $50 off the share price.

I'd be interested in your thoughts.

---

I also disagree with the "comfort zone" limitation idea; cf. Shaw and the Unreasonable Man. Apple's successes have all been revolutionary innovations; the few times that a "me too" product has come out of Cupertino (Apple III, Lisa), it's sunk at high warp speeds.

If the iPad is even halfway affordable (by Apple standards, anyway), they'll sell like iPods on steroids; I'd certainly buy one. Even having a big, beautiful iMac and a trusty MacBook Pro, I can see several useful areas for an iPad. More-convenient-than-laptop media browsing and ebook reading is the one everybody latches on to. But I could take an iPad onto the train every morning and get stuff done - standing on a lurching train filled with people from cultures that don't know how to move in crowds efficiently. I could take it into meeting-room meetings and hallway meetings more easily and effectively than a full-clamshell laptop. And it would probably be easier to gather a couple of people and run through a quick presentation with an iPad than a laptop, too. (Think "elevator pitch.")

Sure, it would fill a niche and scratch a few itches that I already know about. But for it to be truly revolutionary, it will need to find uses that I/we can't really predict yet, because we've never really had something capable of filling them - we've just sort of worked around limitations we weren't really aware of. (See 'iPhone' for previous example.) I've got more faith in Apple to pull something that revolutionary out of their hats than I do in any other tech company today — or in the 30 years I've been in this industry.

We ain't seen nothin' yet.

ANFSD: The Present Isn't Always Perfect

If anybody has had doubts about the decline and fall of the English language over the past two or three decades, current media (print and online) should clear that up very nicely.

If I see another reviewer who starts off with "I've had the (Product X) for a week and I am loving it," it will be very difficult to restrain myself from throwing a brick through the display. There's this widespread tendency to use the present perfect in place of the simple present.

Part of this, I understand, is simple cultural differences. Many of these writers either come from or work in an environment heavily influenced by South Asian variations of British English. Somebody in a position of educational or institutional power, somewhere along the line, decided that if the "present perfect" was indeed "perfect", then it was better to use that in any conceivable sentence.

This influence is relatively new in the larger English-literate world. I don't remember seeing this style when I started writing seriously 30 years ago, or even during the time of my first real multi-national distributed development team back in the early 1990s. People had varying levels of language ability, just as we all had varying abilities in other areas. One of the great things about those early teams was that each participant clearly understood that they had something to learn; that other teammates had skills and abilities that they could learn from. Since that time, the offshoring/outsourcing industry has devolved into what too often seem a series of sealed bubbles bumping against each other, but that's a topic for another post. Just let me say that, as has happened so many times before throughout history, language patterns mingled (or 'mangled', depending on your views). Absent widespread, effective English language instruction in American schools over the last 30 years (yet another topic unto itself), many younger American technical folks adopted the "new", "cool", "exotic" English usage borrowed from their peers. And since the software (and Web) industry is unrelentingly ageist, at least as much as, say, theoretical physics (where, I've been told, you're over the hill by your mid-20s), what the "cool kids" were doing quickly became the norm for the industry.

It may well be that "all this has happened before, and all this has happened again." That does not make it any more comfortable, or comforting, to those who are aware of how imperfect the present really is.

Oh, and by the way, Happy New Year! One is sorely tempted to say "It can't possibly be as bad as 2009 was..." Don't give in to the temptation. $DEITY tends to take that as a personal challenge... one that you or I can't possibly win.

Blast from the Past

Another in a continuing series...

Microcomputer(as PCs were called before the IBM PC) veterans of a certain vintage well remember that most counterintuitively productive of productivity tools, WordStar 3.3 (and earlier). The hegemon of its day, WordStar used what at first (and usually fifth) inspection appeared to be whimsical, arbitrary key combinations for commands. Ctrl K-H for Help was invariably what new users first memorised. All through the 1980s and well beyond, any word-processing software that came onto the market had some degree of WordStar-compatible commands, either as their main command set or as a bolt-on to wean folks onto the "new" way of doing things. This was even true for the first several releases of WordPerfect and of Microsoft Word. (Word today has several available add-ons to add WordStar command compatibility.)

Why was this so popular? As noted in the Wikipedia article:

...the "diamond" of Ctrl-S/E/D/X moved the cursors one character or line to the left, up, right, or down. Ctrl-A/F (to the outside of the "diamond") moved the cursor a full word left/right, and Ctrl-R/C (just "past" the Ctrl keys for up and down) scrolled a full page up/down. Prefacing these keystrokes with Ctrl-Q generally expanded their action, moving the cursor to the end/beginning of the line, end/beginning of the document, etc. Ctrl-H would backspace and delete. Commands to enable bold or italics, printing, blocking text to copy or delete, saving or retrieving files from disk, etc. were typically a short sequence of keystrokes, such as Ctrl-P-B for bold, or Ctrl-K-S to save a file. Formatting codes would appear on screen, such ^B for bold, ^Y for italics, and ^S for underscoring.

Although many of these keystroke sequences were far from self-evident, they tended to lend themselves to mnemonic devices (e.g., Ctrl-Print-Bold, Ctrl-blocK-Save), and regular users quickly learned them through muscle memory, enabling them to rapidly navigate documents by touch, rather than memorizing "Ctrl-S = cursor left."

Why is this relevant (or even interesting) today? Besides the lessons to be learnt about interface design, it's interesting to note how many editors out there still pay homage to WordStar. I stumbled across joe again this morning; it's available on essentially all Linux and BSD distributions, with versions built for other systems as well (e.g., Mac OS X and Cygwin/Windows), and source freely available if your platform isn't yet supported or you just like to tinker around on one that is.

What makes this fairly scary for us old-timers is just how quickly the old finger habits come back. If you had more than a year's experience beating your head against the original WordStar, I dare you to work with joe or its ilk for more than a few minutes before "how do I do...?" completely falls away from your thoughts and you're just typing as fast as you can think.

For that's the real beauty of this type of "primitive", what-you-see-isn't-what-you-get interface: you're not distracted by the ephemera of making your work appear "just so", and can actually focus on the work of writing. And that, in our click-and-drool modern interfaces, is what we've lost -- and no amount of clever code wizardry on the part of the interface designers can bring us back to that. Why? Because of basic human nature - if we see a button, at some point we'll want to push that button - "just to make things look better." And, all of a sudden, we notice that the entire morning has flown past while we were focusing on the first three paragraphs of a major report that's due this afternoon. Oops.

There's a reason why almost every tool aimed at professional writers -- people who make their living at x cents per word -- have "stripped down", minimalist interfaces, at least as an option. It's the same reason that far too many truly "old-school" writers give for writing on paper and then typing (or having someone type) their words into a computer: the fewer distractions you have, while still being able to do what you're trying to do, the more productive you'll be at it.

That concept extends far, far beyond the writing of prose -- and has too often been lost or forgotten in those other areas as well. Pity.

When Standing on the Shoulders of Giants, Don't Trip

I've been doing a lot of software installation lately, on Mac OS X and various BSDs and Linuxes. Doing so reminds me of one of the major banes of my life when developing for Windows. If you're a Windows usee, you're acutely familiar with the concept.

DLL Hell.

On Windows, as you install a dozen apps, each of them will try to install their own versions of many system-wide (or vendor-wide) libraries. The poster child for these is msvcrt.dll, the Microsoft C Runtime Library, upon which virtually everything in or running under Windows depends in some fashion. Many unhappy man-millenia have been spent by admins and PC owners the world over trying to resolve compatibility issues, usually introduced when a newly-installed app overwrites a more recent version with the earlier version that app shipped with. Other things — including other libraries the app may depend on or even Windows itself — may break since features and bug fixes they rely on (present in later versions of the library) are suddenly gone.

Why think about this, when I'm as likely to write (or even use) any Windows software in the next month as I am to win the local lottery without buying a ticket?

Because, one of the things I added to my new MacBook Pro was the MacPorts software, which is a Mac analogue to the BSD ports collection. The Linux equivalent to this is (more or less) your package manager, whether it be aptitude or yum or portage or whatever. Windows has no equivalent concept; every app you install is its own universe, or thinks it is (with the "benefits" noted earlier), and no central one-stop "update everything" capability.

Modern operating systems may not have DLL hell, but they do tend to have numerous different versions of different libraries and support frameworks installed. Since those can be accessed by apps as either "give me the most recent version" or "give me version x.y.z", no equivalent to DLL hell takes place. But it does tend to eat up disk space. And it makes it harder to mentally keep track of what's installed; installing the MacPorts edition of Bluefish, an HTML editor, installs some eighty other "ports" — direct or indirect dependencies of Bluefish (which itself is primarily a Linux app). Some of these dependencies are last-month recent updates; a few are several years old. But MacPorts determines which packages are needed by Bluefish (and its dependencies, and...) and installs the latest version (by default) of those dependencies, or the specific version asked for. Thus, several files with the same basic name, but different version numbers encoded into the filename, can coexist peacefully in virtually any modern OS.

Recent Microsoft system software, particularly the "managed" software using the .NET Framework, avoids most of the cavalier instances of DLL hell, but introduces other quirks and brittleness in compensation.

But I'm still a bit unnerved by a software package — any package mdash; that has 80+ designated dependencies. I'm certainly thankful that I don't have to manage that dependency matrix myself.

If modern software development is "standing on the shoulders of giants", to appropriate a Newton paraphrase, then careless dependency introduction would be the functional equivalent of drunken tap-dancing: almost impossible not to trip.

Reuse, Renew, Recycle - Data Structures Edition

Everything gets recycled these days, including acronyms...

Whether you're developing Web pages in your language of choice, writing a new game to take on the world, or (insert your project here), at some point, you're going to deal with structured data.

You're quite likely, at some point, to deal with fairly complex, nested structured data - for configuration, to preserve the state of objects within your software, or whatever. In recent years, "complex, nested structured data" has often been coded up in XML. This can save your coding bacon; there are quite a few nice tools out there to work with XML, insulating you from having to deal with the "raw" <tag ...="" attribute="value">data<nested_tag>more data</nested_tag></tag> soup.

But at some point you may well end up having to hand-edit the thing — you may want to pull something out for reuse, or you suspect that one of the tools you're using is messing up in some way... it happens. And even during that bright, wonderful time when you're building a POC (proof of concept/ possibly overly c***tastic; see, I said we could reuse), you wind up humping a lot of "extraneous" data around that just really wraps a few small things. Ugh.

Even with compression, you're still taking a hit; you've just moved it from the pipe to the CPU. Twice.

I got handed a client project a little over a year ago that used this nifty notation called YAML. While YAML was apparently originally developed for Ruby apps, it's evolved; there are now support tools/libraries for everything from PHP to Visual Basic to Haskell. "YAML Ain't Markup Language," but it can express very complex data structures; nesting is defined by indentation, item types by decorations. (See the spec or this PHP implementation for more detail.)

For projects which have been using binary data storage for non-BLOB data, simply because XML was seen as overkill, this might well simplify things. Likewise, if you're pushing data around between different apps/platforms, thinking "I maybe oughta look at JSON", this is for you.

I've been using it primarily in PHP to encode various data and content models, and it has allowed me to dramatically simplify my code in several places - reducing complexity, inefficiency, and possible bugs. The Ruby and Python crowds have been using this for a while; PHP folk have tended to look at it suspiciously, since it came from Over in Those Languages.

---

Good tools work in any language.

---

Reuse? I was talking about recycled acronyms earlier?

At about the same time I found YAML the non-markup-language markup language, I also came across a nifty CSS framework with the highly original name of YAML. This code, which makes it pretty easy to style sites using Yet Another Multicolumn Layout, is a set of CSS (2.x) stylesheets, with minimal graphics and JavaScript thrown into the box. What it gives you is a nice, consistent layout on just about any browser (including hideously-broken variants of MS Internet Exploder) which you can slice and dice pretty much at will. It even degrades sensibly for non-CSS, non-JS browsers — like the Googlebot. If you know how to read CSS reasonably well and this takes you more than an hour to grok, you're working too hard. Browse the docs, the samples, the tutorials, or the online community rumblings, and you'll be productive in no time.

And yes, you can use both together on the same Web dev project. Just make sure you keep your tools straight. After all, YAML is not at all the same thing as YAML.

Two steps forward, three steps back

Alternate title: ARRRRRGGGGGHHHHHHH!!!!!

Both of you Gentle Readers may have noticed that I've been away from the blog for a while, and that a few posts that were previously published have gone missing. I've been busy fighting some other fires for a while, and my current network access has lacked the stability and efficiency that local propaganda would have you expect.

This evening (Wednesday 18th) I came across a piece of nifty-looking software, MacJournal by Mariner Software. It looks great — software that would let me compose/revise blog posts offline, in a native Mac app with nice organizational features and so on, at an attractive price, and with a 15-day evaluation period thrown in, just so you can try before you buy.

"Cool," I thought; "I'll be able to multitask on my shiny new MBP that's coming Any Day Now™."

Downloading and installing the eval copy went just as you'd expect; drag an icon into a folder, wait for the "Copying" progress dialog to go away, and it's done. Standard Mac user experience; nothing to see here, folks — unless you were expecting a Windows-style "Twenty Questions" installation.

I decided to do a really simple, trivial first exercise: select the five posts I'd written so far in a tutorial series; add the keyword ("label" in Blogger.com parlance) tutorial to them; save them back to Blogger. No animals were harmed in the performance of this experiment, and very explicitly, no content was directly, intentionally edited. (Note the qualifiers; they're important.)

(insert "train wreck" sound effects here.)

The first two parts were (relatively) unmolested; they didn't have any code blocks in them. The latter three did, however, and those were completely deformed. Numerous span elements were added, particularly around links (MacJournal seems to think links shouldn't be underlined, ever). Other formatting was changed; in particular, code tags were replaced by spans that set the font size to 13 points.

WTF?

It's going to take me a bit of noodling around in the software to figure out how to change the defaults to something that makes sense (at least for me), and until then, I'm back to editing in the browser. If the evaluation period expires before I'm happy with the configuration, then I'll comply with the license and blow it off my system. I'd really rather not do that; the feature list looks good, the interface is clean, and best of all, I don't have this Could not contact blogger.com line underneath my editing area as I type.

I understand that MacJournal, like most apps, has default ways of laying things out and working with things. I'm well aware of the difference between an "import" and a "copy" of something. But... I believe very strongly that the first rule of software, as medicine, should be "First, do no harm" — and that includes "don't mess with my formatting without even putting up a confirmation dialog asking my permission!" I really don't think that's too much to ask, or too hard to implement — and doing so would a) make a much more positive initial user experience by b) showing that you've thought things through well enough that c) your still-potential user isn't looking at an hour or two of careful, detailed work just to get back to where he was before he touched your product &emdash; or, rather, it touched his work.

Like most Mac users, I've gotten spoiled by how well most software on this platform is thought through to the tiniest details. Like most, I get annoyed when I have to deal with Windows or Linux apps that simply aren't thought through at all, apparently. (Spend a week with Microsoft Office or, even better, Apple iWork on the Mac; I dare you to go back to Office on Windows and be happy with it.) To run into a Mac app that fails such a simple use case so spectacularly (granted, in its default configuration) simply beggars explanation.

You want to start a tutorial; well, you know...

Not as catchy as the Beatles' Revolution, even if the meter works.... oh well....

Continuing from the first post in this tutorial. What do I think is important when starting to demonstrate some code? As with most writing, it depends on the audience. For the purpose of this series of posts, I'm assuming that you fit comfortably in or near the following:

• You're comfortable with HTML and XML doesn't make you run screaming from the room;
• You have a basic understanding of databases; you've run across SQL before and understand the basic concepts;
• You understand PHP; you've written some code before;
• You understand the concepts of "object-oriented development", "patterns", "best practices" and ideally "test-driven development" (usually abbreviated as "TDD"), even though you may not have loads of experience (yet) with them; and crucially
• You want to improve your ability to write code that you can refine and possibly reuse over time.

The assumption that you know or at least are interested in PHP is a given, since that's the language we'll be using here.

What will you need to have installed and available to follow along?

1. Access to a system with PHP 5.2 or higher, available both from the command line and the Web server (via a module or CGI);
2. The PHPUnit and MDB2_Driver_mysql modules installed and available;
3. A text editor of your choice;
4. The ability to create PHP scripts and HTML files and have those accessible from the Web server as well as the command line.

These should all be pretty obvious to more experienced PHP developers, but making sure that we're both operating from the same set of assumptions — and no others — greatly reduces the likelihood of confusion and breakage along the way. Many of you haven't yet dealt much with unit tests using PHPUnit or similar systems; that's going to be a starting point for us.

Time v. Money v. Risk v. Frustration

I just spent three hours beating my head against The Joys of Wi-Fi Networking. No doubt because it saved $50 or so, the HDB public-housing flat I live in here in Singapore (which are "affectionately" known as 'chicken coops' for their quality and structural integrity) didn't put telephone jacks in every room. Nor did they run conduit between rooms so that suchlike could be added later. The upshot of this is that my DSL modem, in the living/dining room, is 15 meters and two concrete-slab walls away from my study/office, in a bedroom (with no phone jack and two power outlets).

Up to now, this hasn't been an insurmountable problem, because the DSL modem/802.11g router the government-linked telco sells you when you open a DSL subscription could punch a (barely) usable signal through those two concrete walls; desktop and laptop computers strewn around the study were on the LAN, with quite serviceable WPA2 encryption to keep private bits private. The router (in the living room) claimed to have a 14 Mbps connection to The Net; the aggregate total bandwidth available in the study ranged from 3 to 5 Mbps; not fantastic but usable. The world was in a survivable state of chaotic flux.

Then, yesterday morning, FedEx delivered a bright, shiny (actually used, in a Target bag rather than original box, no longer in the Polycom US catalog but still serviceable) Polycom SoundPoint 501 SIP phone, courtesy of my newest client (thanks, Nathan, Matt and Margaret - I'm not kvetching, honest!). An hour of work with jackhammer and flamethrower cleared sufficient space on the desk for the new trophy. Now to plug it in and fire it up.

Plug it in? To what? My ear?

Just to make sure I had my head straight, I looked at what it would take to run line from point P to R. Two solid concrete/rebar walls, routing around various doors/windows, across the span of the living room.... no, that wasn't going to happen, certainly not by Thursday. Put the phone in the living room? Not a chance. Wait a minute - they have these things called wireless Ethernet bridges; I should be able to get one of those, stick it in the study, connect the phone to it and it to the existing wireless LAN, and I'm good to go.

I went out to the local "IT super-mall", Funan. It's the best place in Singapore to buy electronic gear of whatever variety from reasonably reputable dealers. After visiting some 25 different shops, large and small, I was firmly reacquainted with one of the basic facts of Singapore life.

This is a firmly Stalinist economy in the areas that matter. All imports (which is to say, anything of value) are brought in through a small number of generally well-connected exclusive distributors. When one shop says "finished, already!" (the de facto motto of the city), everybody is going to say the same thing. That is, if you're fortunate enough to even run into some salesperson who actually understands what you're looking for. I could find things easier (and often cheaper!) in Vietnam.

Then I happened to pop into one hole-in-the-wall one-man shop, Crystal Systems. The "one man" said "say, I understand what you're trying to do; I did exactly that for a firm here recently; all you need are two routers." The archetypal light-bulb moment. (Remember, I'm a software craftsman, not a hardware/network engineer.) This is where the "time v. money v. ..." of the title comes in. What I should have done was to buy two identical brand-new 802.11n routers. That would have cost me about S$190 (~US$136 or so), and would have given me a perfect excuse to replace my aging 2Wire 2700HGV-2, But this has already been a bit of an expensive month - new hard drive, various software - so after some consideration I decided to try to salvage the existing router (apparent POS though it may be) and just spend the minimum necessary. So I went home with one S$69 TP-LINK TL-WR340G router.

It features the now-customary browser-based setup - just remember to use a hard-wired link rather than wireless (d'oh!), and it's almost as easy as falling off your seat. Bridging mode is pretty obvious; the TP-LINK wants to know the MAC address of the router you're connecting to, along with its encryption setup.

In bridge mode, the TL-WR340G only supports the obsolescent WEP encryption, not the current and generally superior WPA2. The 2Wire - along with every device currently connected to it, obviously - uses WPA2, and does not support a mixed WPA2/WEP environment.

Back to the store I go tomorrow; the only question now is: one new Netgear N router (which can fall back to 802.11g) or two?

Responding while Forbidden; Gender and Other Issues in OSS and PHP

This post is what would have been a response to a post on Elizabeth Naramore's blog, which she titled Gender in IT, OSS and PHP, and How it Affects Us *All*. Quite a good post, actually, with a long and often thoughtful (but as often thoughtless) comment thread following. I was hoping to respond to a comment on the post, but that apparently is no longer allowed, even though the "Leave a Reply" form at the bottom of the page is functional. There's also a mysterious "Login/Password" pair on the page, but no indication of which ID one uses, or how to go about getting one.

Following is the content of the reply-that-wasn't: I (perhaps unreasonably) think this has some points worth pondering. Please do read the original post first and then come back here - where the "comment" feature definitely does work.

---

@A Girl: Great that you're doing AP CompSci next year. As someone who's been in the Craft for 30 years, I have a great sentimental attachment to your idea that "teachers and professors have the chance to shape the mindsets of their students towards women in the industry". If we were a true profession, where essentially all practitioners have a certain common level and content of educational background combined with qualifying experience (e.g., apprenticeship, internship), I'd agree wholeheartedly.

The fact is that many if not most of the people in the industry - both the "coders in the trenches" and the ex-coders who got promoted into management ("because they were such great coders" - thereby removing two qualified people from an organization)... far too many of these people have noformal education in CS (or, often, anything else). And by the time they realize how important that might be, they're old enough that they're facing ageism in the workplace already - they're not confident enough to put the "big hole in the middle of [their] career" and "go back" to school. It doesn't help that schools the world over do such a lousy job of outreach and marketing to those potential students - they're focused on the Executive MBAs and other graduate-level returning students, who can have their pricey programs paid for by their employer. Joe or Jane Schmuck trying to keep head above water in the face of cut-throat competition from planeloads of new arrivals with mimeographed certifications, who've been taught their entire lives to never think out of the box to begin with... things start getting really tough out there. I'm not surprised that enrollment in CS programs is down. I'm amazed beyond words that it's still as high as it is; a less starry-eyed observer might expect the number of CS majors to closely track, say, majors in Phoenician economics.

A lot of the new entrants into CS and IT over the last ten years or so have degrees - they're just not in the "obvious" field. Someone, and I wish I could find the original, wrote an article in one of the industry mags (like C/C++ Users Journal, not IEEE Computer) that, to be good at software in the modern era, one needed to have exposure to "behavioral science, psychology, linguistics, human factors, sociology, philosophy, rhetoric, ethnology, ethnography, information theory, economics, organizational politics, and a dozen other things - and please, please learn to write competently in English!" - I've had that taped above my display for years now. So it's not that we're not educated; the problem - and it is a problem - is that there is no universal common body of knowledge for software "engineering" - which is one of the necessary precursors of any true profession. We're not going to have a CBOK without either a broad consensus within the industry, or imposition of a system from outside (and very narrowly-focused) forces in government or the larger economy. Given the prevailing social and political attitudes of current practitioners ("herding libertarian-poseur cats" is a phrase not infrequently heard), that would seriously disrupt the Craft and, by extension, any industry or field dependent upon software (which by now is pretty much everything).

How to solve the problem - and, in so doing, help redress the pandemic sexism, racism and ageism (in huge parts of the world, recruiting with explicit age limits is perfectly legal, and here in South Asia, you're old for coding at 28)? I've got no idea. When I first started doing this, I thought that within the next thirty years or so (from 1979), we'd be able to turn this informal craftwork that had taken the industry away from the "educated CS types" and turn it into a real profession. Now? I'd say we're 30 to 50 years away from now - unless we have a software equivalent of the New London School explosion and a "solution" gets imposed from outside. We need to grow up, and quickly.

We Interrupt This Program...

We interrupt this tutorial to interject an observation about goals, methods and promises. Goals we have for ourselves as people and as professionals; the method we use to pursue those dreams; perhaps most importantly, the promises we make, both to ourselves and to our customers, about what we're doing and why.

I consider this after reading the Website for some consultants who've done some (relatively nice, relatively low-key) link-dropping on LinkedIn. I'm not naming them here, not because I don't want to draw attention to them (their own site is very clean and well-done), but because the point that I'm going to be making here isn't just limited to them - we, as a craft and an industry of Web development, have some Serious Problems™.

The "problem" is nicely summarized by this group's mission statement:

Our mission is to produce the perfect implementation of your dreams.

What could possibly be wrong with that?

As a goal, implied but left unspoken, absolutely nothing; both as practitioners and as clients, we tend to set ever-higher goals for ourselves. Indeed, that's the only way the "state of the art" - any "art" - can advance. But we who practice the art and craft of software (including Web) development (as opposed to the engineering discipline of hardware development) have a history of slashed-beyond-reality schedules and budgets coupled with a tendency for stakeholders not to hear "if all goes well" as a condition to our latest schedule estimate. We have a history, perceived and actual, of promising more than we can deliver. Far more attention is paid by non-technical people to the "failures" and "broken promises" of software than to things done right. For a craft whose work is accruing increasing public-policy and -safety implications, the effect of unrealistic expectations, brought about by poor communication and technical decisions being made by people who aren't just technically ignorant but proud of the fact, is disturbing. What started as a slow-motion train wreck has now achieved hypersonic speeds, and represents a clear and present danger to the organisational health and safety of all stakeholders.

I don't mean to say that projects always fail, but an alarming number of them do. If, say, dams or aircraft were built with the same overall lack of care and measurable engineering precision that is the norm in commercial software development, we'd have a lot more catastrophic floods, and a lot few survivors fleeing the deluge by air. When I entered this craft thirty years ago (last May), I was soon led to believe that we were thirty to fifty years away from seeing a true profession of "software engineering". As a time frame beginning now, in 2009, I now think that is almost laughably optimistic.

Why have things gotten worse when we as a tool-building and -using society need them to get better? Some people blame "The Microsoft Effect" - shipping software of usually-dubious quality to consumers (as opposed to 'customers') who have bought into the (false) idea that they have no realistic choice.

It's more pervasive than that; commercial software development merely reflects the fashion of the business "community" that supports it, which has bought into one of the mantras of Guy Kawasaki's "The Art of Innovation", namely "don't worry, be crappy." Not that Kawasaki is giving bad advice, but his precondition is being ignored just as those of other software people have been: the key sentence in his "don't worry, be crappy" paragraph is "An innovator doesn't worry about shipping an innovative product with elements of crappiness if it's truly innovative" (emphasis mine). In other words, if you really are going to change the world, nobody will notice if your Deus ex Machina 1.0 has clay feet as long as you follow up quickly with a 1.1 that doesn't...and follow that with a 2.0 that changes the game again. But that space between 1.0 and 1.1 has to be fast, Kawasaki argues (in the next paragraph, titled "Churn, Baby, Churn"), and the version after that has to come along before people (like possible competitors) start saying things like "well, he just brought out 1.1 to fix the clay feet in 1.0." If the customers see that you're bringing out new versions as fast as they can adapt to the previous ones, but that each new version is a vastly superior, revelatory experience compared to the earlier release that they were already delighted by, they'll keep giving you enough money for you to finish scaling the "revolutionary" cliff and take a (brief) rest with "evolutionary" versions. Business has not only forgotten how important that whole process is to their continued survival, but they've removed the capability for their bespoke software (and Web) infrastructure to use and reuse that model. All that remains is "it's ok if we ship crap; so does everybody else." That's the kind of thinking that made General Motors the world-bestriding Goliath it is today - as opposed to the wimpy also-ran it was (emphatically NOT) half a century ago. We really don't need any more businesses going over that sort of cliff.

What we do need, and urgently, are two complementary, mutually dependent things. We need a sea change in the attitude of (most) businesses, even technology businesses, towards software - to realise and acknowledge that the Pointy-Haired Boss is not merely a common occurrence in the way business manages software, but actively threatens the success of any project (and business) so infested. Just as businesses at some point realise that "paying any price to cut costs" is an active threat to their own survival, they need to apply that reality to their view of and dealings with the technical infrastructure that increasingly enables their business to function at all.

Both dependent on that and as an enabler of that change, the software and Web development industry really needs to get its house in order. We need to get away from the haphazard by-guess-and-by-golly estimation and monitoring procedures in use by the majority of projects (whose elaborate Microsoft Project plans and PowerPoint decks bear less and less resemblance to reality as the project progresses) and enforce use of the tools and techniques that have been proven to work, and have an organised, structured quest to research improvements and New Things.. Despite what millions of business cards and thousands of job advertisements the world over proclaim, there is no true discipline of "software engineering", any more than there was "oilfield engineering" in widespread use before the New London School explosion of 1937. Over 295 people died in that blast; we have software-controlled systems that, should they fail, could in fact hurt or kill many more - or cause significant, company- or industry-ruinous physical damages. We should not wait for such an event before "someone" (at that point, almost certainly an outside governmental or trans-governmental entity) says "These are the rules." While I understand and agree with the widespread assertion that certification tests in their present form merely demonstrate an individual's capability to do well on such tests, we do need a practical, experiential system - probably one modelled on the existing systems for engineering, law or medicine. Not that people should work 72-hour shifts; there's enough of that already. But rather that there should be a progression of steps from raw beginner to fully-trusted professional, with a mix of educational and experiential ingredients to ascend that progression, and continuing educational and certificating processes throughout one's entire career. The cost for this is going to have to be accepted as "part of the system" by business; if business wants properly competent engineers, and not just the latest boatload of unknowns with mimeographed vendor certs, then they're going to have to realize that that benefit does not come without cost to all sides. The free ride is over - for all the stakeholders at the table.

Tutorials, best practices and staying current

A gent by the name of Brian Carey has written a very nice little tutorial on "Creating an Atom feed in PHP", and gotten it published on the IBM DeveloperWorks site. In the space of about ten pages, Brian gives a stratospheric overview of what Atom is and why PHP is a good language for developing Atom-aware apps, and then gets into the tutorial - defining a MySQL database table to hold the data used to 'feed' the Atom feed, and writing code to get the data out and put it into the form that a reader such as NetNewsWire expects an Atom feed to be in.

Now, to be fair, Brian describes himself as "an information systems consultant who specializes in the architecture, design, and implementation of Java enterprise applications", and the paper is clearly meant as a whirlwind tutorial, not to be taken by the careful/experienced reader as necessarily production-quality code. And if this had been published in, say, 2002 or so, I'd have thought it a great how-to for banging out some PHP 4 code. But this is 2009 PHP 4 is a historical artifact, and blogs and industry journals of seemingly every stripe are decrying the poor quality (security, maintainability, etc.) of PHP code... much of which is still written as if it were the turn of the century, ignoring PHP 5's numerous new features and the best practices that both spawned them and grew from them.

So what's really wrong with doing things like they did in Ye Olden Tymes™?

• Procedural code makes it harder to make changes (fix bugs, add features, change to reflect new business rules) and be certain that those changes don't introduce new defects. This is largely because...
• While it is possible to test procedural code using automatable tools like PHPUnit, it's a lot harder and more complex than testing a clean, object-oriented design.
• Hard-coding everything, interspersing 'magic values' throughout code, is a major hindrance to future reuse - or present debugging;
• Using quick-and-dirty, old-style database APIs exposes you to the risk of input data that is more 'dirty' than it should be - opening the door to SQL injection and other nastiness;
• Not staying current exposes your code to the risk that it's either using features that have since been deprecated or removed entirely, or (arguably worse), the risk that new features (such as standard library additions) make some of your existing code redundant at best.

Each of these, to varying degrees, is true of much of the PHP code I've read in the last couple of years, including that in the DeveloperWorks paper. For example, the DW paper's code makes use of the old-style PHP4 mysql_* API rather than the more abstract/portable MDB2 database abstraction layer. And there's also the rather terse implementation of a date3339() function for converting a timestamp to RFC 3339 output format, that's now nicely handled through the standard PHP DateTime class.

How would I have done things instead? Read the next few posts in this blog to find out. And, of course, comments are always welcome.

Smokin' Linux? Roll Your Own!

As people who've encountered the "business end" of Linux have known for some time, the system (in whichever distribution you prefer, greatly rewards (some would say 'requires') tinkering and customisation. This can be done with any Linux system, really; some distros, like LinuxFromScratch and, to a lesser degree, Gentoo and its derivatives, explicitly assume that you will be customizing their base system according to your specific needs.

Now the major distros are getting into it. There have been various Ubuntu and Fedora customisation kits on the Net, but none as far as I can tell that are as directly supported (or easy to use) as from OpenSUSE, the "community-supported" offering from Novell (who also offer SUSE Linux Enterprise Desktop and Server.

Visit the OpenSUSE site, and prominently visible is a link to the OpenSUSE Build Service, which "allows developers to package software for all major Linux distributions", or at least those that use rpm packaging, the packaging system used by Red Hat, Mandriva, CentOS, and other similar systems. But that's not all...

SUSE now have a new service, SUSE Studio, which allows users to create highly customized systems based on either the community (OpenSUSE) or enterprise versions of SUSE Linux. These "appliances" can be put together on the basis of "patterns", such as lamp_server (LAMP, or Linux/Apache/MySQL/PHP Web server) or technical_writing (which includes numerous tools like Docbook). You can even supply your own (either self-built or acquired elsewhere) RPM packages to include in the appliance you're building, and SUSE Studio will deal with the dependency matching (warning you if packages are required that aren't either among its standard set or uploaded by you).

Startup scripts, networking, basically anything that is usually handled through the basic installation or post-installation configuration - all can be configured within the SUSE Studio Web interface.

And then, when you've got your system just the way you want it, you can build it as either an ISO (CD/DVD) image to be downloaded and burned onto disc, or as a VM image for two of the most popular VM systems (VMWare and Xen).

But wait, there's more...

Using a Flash-enabled browser, you can even "test drive" your appliance, testing it while running (transparently) in an appropriate VM hosted within the SUSE Studio infrastructure. Especially if you have a relatively slow connection, this will let you do preliminary "smoke testing" without having to download the actual image to your local system. Once you're ready to do so, of course, downloading is very nearly a single-click affair. Oh, and you're given (presently) 15 GB of storage for your various builds - so you can easily do comparative testing.

What don't I like about it? In the couple of hours I've been messing around with it today, there's really only one nagging quibble: When you do the "test drive" of your new creation, the page you're running it in is a standard, non-secure http Web page. The page warns you that any data and keystrokes sent will not be encrypted, and recommends the use of ssh if that is a concern (by which most people will think https). But there's no obvious way to switch, and shutting down the running appliance (which starts by the time you read the warning) involves keystrokes and so on...

In fairness, this is still very clearly a by-invitation beta offering (but you can ask for an invite), and some rough edges are to be expected. I'm sure I'll run into another one or two as things go on. I'm equally certain that all the major problems will be smoothed out before SUSE Studio goes into general public availability.

So, besides the obvious compulsive hackers and the people building single-purpose appliance-type systems, who would really make use of this?

One obvious use case, which the SUSE Studio site describes, is as a canned demo of a software system. If you're an ISV, you can add your software or Web app to a SUSE Studio appliance, lock down the OS image to suit (encrypting file systems and so on), and hand out your discs at your next trade show (or have them downloadable from your Website). No worries about installing or uninstalling from prospective customers' systems; boot from the CD (or load it into a VM) and they're good to go.

Another thought that hit me this morning was for use as an interview filter. This can be in either of two distinct modes. First, you might be looking for people who are really familiar with how Linux works. Write up the specs of a SUSE Studio appliance (obviously more demanding than just the click-and-drool interface) and use an app of your own devising to validate the submitted entries. This validation could be automated in any of several ways.

The second possible interview filter would be as a programming/Web dev system. As a variation on the "ISV" example above, you load up an appliance with a set of tools and/or source files, ready to be completed and/or fixed by your candidates. They start up the appliance (either a live CD or VM), go through your instructions for the test, and then submit results (probably an encrypted [for authentication] archive of all the files they've touched, as determined by the base system tools) via email or FTP. On your end, you have a script that unpacks the submission into a VM and uses the appropriate automated testing tools to validate it. I can even see this as a business model for someone who offers this capability as a service to companies wishing to have a better filter for prospective candidates than resume-keyword matching - which as we all know is practically useless due to the high number of both false negatives and false positives.

What do you all think?