Browser Support: Why "Internet Explorer 6" Really Is A Typo

(Experienced Web developers know that the correct name for the program is Microsoft Internet Exploder — especially for version 6.)

Case in point: I was browsing the daringfireball.net RSS feed and came across an article on the 37signals blog talking about Apple's new MobileMe service dropping support for IE6. The blog is mostly geared towards 37signals' current and potential clients who, if not Web developers themselves are at least familiar with the major technical issues involved. Not surprisingly, virtually every one of the 65 comments left between 9 and 13 June was enthusiastic in support for the move; not because the commenters necessarily favor Apple (though, clearly, many do), but because anybody who's ever cared about Web standards knows that IE6 is an antediluvian, defiantly defective middle finger thrust violently up the nostril of the Web development community; the technological equivalent of the Chevrolet Corvair: unsafe at any speed.

The degree to which this is true, and to which this truth continues to plague the Web developer and user communities, were brought into sharp focus by three of the comments on the post. The first, from 37signals' Jason Fried, estimates that 38% of their total traffic is IE, of which 31% is IE 6.0 (giving a grand total of 11.8% of total traffic — not huge, but significant).  The second is from Josh Nichols, who points out that Microsoft published a patch to solve the problem with IE6 in January, 2007; he notes, however, that an unknowable number of users may not have applied that patch. Finally, Michael Geary points out that later versions of Internet Explorer (version 7 and possibly the now-in-beta Version 8) also have the problem of not being able to "set cookies for a 2×2 domain, i.e. a two-letter second level domain in a two-letter top level domain with no further subdomain below that," (including his own mg.to blog domain). The fact that relatively few domains fall into that category can be argued to be part of the problem; users running IE, particularly an out-of-date version of IE, are likely to be less experienced, less able to recognize and solve the problem correctly, than to blame it on "something wrong with the Internet". For those people and companies who've paid for those perfectly legitimate domains, the negligence and/or incompetence of the browser supplier and/or user mean that they're not getting their money's worth.  And ICANN, the bureaucracy "managing" the domain-name system, is now "fast-tracking" a proposal to increase the number of top-level domain names (TLDs) used. (In time-honored ICANN custom, the press release is dated 22 June 2008 and "welcome[s]" "Public Comments" "by 23 June 2008." Nothing like transparency and responsiveness in governance, eh?

Good Things are good things....aren't they?

Anybody who's worked with me over the last 20 years or so knows that I generally evangelize conforming to standards when they exist, are relevant and widely agreed on. As the famous quote from Andrew Tanenbaum (in Computer Networks, 2/e, p. 254) reminds us, "The nice thing about standards is that you have so many to choose from." When "standards" are used to promote vendor agendas (e.g., Microsoft force-feeding OOXML to a hapless ISO) or when they go against the common sense built up through hard-won experience by practitioners. And when multiple standards for a product or activity exist, and those standards are each widely used by various users (who could have chosen other alternatives), and when those standards conflict with each other in important ways that can't be amicably resolved, then those "standards" cause reasonable people to not merely question their validity, but, too often, the entire concept of "standards".

As any developer who's worked in more than one shop, or sometimes even on more than one project in a shop, knows, coding standards are sometimes arbitrary, often the prizes and products of epic bureaucratic struggle, and (in the absence of automated enforcement such as PHP CodeSniffer) often honored more in the breach than the compliance. What makes things even more "fun" is conflicting standards. It's not all that unusual for a company to contract out for development work, specifying that their coding standards be complied with (since they're the customer and they're going to maintain, or control maintenance of, the code). If the contractor has their own set of standards that conflict with the customer's, then problems arise with internal process compliance, customer involvement and final delivery. It can be — and too often is — a sorry mess. Simple code reformatting problems can be taken care of with a pretty-printer program; oftentimes, though, one sees entire programs (which have to be debugged, documented and maintained) developed just to "translate" one format to another. Many shops just give up, declare the project to be an exception or exemption from their own internal standards and processes, and try to conform to the customer's demands. "Try to", since their developers, both writing and reviewing the code, are going to be fighting against it tooth and nail because it just "feels wrong".

This whole rant was inspired by reading through yet another coding-standard document; this one the Zend Framework PHP Coding Standard. One item in particular struck me as counter-intuitive. In Item B.2.1.1, PHP File Formatting — General, it says:

For files that contain only PHP code, the closing tag ("?>") is never permitted. It is not required by PHP. Not including it prevents trailing whitespace from being accidentally injected into the output.

Experienced PHP developers are quite likely to have problems with this, not least because it conflicts with earlier behavior of the PHP interpreter and with tools that expect well-formed code. This is one of the oddities which tools like the aforementioned PHP CodeSniffer need to take into account. (There are other, more blatant "yellow flags"

If you're in a shop which takes standards seriously, uses PEAR code and uses the Zend Framework, your code review meetings are likely quite interesting.

• "OK, we're going to look at foodb.class.php first, and then the others I mentioned in the email yesterday."
• "Which standard does it use?"
• "Well, it ties in with PDO, so it ought to follow the PEAR standard, right?"
• "OK, that sounds reasonable."

As the meeting continues...

• "Hey, what's this at the end of omnibar.class.php? There's no 'close-PHP' tag! If we start using the code-search Wiki plugin that the Bronx Project folks keep raving about, it's not going to like that...."
• "Oh, yeah, but that's because  it uses all this Zend Framework stuff, so we use Zend's coding conventions... see that comment at the top about how to run CodeSniffer?"
• "Riiiiight...."

and so on. Weren't process and standards supposed to make development easier and more reliable?

I agree with the sentiment, apocryphally attributed to one or another of numerous software gurus, that, in the presence of otherwise adequate and sufficient standards, we shouldn't be so "egotistical" as to think developing a "better" standard than others already have is worth our time; take what's already out there, adapt as necessary, and move forward. The trick, of course, is in evaluating that condition, "otherwise adequate and sufficient." Also, since our craft is (hopefully) continuing to advance and adopt standard patterns for things done before, striking out on your own (after careful consideration) demands that the question be revisited from time to time. What are other development groups using (broadly) similar techniques to solve (broadly) similar problems using? Is a consensus forming, and do we have anything useful to say about it? Or has a single standard already taken hold, and we can take advantage of it (at least for new or reworked code)?

Code analyzers like lint and PHP CodeSniffer can be amazingly useful. But for them to function as standard/policy enforcement tools, there must be a standard, or a small group of similar standards for them to enforce. When development teams have to juggle between incompatible standards, it discourages them from following any standards. And in that direction lie... the 1970s.

g++ != gcc (arrrrrrgh!)

Coming back up to speed on Mac programming, now that I've finally got a shiny new iMac. Their XCode IDE looks like a great tool (Objective C, C++, C, etc., etc.), but I was hacking around building some simple test code. Being a fully-certified Unix operating system, of course that's an easy way to get something done while minimizing the number of known and unknown unknowns that need to be dealt with.

This mostly transpired between 2300 Sunday and 0130 Monday (last night/this morning). I installed CppUnit (Boost was already on the system), and kept running into the same problem.

jeffs-imac:Foo jeff$ gcc -I /opt/local/include -I /opt/local/include/cppunit foo.cpp -L /usr/local/lib  -lcppunit  -o foo
ld: in /usr/local/lib, can't map file, errno=22
collect2: ld returned 1 exit status

Hmmm. Maybe the library that's on there is screwed up somehow? Go to Sourceforge, pull down the library source, build it, install it, and try again.

Same thing. Fiddle with the code, fiddle with the command line, nothing fixes it. Go out and Google for help. The very first hit, from MacOSXHints, had a silly-sounding but tantalizing "clue":

You're right. I figured out the problem is that I was missing the -c switch when building the .o file with gcc. For some reason the linker doesn't complain about it, but when I try to link the shared lib with my main program I get the obscure can't map file error. Now it is working. Thanks.

Hmmm again. Go fiddle some more, this time compiling and linking my trivial proof-of-concept in separate gcc command lines. Still no joy.

I go back and play with building libcppunit again, wondering if I've missed some funky option to configure. Nope. It's pushing 0130, I need to be up at 6-something, and my brain is fried, so I shut down for the night.

(Later) in the morning, something's niggling at the back of my mind, saying I missed something while watching libcppunit compile, so I do it again. Yep, cue the "you dumb palooka!" moment: it's not using gcc to compile; it's using g++. For those who've never had the (dubious) pleasure - gcc is the "general-purpose" front-end to the GNU Compiler Collection, a set of (numerous) language systems, of which g++ is the C++-specific toolchain (and standalone front end). All compilers in the Collection produce object code in compatible format (the same back-end is used for almost everything), so usually all you have to do is invoke the One Command to automagically compile your Ada, FORTRAN, Java, PL/I, whatever. And, to be honest, it had been a few months since I'd dealt with C++ on gcc/g++ from the command line. (Thank you, Eclipse!) But I remembered a bit of wisdom lost in the Mists of Time...

They can both compile your C++ (in a single step). But: They're. Not. The. Same.

So, I re-do my (now-) two-step process, substituting g++ for gcc:

jeffs-imac:Foo jeff$ g++ -c -I /opt/local/include/ -I /usr/local/include/cppunit/ foo.cpp
jeffs-imac:Foo jeff$ g++ foo.o -L/opt/local/lib -lcppunit -o foo
jeffs-imac:Foo jeff$

Ta-daaaaa!  OK, can we go back to a single step? That is, compiling and linking (using g++) in one go:

jeffs-imac:Foo jeff$ g++ -I /opt/local/include -I /opt/local/include/cppunit foo.cpp -L /usr/local/lib -lcppunit  -o foo
ld: in /usr/local/lib, can't map file, errno=22
collect2: ld returned 1 exit status
jeffs-imac:Foo jeff$ 

Nope. This is where the MacOSXHints commenter was right on the money. But why? I used to be knee-deep in the (FORTRAN-specific) code, a feeling akin to being knee-deep in the dead on occasion, and the answer doesn't come immediately to mind. Any ideas?

ANFSD: starting a series to scratch an itch

(And Now For Something Different, for the 5LA-challenged amangst you...)

I've made my living, for about half my career, on the proposition that if I stayed (at least) three to six months ahead of (what would become) the popular mean in software technology, I'd be well-positioned to help out when Joe Businessman or Acme Corporation came along and hit the same technology — with the effect of "Refrigerator" Perry hitting a reinforced-concrete wall. This went reasonably well as "the market" started using PCs, then GUIs, then object-oriented programming, and then "that Internet thingy" (Shameless plug: résumé here or in PDF format).

In other ways, I've been a staunch traditionalist. I've used IDEs from time to time, because I was working as part of a team that had a standard tool set, or because I was programming for Microsoft Windows and the Collective essentially requires that that be done in their (seventh-rate) IDE unless you want to decrease productivity by several dozen orders of magnitude.

Otherwise, just give me KATE or BBEdit and a command-line compiler and I'm happy. This continued for a significant chunk of the history of PCs, until I decided that, for the Java work I was doing, I really needed some of the whiz-bang refactoring and other tie-ins supported by Eclipse and NetBeans. Then I started hacking around on a couple of open-source C++ packages and thought I'd give the Eclipse C/C++ Development Tooling a try. Now I'm coming up to speed on wxWidgets development in C++.

During this learning-curve week, I spent a lot of time browsing the Web for samples, tutorials and so on. To call most of them execrable is to give them unwarranted praise. Having recently resumed work on a Web development book dealing with useful standards and helpful process, and since I've been doing C++ off and on since the mid-80s, I thought I'd start a series of blog entries that would:

• Document some of the traps and tricks I hit to get a simple wxWidgets program into Eclipse;
• Illustrate some early, very simple refactoring of the simple program to get a bit more sanity;
• Get Subversion and Eclipse playing well together;
• Explain why I think parts of teh Agile method are simulataneously nothing new and the best new idea to hit development in a very long time.
• Start using an automated-testing tool to build confidence during debugging and refactoring; and
• Using a code-documentation tool in the spirit of JavaDoc to produce nice technical/API docs.

At the end of the series, you'll have a pretty good idea of how I feel most projects (regardless of underlying technology and specific tools) "should" be done. You'll have seen a very simple walk-through of the process, demonstrated using Linux, Eclipse, C++ and wxWidgets, but actually quite broadly applicable well beyond those bounds.

Please send comments, reactions, job offers, etc., to my email. Death threats, religious pamphlets, and other ignorance can, as always, go to /dev/null. Thanks!

Oh. Mah. Gawwwwwwwwd.

You no longer need to reboot a running Linux system to apply security patches to it.

Check it out.

Excuse me whilst I pick my jaw up from the sub-sub-sub-basement floor. If this checks out in the field, on multiple distros, then a lot of sysadmins are going to be able to get a lot more sleep at night. And the comment by one David Pottage:

This should be good for distro kernels.

Just think if you can prepare a special kernel module that will apply security patches to a running kernel, then so can your favorite distro. In future when there is a security update, instead of downloading a ~20Mb kernel package from security.debian.org or the like, and then waiting until a suitable time to install it and reboot the system, you can download a small package containing patching modules for the standard kernels from that distro, and install it immediately...

The mind reels a bit. Security patches are the most gotta-do-it-right-NOW things that come down the pipe for any system. Open-source systems that are widely audited, like Linux, tend to get patches a lot quicker than Windows (which had attacks in the wild with no fixes available for some 271 days in 2007), or even Mac OS X. Any closed system that depends on a single organization to secure it will always have slower reaction time than an open system with enough (mutually independent, distributed) resources to throw at it. As Eric S. Raymond wrote in The Cathedral and The Bazaar, "given enough eyeballs, all bugs are shallow". As long as there is some meritocratic control over the "official" patch-submission process - and there is - it's now easier than ever to keep critical systems up and secure, in ways and at speeds that simply can't be matched in the Microsoft world. Remember, even if your uptime is 99.999%, (the so-called "five nines gold standard"), you're still down five minutes and fifteen seconds every year. Murphy's Law says at least five minutes of that time will be when it was really, truly important that the system not be down.

You can't repeal Murphy's Law, but I think it does give us a big step towards an insurance policy.

Rant - can people at least engage brains before asking stupid questions? (Or: Paging Andy Rooney)

I read several different development blogs and message boards, such as those associated with phpclasses.org, codeproject.com, IBM DeveloperWorks, and so on. Usually pretty useful both for learning new techniques and keeping an eye on what other people are doing. Several of the boards, particularly on CodeProject, have been getting hot and bothered lately about the declining quality of questions being asked on the public lists; one large category of these is "affectionately" known as "homework questions'. These usually aren't for actual classwork. A more typical scenario seems to go like this: Sanjay is brought into an outsourced project because his agency assures the client that he's a hotshot — fully qualified in J2EE, PHP, XML, RPC and LSMFT (obviously a key qualification, but since the client HR person is neither technical nor over 40, it just appears to be "tech jargon.")

The client manager thinks, "I'll have to let a couple of my other guys go to stay in budget, but if this guy can save our bacon, it's worth it." Sanjay starts one bright Monday morning at 9.00,, gets the usual here's-where-things-are spiel, sits around (billing time) waiting for his computer to be set up and connected to the network (these things almost never happen before the warm body shows up), and by 1 PM is browsing the codebase for the project. By 1.30, he's on the Web, posting questions on sites that make it absolutely, crystal clear that he wouldn't know his ear from a hole in the ground if you gave him a flashlight, a map and six hours' head start.

The most extreme example of this I've personally witnessed was when I was part of a team consulting to a major Southeast Asian telecoms firm, working on integrating the homegrown billing system that one division used into the (telecom-)industry-standard one used by most of the rest of the company. This benefited greatly from knowledge of Java, of both Linux and Windows, and especially of the commercial system which was the target for the migration. The firm which produces this system, in true Java-ecosystem fashion, offers their own training which leads to certification in various aspects of the system. The team had a good mix of knowledge and experience, with the single yellow flag being that the major new-system expert was a member of the client's staff. After being encouraged to bring our own domain expert in (apparently so the client could reassign the existing one if desired), our headquarters (in India) found the "perfect guy". He had Java and J2EE certificates. He had a BSCS from "one of the top universities" in India. He had all the certifications the vendor offered. Oops, they were all for the previous version of the system....but this guy could, on paper, walk on water as far as our project's needs were concerned. So we flew him out from Bangalore. We had, as I recall, six weeks before major schedule slips would hit the fan.

A month later, my team manager and I sat watching this guy type in sample programs from the manual, try to build them, watch them fail, erase everything and start over. The manager said to me, "well, why don't you and (the junior guy on the team) start trying to pick up the pieces? Maybe you can pull something out." Then we noticed something else that was interesting. The client manager kept saying nice things about the "expert", how diligent and hardworking he was. He started taking the expert out for lunch and so on. My manager and I were in shock: this guy had already cut into the project for thousands of dollars and had yet to produce a single project artifact. Worse, in our view, his "experience" and "qualifications" were obviously complete fabrications. Headquarters (back in India), however, wouldn't send out a replacement because they had been told that the client was happy with the guy. We started looking around for brown-projectile-proof mackintoshes, anticipating the storm sure to come. And it did, eventually — shortly after the "expert" failed to follow explicit, idiot-proof instructions on how to extend his visa so he wouldn't have to go back home. Instead of buying a round-trip bus ticket to Singapore, he bought a one-way ticket, later claiming he wasn't sure which bus he'd want to take back up. The Singapore immigration officials blocked him from entering Singapore as they had no reasonable assurance of when he'd leave. The Malaysians wouldn't let him back in because his (tourist) visa had expired that day and he'd need to enter another country before reentering Malaysia. After several frantic midnight telephone calls and (I was later told) negotiation and pleas from our firm and the client, he was unceremoniously dumped on a plane back home, and our firm was billed for a full-fare coach ticket. The client manager became upset because his buddy, the "expert" was nowhere to be found and on no notice, at that.

Why am I blathering about all this now, several years after the event? My ire was raised by one of these "homework questions" I'd mentioned earlier, this time posted on the WeberDev general PHP forum. Entitled "Which is compatible PHP or Java with SQL SERVER 2005", the writer, using the nick "kumarsudu", starts out with...

Hi All, I would like to know which one is have compatible 1. PHP with SQL SERVER 2005 2. Java with SQL SERVER 2005 as i have to develop a project, ...

As another popular forum's members often ask, "how many WTFs are there in this message?" The mind reels. Asking totally nonsensical, absolutely open-ended questions on technical sites and lists is now at a flood stage not seen since the AOL infection of the Internet back in the early 1990s. The writer, besides the sterling specificity of the question as mentioned earlier, shows little knowledge of or interest in proper use of the English language (despite ample materials and information available online as well as off).

It pains me that:

• a person of such deliberate ignorance and lack of brilliance would not only choose to waste my and other readers' time with the request to do his work for him;
• any contracting agency would have such miserably low non-standards that this guy would even get the time of day, let alone a job for which there are an ample supply of other (by definition more qualified) candidates for (granted, they may have to raise their pay to a level higher than a fast-food burger-flipper);
• any client company would not only pay money for such an individual, but continue to do business with the "recruiter" that brought him in;
• there are no project-local, competent individuals whom this writer could ask for help, who would apply appropriate informational and organizational responses to the question; and finally
• that the prevailing business contempt of software development has sunk so low that this type of thing is more unusual for the fact that anyone bothered to notice it than that it happens at all. Would you want to fly with an airline whose pilots were the cheapest available, using forged certificates and qualifications to highlight (non-existent) experience? If your answer to that question is anything less emphatic than "hell no!," please inform me of your flying plans so that I may ensure that I am neither flying nor anywhere on the ground along your route during such a flight.

The "project" mentioned by the initial writer, if performed by individuals of this apparent calibre, is highly likely to fail — wasting the client's time and money, leaving the client wih a problem that still needs to be solved, and continuing to erode the opinions of the client and the client's associates of the business value of software, since, obviously "IT projects always fail."

Buzz About the Code Buzzards

Scott Hackett over at SlickEdit has a blog entry where he talks about code scavenging as "a new software development methodology." The point being, of course, that code scavenging isn't new at all; it's at least as old as the UNIVAC I. Every new, and not so new, software developer has used it at one time or another, usually for quick bootstraps to get a specific part of the software under development working in spite of limited knowledge of the language and/or domain in question by the programmer, or limited resources (time/budget). Most times, it's at least felt to be some combination of the two.

What's changed recently, and what makes five-finger coding a screaming, begging candidate for formalization, as Mr. Hackett notes, are two distinct phenomena: the rise of new repositories on Web sites like Koders, Krugle, Google Code Search, among others, that at least hold the promise of finding "what you need right now", right now. Also, development professionals face ever-increasing resource constraints ("can you get that for us yesterday?") in a craft that is continually expanding in scope and detail.

What I found interesting, having visited sites like Krugle many times in the past, was the effect that open-source licensing like the GNU General Public License and the BSD License are having on one of the main limiting factors that Mr. Hackett identifies: trust. Many developers, for various reasons, don't trust the work of someone they don't know when their own jobs or reputations are on the line. What open source and the new search engines bring to the table is the idea that now, you don't always have to just accept a code fragment "on faith". Many of the more experienced developers have a track record of other code that they've written, often accessible via the same search engines that helped you find the one you're looking at. Open source and free licensing mean that you can look at other stuff that the guy in question has written, get a feel for how his style and competence match your own, and use that information as additional basis for evaluating the code fragment you're scavenging. The trick, obviously, is to keep the time and effort required for all this significantly below what it would take for you to write your own implementation from a clean sheet of paper. That's why search engines and good indexing are important.

Of course, none of this will help you make the changes that always have to be made to put a piece of "foreign" code into your handiwork. That's still going to take some work; hopefully not on the order of putting a Porsche engine into your Ford pick-up. We'll have to wait for — and work for — serious improvements in the state of the software development craft to have any effect on that. Development methods, languages, and so on are still in the cathedral at Rheims. I've been waiting 30 years for the binary Renaissance to hit my professional life.... anybody need a virtual stonemason?

Improvement as opposed to Change

A link on the Agile CMMi blog had some very interesting things to say about a gentleman named Brian Lyons, the CEO (as well as CTO and founder) of Number Six, a Vienna, Virginia (Washington, DC area) software technology services/business consulting firm. Number Six quite obviously 'get' the concepts behind Agile CMMi, Hillel Glazer of the blog wrote. Interested, I flipped over to their site to hopefully learn a bit more, maybe drop off a CV.

Mr. Glazer wrote the blog entry in question on 25 July. The first thing you notice when viewing the Number Six site is the home-page obituary of and tribute to Brian Lyons, who died on (Monday) 3 September 2007 in a motorcycle accident. There are various links to family and tribute sites, a press release, and a request that "in lieu of flowers, the family has requested that donations be made to" a scholarship fund at the University of Maryland (excellent, particularly for those too far or too late to attend the funeral). I wish the family and company all the best in their times of grief and trouble.

The point I originally started writing this entry to make, however, was inspired by one of the bullets on Number Six' careers page, under the heading "Why Six?": "We are committed to consistent improvement, not continual drastic change."

Think about that distinction for a moment. Many of us, particularly those in the software-geek persuasion, start our careers trying to remake everything; not necessarily because it's broken, but so that we can do it (whatever it is), make it ours, introduce new techniques or technologies, and hopefully (but improbably) provide a better solution to the problem at hand than what was there before.

The forces arrayed against that impulse are formidable. Playing the role of (and often in actual fact) those who are by nature suspicious of any radical approach to a problem, too often dismissing out of hand any alleged improvement or innovation without "giving it a fair chance", in the eyes of the wet-behind-the-ears Young Turk. Occasionally, of course, improvements and innovations too beneficial to ignore do come out of this process.

As the developer matures (which may take days, decades or eternities), the Young Turk recognizes that not all of the hindrances were traditionalist per se; rather, they were operating from a different (usually business-oriented) set of priorities. Learning to understand and appreciate those priorities is one of the fundamental aspects of any successful transformation from ivory-tower Geek to journeyman software craftsman (or -woman), able to make a contribution to customers and to the craft of software development.

What becomes obvious, to all thoughtful participants and stakeholders in the process of developing and maintaining any software system is that a natural tension exists between three apparently conflicting ideas:

• Don't change what works well enough when there are more pressing needs to attend to;
• New capabilities, chosen and implemented properly, can have strongly beneficial effects (efficiency, productivity, wider customer scope, 'better' according to various aspects of quality);
• Any change to an existing system involves direct and indirect costs which must be carefully evaluated and compared against the expected improvements.

Balancing and managing the conflict between those concepts throughout the lifetime of a piece of software is an art that has yet to be thoroughly and reliably mastered with a level of reliability and cost widely acceptable to business customers. Several philosophies and techniques (often marketed as technologies) have been developed and marketed as (attempted) solutions to that problem. Three that I have found extremely useful, in complementary ways, over the last few decades are agile development, refactoring and the CMMI. (Those already familiar with the concepts may skip down to the paragraph which begins "As argued by practitioners such as Hillel...".)

The CMMI (previously more widely and less precisely known as the CMM, for Capability Maturity Model) has been around for quite some time as a formal means of evaluating the maturity and coherence of an organization's development efforts (among other activities). It requires its users (organizations) to acquire and document ever-increasing detail of precisely how they go about the process of development and how they are required to be able to prove (in an internal or external audit) that those processes are being followed and any discrepancies noted and accounted for. This suffers as a standalone policy framework for software development on two grounds:

• the "what" and "why" of development are completely ignored. CMMI fundamentally doesn't care if you're building a pile of junk that has serious practical problems, as long as you do it using the process you've defined for yourself.
• The CMMI, along with US DOD Standard 2167A, have been blamed for the logging of more trees (to make paper) than virtually any other industry business practice. This is largely because both are seen (in CMMI's case, somewhat unfairly) as pushing "hidden mandates" for the rigid , never-far-from-obsolete "waterfall" model of development.

Agile development, on the other hand, is a survival response to both the (largely management-driven) mantra that "Real Artists Ship", and on the other hand, to seemingly interminable periods between "milestones" (particularly in the waterfall model) where nothing of intrinsic value is visible to an outside observer (e.g., management or customers). Its main criticism from opponents is that it produces too little documentation (the product is the documenatation, to a large degree).

An aspect of, or adjunct to, agile development is refactoring, pioneered and popularized in the Martin Fowler-written book. Refactoring is all about how you can make (sometimes radical) changes to the internals of a software system, provided you keep the external (customer-facing, revenue-generating) interfaces constant. Wrapping one's mind around this as a formalized process, as opposed to the "don't-break-what-works" mentality common to experienced developers (and managers), is a threshold experience in a software craftsman's progress.

As argued by practitioners such as Hillel, the combination of the process-centric CMMI and the solution-centric, visible-progress Agile set of processes (including, particularly, refactoring) gives the "best of both worlds" for development — a focus on artifacts and products created through a survivable process, matched with a demonstrated and documented knowledge of exactly what is meant by "that process" under current circumstances. This also implies demonstrating understanding of the specifics of "current circumstances" that drive the decision-making and artifact-development processes — which brings us back full circle to understanding why Agile is a good fit to begin with.

Unlike the Rational Unified Process (RUP), which grew out of a waterfall-driven management-oriented mentality (and the opportunity to sell software tools and consulting services to any shop using the heavily-marketed process), Agile and CMMI development can be successfully started after passing around a few books among the development and management staff. Tools and training are, chosen thoughtfully, extremely helpful, and the appraisal process within CMMI (formally SCAMPI, the well-known "Level 1" to "Level 5" assessment of process maturity) does eventually require an outside assessor, or registrar, to formally certify the organization. But to take a typical small development group from the customary initial chaos to a finely-tuned, market-leading, customer-satisfying machine, it has been my experience and observation that it is far easier (and more cost-effective) to implement CMMI in an Agile fashion than to go down the (vendor-specific) RUP route. Imposing either on a large organization would require an unlikely combination of managerial brilliance, sadism and masochism; revolutions boiling up from below and winning eventual management sanction have proven much more likely to be successful in that type of environment.

So what's a Young Turk (or a more seasoned craftsman) to make of all this blather? The simple point: Consistent improvement is practical, and without life- or career-threatening implications, achievable on a repeatable basis. Continual drastic change, in contrast and almost by definition, will lead the development group to many sleepless nights trying to get that last show-stopper bug fixed, management to wonder why those bozos in development can't be trusted to ship anything on time, and customers to wonder what they're really getting for their money beyond the hype and bling of the marketing materials. Which team would you rather be on?

Crap Is Not a Professional Goal

I recently, very briefly, worked with a Web startup based in Beijing (the "Startup"). The CEO of this company, an apparently very intelligent, focussed individual with great talent motivating sales and marketing people, takes as his guiding principle a quote from Guy Kawasaki, "Don't worry, be crappy".

The problem with that approach for the Startup, as I see it, is two-fold. To start, Kawasaki makes clear in his commentary that he's referring strictly to products that are truly innovative breakthroughs, exemplars of a whole new way of looking at some part of the world. Very few products or companies meet that standard (and even if yours does, Kawasaki declares, you should eliminate the crappiness with all possible speed). No matter how simultaneously useful and geeky the service offered by the Startup's site is, it is, at best, a novel and useful twist resting on several existing, innovative-in-their-day technologies. (A friend who I explained the company to commented that "it sounds as innovative as if Amazon.com only sold romance novels within the city limits of Boston" — hardly a breakthrough concept). Indeed, Kawasaki makes clear that he's talking about order-of-magnitude innovation; the examples he cites are the jump from daisy-wheel to laser printing and the Apple Macintosh.

The second, more insidious, problem with the approach, and the trap that many early-1990s Silicon Valley startups fell into, is that you take crappiness as a given, without even trying to deliver the one-two punch of true innovation and a sublimely well-engineered product that immediately raises the bar for would-be "me-too" copycats. (Sony, for instance, has traditionally excelled at this, as has the Apple iPod (learning from the mistakes Kawasaki cites for the earliest Macintosh). Deliver crap, and anybody can compete with you once they understand the basics of your product. The wireless mouse is a good example of this.

If you tell yourself from the get-go that you'll be satisfied if you ship a 70%-quality product, what will happen is that, as time goes by, that magical 70% becomes 50%, then 30%, then whatever it takes to meet the date you told the investors. And if management doesn't trust engineering to give honest, realistic estimates (as is typical in software and pandemic in startups), you have a recipe for disaster: engineering takes a month to come back with an estimate that development will take 12-18 months; management hears '12' and automatically cuts that to 6 and pushes to have a "beta" out in 4. The problem is that, if you're dealing with an even marginally innovative product, things are not cut-and-dried; the engineers will have misunderstood some aspects of the situation, underestimated certain risks, and been completely blind to others. This was pithily summed up, in another field entirely, by Donald Rumsfeld:

There are known "knowns." There are things we know that we know. There are known unknowns. That is to say there are things that we now know we don't know. But there are also unknown unknowns. There are things we don't know we don't know. So when we do the best we can and we pull all this information together, and we then say well that's basically what we see as the situation, that is really only the known knowns and the known unknowns. And each year, we discover a few more of those unknown unknowns.

Companies that simultaneously forget the ramifications of this while taking too puffed-up a view of themselves are leaving themselves vulnerable to delivering nothing more useful or profitable than the old pets.com (not the current PetSmart) sock puppet — and probably nothing that memorable, either.

And that further assumes that they don't fall prey to preventable disasters like losing the only hard drive running a production server built with non-production, undocumented software. If Web presence defines a "Web 2.0" company in the eyes of its customers and investors, going dark could be very costly indeed.

Back in the Saddle, Again

....with abject apologies to Gene Autry...

I haven't posted here for a while (about three months — gak!). For the six or eight of you still hanging on, humble apologies and my deepest appreciation (sympathies?). Some have publicly wondered (offline) whether I am merely offline or have flatlined. Actually, I've been in hospital twice during that time, and my personal and professional lives have undergone more than the usual random fluctuations. Be that as it may...

As with roughly half of the Linux-aware folks out there, I've been playing with Ubuntu Linux for a while now. The job I just started — as Principal Technologist and alleged future CTO for FoneVillage.com in Beijing — is with an Ubuntu shop, so that's one motivation. I've been a Debian evangelist for a few years now — formerly a Kanotix (now Sidux refugee, now with Ubuntu and Mepis installed and happy on laptops and a desktop (and lusting after a Mac Pro (but that's another blog entry)...

Half of me LOVES Ubuntu. Point-and-click everything; all the applications (except mainstream violent Windows games) that a user could want immediately available, name-brand Big Applications for the enterprise; more-solid-than-most-rocks Debian under the hood; regularly updated Live CDs (but get the better Live DVD); what's not to like?

The other half of me, the guy who's been intimate with the care and feeding of Unix systems for almost 30 years, has an easy answer for that; sudo (as superuser/SystemGod, do) everything, but in particular, sudo bash (as superuser, open up a shell [terminal] and let me run arbitrary commands with no restrictions).If you read Ubuntu guides and Web pages, almost everything a user does from a command shell that affects the system is done as sudo command, while logged in as an ordinary user. A bit of poking around with Google led me to a page on About.com's Ubuntu Desktop Guide that put things into better perspective:

The first user account you created on your system during installation will, by default, have access to sudo. You can restrict and enable sudo access to users with the Users and Groups application.

My knee-jerk reaction having subsided, I'm back to generally liking what I see in Ubuntu. It's intended to achieve — and generally succeeds at — being "easy enough for anybody to use", not just "geeks", as Linux has heretofore been viewed by Windows usees. It's another answer to the classic "what's the difference between a Windows usee and a Mac user?" question: The Windows usee talks about everything he had to do to get his work done; the Mac user (or, generally, the Ubuntu user) talks about all the great work she got done.

For the technophobes out there who still want to join the modern world, definitely worth a spin.

If You Can't, It Doesn't

If you can't measure something, can you prove that it exists? Or at least that you understand it? If we're having a theological discussion, you might have one answer, but in the material world, and more specifically in technology development, the answer to both questions is "probably not". To quote what several nurses and doctors tell me was pounded into them during their training, "if you don't write it down, it didn't happen" — because there's no record that it happened the way you think you remember it. Memory's a funny thing — even about events that actually happened.

This less-than-breathtakingly original, but relevant, train of thought occurred to me after I spent an hour kicking the tires of the PHPUnit code coverage analysis features. For you PHP coders out there who aspire to professionalism, you will almost certainly find that rigorous, formal testing procedures will save you time and grief in the not-very-long run. If you've been "test infected" or you're working in an XP (the development method, not the massively defective software), you already know this; otherwise, you're likely to be shocked by how good your code is likely to become.

Automated unit tests (using tools such as JUnit (for Java), NUnit (for Microsoft .NET) and PHPUnit are good for proving a) that your code works the way you expect and b) that it keeps working as you make changes to it. Automating the process (so that it runs all your tests whenever you make changes) ensures that you get continuous feedback that all the tests still work (reducing the likelihood of hidden dependencies breaking). Code coverage testing, as the phrase implies, tracks which lines of your code are exercised. Code that has been tested extensively and found to work as expected can be differentiated from untested code and "dead" code. Dead code is code that can't be executed because no logical path through the program exists that will execute that code. A minimal number of lines of code will be marked as dead code in many languages and test systems due to the textual structure of the language itself (e.g., PHPUnit marks a line of code consisting solely of the brace ending an if-block as dead code). Code identified as dead should be eliminated except for this structural detritus; why maintain code that will never be executed?

Many beginning developers (and not so beginning) tend to assume that their code works if it doesn't contain any obvious syntax errors flagged by the interpreter or compiler. These language systems are (generally) reasonably competent at interpreting what you wrote; they have significant constraints in discerning what you intended beyond what you wrote. Hence, just because the code "compiles cleanly" doesn't mean it is free from defects. That's the job of testing — starting with you, the developer, running unit tests.

If you're doing unit tests without any specialised tools ("rolling your own" tests), or your testing tool does not provide code coverage analysis, it may often be difficult to determine whether specific blocks of your code have or have not been tested successfully. Thus, the assumptions that you made while writing the code are unlikely to be challenged during testing — the same assumptions will guide an ad hoc testing regime as the original coding.

One of the benefits of test-driven development is the ability to cut out unnecessary code and other development artifacts; the development team is very confident that exactly and only what is part of the desired system is actually in that system and can expand/refactor the code to adapt to changing requirements; "see a need, fill a need". You can get a lot more done, more quickly, when you have total confidence that your code will continue to work as features are added or changed, and that anything that breaks will be immediately and obviously detected. You can't do that — really — without code coverage analysis.

For you plinkers out there who aren't doing sufficient (or sufficiently organised) testing yet — your competition is, and if you spend any time at all in this craft, you will bang up against "difficult" bugs that wouldn't have been so difficult with pervasive testing. Those of you who have been writing unit-test cases shouldn't automatically get too comfortable, however... how do you know how much of your code is being tested? If you're testing the same block of code eight different ways and other sections of your code don't get tested at all, can you ship a quality product? Coverage analysis will save you time (by not writing redundant test cases), grief (by prodding you to test areas of code you thought you tested but hadn't), and money (from #1 and #2). If you're trying to run a completely instrumented shop — where everything that can be measured for a lower cost than the failure of that thing, is being measured — then I (should be) preaching to the choir.

To sum up:

• If you don't write it down (in a way that "it" can be found again), it never happened. If you've never tested your code, it's broken until proven otherwise.
• If you can't repeat the test at will, it hasn't been tested.
• If you don't or can't know how much of your code has been tested, your users will.

We can't all be Microsoft and expect our paying customers to find our problems!

XHTML Is (Nearly) Useless

EDITED 2010/11/02: If you saw this post as a single massive block "paragraph," my apologies. The definition of "what is a paragraph?" changed after this was originally posted, and it had scrolled far enough back that I didn't notice the carnage. Let me know if there are still any problems. Thanks.

---

If you've written any Web pages in the last five years (at least), you've at some point bumped into the difference (schism?) between "original" HTML and "new, improved - now based on XML!" XHTML. If you don't write Web "content" (thanks for reading my blog, but why are you here?), or deal professionally with those who do, you may not know the difference, or care that there is a difference. There is, and people should care about it if they care about the Web.

(Briefly, for those who care but don't know; the rest of you can skip this and the next paragraph.) HTML is often known to developers as "tag soup", because very, very many sites don't follow the strict interpretation of the standard, and are "broken" in all sorts of ways. This was initially justified as working around the myriad bugs in grossly defective browsers such as Microsoft Internet Explorer. XHTML was different and better because it was HTML reformulated as XML, which could then be "validated" (checked) by any validating XML parser. HTML-as-XML also (should have) driven the development and use of all sorts of nifty techniques and tools that are only practical when assumptions can safely be made about the structure and format of the document - which would be true in XML/XHTML but not necessarily in "classic" HTML.

The problem, of course, is Microsoft's Internet Explorer browser, affectionately known to Web professionals as "Internet Exploder". Among the many "quirks" (defects) that has unknowingly afflicted usees of that browser, all versions up to and including the current Version 7 fail to understand XHTML as XHTML. The "conversation" that takes place between a browser and a server when the browser requests a Web page is defined by the open standard known as HyperText Transfer Protocol, or HTTP. Part of that conversation involves the server informing the browser what type of data it will be sending. This is done using what is called a "content type" "header".

All together now? Good. When a server wants to send a browser a page of "tag soup" HTML, the correct content type is "text/html". A properly-formatted and -served XHTML page will instead use "application/xhtml+xml". This will inform the browser that, in fact, the page being transferred is a proper XHTML page (per the open standard defining it), so the browser will kick in the assumptions and processing that works for XHTML but not for "tag soup".

Of course, Internet Explorer is now the only major browser that gets this wrong (as indicated by this vintage-2005 blog entry). As far as I am aware, every other major graphical browser in the world - Firefox, Opera, Konqueror, Galeon and the rest - all support The Right Thing. Unfortunately, IE is still the 300-pound gorilla in the china shop; the majority of Windows usees still browse the Web using IE, and though the trend is improving steadily, that will likely continue to be true for the next couple of years (say, 2009-2010 barring unforeseen circumstances).

What kinds of things would a properly XHTML 1.0-compliant browser let us do with our site? One trivial example: let's say you're writing a political-commentary site that is geared towards an upcoming election, and you want to consistently name your candidate as "The Honorable Senator Francis X. Snort (email senator@senatorsnort.org)". When your guy goes down to defeat (one too many campaign-finance scandals, mayhap) you want to change the blurb to "The Honorable former Senator F. X. Snort (email snort@somefreemail.com)". Trivial to do with whatever CMS or scripting system you're using, right? But by using an XML entity, you can simply say "&snort;" in your document, and an entity declaration in your document's header will tell the parser what you really mean. Change the declaration, and every instance of that entity expands to your new meaning. People who use other XML-based markup systems, such as DocBook, have been using this technique for years. Using XML entities in pages shown in correct (non-IE) browsers will do exactly what you tell it to. In IE, or, to be fair, several text-based browsers, the entity name will be displayed exactly as it is in the document - in our case, as &snort;. This is unlikely to have the desired effects on the folks "back home" for the Senator.

Web developers have, as I mentioned, several well-known workarounds for this type of thing, using their authoring tools rather than the document itself. It is, however, a reasonably easy example for people to understand. Given the increasing popularity of systems such as PHP Smarty that let you use large chunks of "raw" (X)HTML along with the scripting goodies, it would come in handy too.

So how does all of this make XHTML "nearly useless?" Because most developers developing pages for the general public (as opposed to corporate intranets), knowing that Microsoft IE doesn't support the correct content type, will either "not bother" developing "correct" XHTML or at best will serve it to all comers as "tag soup" HTML.

This also has the "benefit" of completely stifling further innovation (as far as the end user is concerned) based on XHTML. All of the comments I've made so far are only germane to the initial version of XHTML, designated 1.0. The newer versions, XHTML 1.1 and XHTML 2.0, provide new features and support new technologies that greatly expand the usefulness of the Web - or would, if Microsoft weren't, as usual, dragging the Web down for competitive lock-in purposes. By doing everything in their considerable power to ensure that IE browsers and sites aren't fully, completely interoperable with other browsers, they discourage Windows usees from using "rival" browsers to browse sites labeled "Best viewed with Microsoft Internet Explorer". There's nothing preventing Web designers from writing standards-compliant sites that also work well with IE; in a well-designed site, it's not particularly onerous to support both standards and Microsoft. If you're using Microsoft tools, of course, it will take quite a bit more work and knowledge to create valid sites. It can be done - several sites and mailing lists describe the techniques and mind-set required - but Microsoft do not go out of their way to make it easy to do so.

Of course, this also applies only to the public Internet. If you're fortunate enough to be developing "real Web apps" for your company's intranet, and your company understands the value of open standards, then you're not going to be subjugating yourself to IE and none of this really applies to you. Go enjoy all the things that new tech lets you implement that can really stomp on your non-standards-using competition!

For the rest of us, until the Web gets out of this proprietary funk it's in now, and IE either falls into a long-deserved oblivion (improving Windows security dramatically, but that's another post) or actually complying with the same standards every other serious browser in the world does, then we're going to have problems. One of the more annoying and frustrating ones, as we've discussed, is that XHTML is (nearly) useless." So much for innovation.

It's the End of the Net as we know it, and we feel fine....

John C. Dvorak has an interesting post on his pcmag.com column blog, entitled "Will the Internet Collapse?" He doesn't think it will, obviously, and he's got some pretty impressive trends to back up his contention. Example: 140,000 terabytes of backbone traffic in 2002 — at a "conservative" 60% annual growth through 2007, that's roughly 25 KB for each of the six billion or so people on the planet. Most of whom (still) wouldn't know what a byte was if it bit them; they've got more pressing concerns, like safe food, clean water, housing... But I digress.

I don't think the Net per se will "collapse", either. What's going to happen — what's already happening — is both more subtle and dangerous. The "Internet craze" that gave rise to Bubbles 1.0 (1990s) and 2.0 (now) and has driven the Net from a quirky research project into a cultural touchstone, has done two things that, by comparison, would make an every-Friday-from-4-to-10-PM crash seem benign in comparison (and "4-t0-10-PM" where? On the Internet, it's always "now".)

The first problem is the Baby's Spoon in the Waterfall. There's so much information (wrapped up in even more "content", which isn't the same thing) that no person, government, entity or corporation can ever comprehend. People who spend large amounts of time surfing the Web and using various tools to pull information off the Net in other ways, soon exhibit a behavior akin to being "punch drunk". Late in te 12th round, The Champ has connected so many times with Joe Palooka's jaw, and we in the crowd can see Joe staggering around, unsure of even from which direction the merciless pummeling is coming, let alone able to control the situation. The Champ, in this analogy, is the onslaught of data/information/"content" from the Net, primarily email and the Web; Joe is standing in for the typical, non-technical ("you mean Yahoo and the Web aren't synonyms?") user. As the user's eyes glaze over and the cognitive mind enters vapor-lock, he is essentially unable (and psychologically unwilling) to refine his usage patterns or seek out new experiences that he wouldn't find in "offline" life (what in an earlier age was called the "You Are There" effect). So, for instance, the stereotypical North American user goes back to the "safe, familiar" online equivalents of his offline television shows — the "news" sites owned by the same multinational corporations that own American media, and YouTube, which can be viewed as a worldwide online version of "America's 'Funniest' Home Videos": another vehicle for peddling the same tired corporate products in the commercials.

The other problem, of course, is that organizing all this "stuff" has become more difficult, and the rate that it becomes more difficult is at least as rapid as the rate of growth itself. While the Net, and the Web in particular, have enabled new ways to express individual personalities (e.g., MySpace) and alowed ordinary citizens of many countries to amass much ore detailed information about what their government is doing, for them or to them (e.g., YouGov and Thomas), if you don't know about YouGov or Thomas (or any similar site set up by your own country's government), then the old Bruce Springsteen song, "57 Channels and Nothing's On" seems quaint and manageable in comparison. People know there's all sorts of stuff out there — they can Google for it, "it must be real" — but, unable to come to grips with how things are organized (they aren't, on purpose) or how to use the available information to achieve a personally important goal, they fall back on the sites that organize and package and sanitize the content, accepting loss of control as the price of freedom from thinking too much. (E.g., AOL — a subsidiary of Time-Warner, and Fox "News".com, a wholly-owned subsidiary of AIPAC.) As people sink safely back into their easy chairs, content to absorb the anti-intellectual pablum that bombards them, they lose touch with the idea, let alone the possible reality, of an energized populace using the new, revolutionary technology at its disposal to improve their own lot in life and that of the world at large. Instead of a medium which challenges the status quo, the Net has devolved into a tool which reinforces it.

A collapse of the Internet? You're right, John; it will never happen. But a collapse of the promise and meaning of the Internet? It's already here, folks; we're just standing around watching streaming video of the rubble bouncing.

Projects and Data Formats, or Scratching a Standard Itch

Fair warning: This post was written in bits and pieces over a week that I spent mostly on my back in bed; it hits two or three hot-button issues that I've been running up against. In the fullness of time, I may come back and break it up, or write follow-on entries pontificating on one point or another, but for the nonce, your patience — and comments! are appreciated.

Real standards happen in one of two ways. One way involves an organisation like the World Wide Web Consortium (or W3C as it is commonly known) puts together different committees and working groups, and over the course of various meetings, seminars, forums, and other corporate expense-account sinkholes, massive sets of documents are ratified; if we're lucky, somewhere within that will be nuggets of information and wisdom around which useful things can be accomplished. Successful xamples of this include standards such as HTML, XHTML and CSS 2. Less sucessful examples include efforts such as WCAG 2. While it may safely be assumed that nothing in the new standard will disrupt the existing order of the Internet, the flip side of this is that there may be no actual working implementation of the new standard (to prove that such is practical), and it may well be that the new standard is not the most efficient or elegant solution to the problem. This may be described as the "top-down" approach.

The other way that standards happen in the real world is for a developer, or typically a small group of developers, to come up with something that works for them, open up community/public comment and collaboration, and eventually submit the standard definition (whihc by then has several working implementations) to standards bodies like the W3C or the Internet Engineering Task Force. This may be seen as the "bottom-up" approach. Its success is largely tied to how effectively it solves what it sets out to, and equally critically, whether it does so in a manner that doesn't convey inherent advantage to a subset of its audience (such as the company employing the creators of the standard). Successful examples of this include vCard and its successor hCard.

Stumbling across the description of hCard (from An Angry Fix by Jeffrey Zeldman, a well-known figure in the online Web-design industry and community) after I had been giving some thought to a problem I had been having with contact information in various formats. Namely, that the information was in various formats, for my (ancient) PalmPilot, each of two different Nokia phones, my email package (Mozilla Thunderbird), and so on, and so on.... Keeping everything synchronised — the mundane necessity of ensuring that any given contact was in each of the needed places with the most recently updated information — is a burden sufficient to preclude any further effort, such as actually communicating anything useful or interesting to those contacts. (Maybe they read this blog...)

What's needed is a free, open source bit of software to take these various directories in varyingly historical formats, apply updates and changes to a single, current-technology directory around something like vCard (or, better, hCard), and then to spit out various dumps of this data to suit the different devices and their differing format requirements. If you think about this for a while, you can think of all sorts of ways that synchronisation could be a real pain...which update gets applied if you enter the same information two different ways on two devices? Suppose that I get energetic and add data to the "Custom Fields" in my Palm to represent data that has specific fields for the phone or Thunderbird — but since I add different data at different times, it's not always consistent? And on and on...

I'm going to keep one eye open over the next few weeks or months for something that does this relatively painlessly (and, of course, if anybody knows of any, please let me know). Otherwise, it's likely to become Item 374 in my medium-priority queue for Tools I Intend To Write (Someday).

Implicit in the first paragraph, and alluded to more directly in the third (see Zeldman et al) is the fact that the W3C has spent the last 2-3 years making abundantly clear who its customers and stakeholders are, and telling those of us who are professionally tied to standard technologies but who are not ourselves multinational corporations flush with cash for endless junkets (and patent payoffs), to take a long walk off the shortest pier available. While this may be seen by some as an efficient use of resources, addressing the corporate sponsors who are the titans of the marketplace anyway, somebody made a good point along the way: the Microsofts and H-Ps and IBMs and so on of the world started out as small shops that nobody had ever heard of. Had the standards of the day been defined less for what made sense from an engineering perspective than a lock-out-the-small-guys marketing directive, the world would be a very different — and likely less advanced — place today. What goes around, comes around — and the W3C in particular is building up a lot of bad blood with the vitally "interested parties" who don't happen to (presently) be among the 200 or so largest corporations on the planet.

What will happen? On the one hand, we'll likely wind up with lots of easily available but proprietary "standards" like Adobe PDF; the word processors I've used for the last four years have supported publishing in PDF without Adobe asking for a dime. On the other hand, we'll have highly marketed, widely Diggable, proprietary-means-you-only-get-it-from-us packages. These may have lively add-on Astroturfed communities, but they won't deliver the business benefits of truly open software; you can't fork the product, you can't completely support yourself, every use you make of the product or technology, in perpetuity, will be subject to the dictates and whims of the company that owns the product. Well and good, you say; they do, in fact, own their product, and have a right to do whatever they like with it. True, but where does that leave customers who incorporate that product into critical business processes? A year or so ago, an American friend of mine told me of one of his clients, who had a hard drive in their accounting server fail. They swapped out the drive, restored from backups, and found they needed to reinstall the order-management package they used — to generate and track every single order from the day the company was founded right up to the guy who just got off the phone — needed a license key. Fine; they call the vendor's toll-free phone number, expecting to be back in business (literally) in a few minutes. Oops. The vendor was bought out by a much larger firm; their version is now three versions out of date and the (new) vendor requires htem to buy an upgrade — at retail — to access the data they've just restored from tape.

When people ask me what the business benefits of open systems are, they don't want to hear a Stallmanesque sermon on the virtues of individual liberties, real though they may be, or the geek chic of cool code, or the cheapskate appeal to "it doesn't cost a thing". It does — in time and effort to convert and adopt within the enterprise. But what you get from it at the end of the day is control over your own business processes; you can keep running a ten-year-old word processor if you choose to, or have your accounting package customised just so, or whatever you can create a business justification for — and it's going to be much easier to cost-justify relatively audacious projects because there are no hidden surprises. Transparency, auditability, control, economy -- those may not be terribly high on the Digg word list; they may not have dozens of ...For Dummies-style books in your local chain bookshop, but people who make their living, and their employees' living, by making the numbers come out right every quarter should understand what I'm talking about. It's about time.

A side note: Anyone who is considering setting up business in Malaysia rather than other nearby countries (Thailand, Vietnam) may well want to consider the level of technical efficiency, customer support, and attitude towards serice of the local telecom quasi-monopoly. For most people and businesses, Telekom Malaysia is the only game in town. As one of the subscribers/victims of their Streamyx ADSL "service" for the last three years, I have watched connection speed and reliabillity plummet as thousands of new subscribers are pushed onto steadily lower and lower tiers of service. I believe, for example, that they now offer a "broadband" connection at 128 Kbps; twice as fast as a standard dialup modem. I am paying for a 2 Mbps — 2,000 Kbps — connection; in the last two months, I have never witnessed transfer rates higher than 400 Kbps, and for the last week never higher than 80 Kbps. If I were living in a capitalist system with competitive markets, I would have choices. In a functionally Stalinist economy where competition against government-linked is tightly controlled, I have no usable choices. It has taken me well over two weeks of trying to post this blog entry. Selemat datang ke Malaysia! (Welcome to Malaysia!)

Promises Kept, Credibility Gaps, and Microsoft: Are we Customers or Consumers?

As reported on Slashdot, quoting Quentin Clark's WinFS team blog (which spun the item mercilessly), and commented on widely, particularly by rjdohnert and Kamal:

WinFS is dead. What has been understood for a decade or so to refer to a "Windows File System", recently rechristened in Microsoftspeak as "Windows Future Storage" (to imply a lack of commitment to a product or in fact anything specific at all); in any form recognisable as the product/technology that has been hyped unrelentingly by Microsoft when they needed something to keep users (and developers) committed to the Next Windows Version, the plug has been pulled for what promises to be the very last time. This could be viewed in a number of ways; the least uncharitable explanation that concievably touches upon our shared reality is the subject of the remainder of this item.

Yet another case of Microsoft overpromising and underdelivering? Since they really don't care about providing great software to consumers — either end users or developers, there is no real penalty for failing to keep promises (though they do, in true Rove/O'Reilly fashion, try to spin the sucker positive as hard as they can, just to keep the yokels giving the slack-jawed "wow....they say it's cool" and, as Michalski originally wrote, crapping cash).

There is absolutely no reason to keep waiting for a relational file store in Windows or any product except SQL Server (and possibly some future version of OFfice that requires SQL Server). There is no reason whatever to believe Microsoft will keep ANY promise made to developers or end users, nor or in future. There is absolutely no reason to believe that any gee-whiz "technology preview" given by Microsoft will ever turn into a real, stable, usable product unless that product is announced (with a ship date) at the show or conference where the demo is made. Stability and usability of said product will, as with all previous Microsoft releases, have to wait for the second service pack.

What this boils down to, in other words, is a matter of trust, and commitment, and honesty, and all the values that a company which values its customers (and workers) is expected to incorporate into its ethos. That Microsoft deliberately chooses not to do this, as it has proven on numerous occasions, shows its complete and consistent contempt for those poor schmucks it sees as consumers, not customers.

We, as developers and users, have two choices. We can either continue to prove Microsoft right, gulping whatever product they deign to deliver, crapping out whatever cash they choose to take, abjectly powerless to exert any change over their behaviour. Or, we can refuse to play their game any more. There are other tools to develop products for Windows. Most of these have the additional benefit of being cross-platform.

"Cross-platform". There's a quaintly radical word in these times. The idea that people could use a variety of systems, tools, applications, to get their work done. Companies don't have to pay US$600 to buy an office "suite" with a heavy-duty word processor, spreadsheet, and yadda yadda for a manager whose work is primarily limited to short memos? Revolutionary. Selecting tools based on the needs of the user rather than the "default" "choice" for the entire organisation? If one choice of office layout doesn't fit everybody from the managing director to the secretarial pool, then by what logic should they use the same software tools to do their work? How ma many users of, say, Microsoft Word use more than a tiny percentage (say, 5%) of the "features" in the product? (According to surveys dating back to 2000, roughly 5%). By looking at the situation as a need to give each user tools appropriate for the task at hand, rather than imposing a uniform "solution" and adapting the task to the "solution"?

This whole WinFS affair is yet another bit of weight pushing the Good Ship Microsoft towards (or past, in some opinions) the tipping point. Those already on board might do well to examine their options; those considering extending their 'booking' may wish to reconsider. The main forces arguing that no 'realistic' options exist have been marketing-driven, rather than technically- or business-driven. Consumers blindly take whatever they're given; customers demand products that meet their needs. It is high time that those who purchase and use business computer software systems, and the tools to work with them, availa themselves of their options.

Who Needs Privacy without Liberty?

I was originally going to call this post "Pretty Good Astroturf — What Happened to PGP at the Grass Roots?"

The Register has a good piece on Whatever Happened to PGP? As a PGP (now GnuPG user for at least ten years, I was immediately interested.

PGP, for those of you who might not remember, stands for "Pretty Good Privacy". It was arguably the first widely-deployed, open, cross-platform public key cryptography (encryption and electronic signatures) software systems. At one time, the growth in usage looked like China's economic output — respectful transitioning to breathtaking, with people confidently forecasting 'incredible' within the near future. Then a funny thing happened.

People — ordinary individuals, what politicos call the "grass roots" — stopped being so interested in PGP, and PKI in general. It turned out that people were willing to be sold on the idea that the only thing they needed encryption for was to work with a "secure Web page" in their browser, so they could order stuff using a credit card. The idea that people might want to keep their personal communication private, or be able to make messages and files that they create tamper-proof, just went completely below the radar. This "just happened" to "coincide" with the increasingly shrill jingoistic/"security" propaganda being drummed into the skill of ordinary Americans; security and identity management were no longer something that many ordinary people could use and control without feeling it all either a bit ridiculouse or seditious, depending on one's politics. Still, public discussion and enthusiasm — at least among "mainstream" Americans — seemed to diminish from about 2001 onwards. The travails that PGP went through didn't help grassroots individual use — first with the US government trying to crush Phil Zimmerman, the original developer, and then the soap-operatic sagas by which Network Associates, Inc. acquired and then almost literally threw away the original PGP code base.

But, as the Register article points out, there was one very significant group of users who jumped on PGP. Since PGP depends on a "web of trust" — A trusts C because A knows and trusts B and B asserts his trust for C — the use of PGP within widespread organisations, where some central IT or other department can certify (and possibly issue) PGP keys, is seen as a natural solution to business problems of identity management. Where in the early days, a PGP user might send and encrypted message from his office email account, comfortable in the belief that his corporate masters would be none the wiser, now the corporation is including PGP in its infrastructure.

Grass roots, meet AstroTurf.

Some might see the tone of the Register article as "how can we solve this problem?" But which problem?

Popular use of PGP, or other public-key crypto, would be desirable in a libertarian culture where people valued and guarded their privacy and identity, particularly against encroachment and/or usurpation by a less-than-trusted corporation or the overweening State. While the justification for this exists in the current American social and political system, more than ever before in living memory.... the social impetus doesn't really exist anymore. An educated, informed, watchful and skeptical American population has largely forgotten how to think for itself, delegating that once-vibrant activity to the likes of Faux "News" and the Lobby.

Corporate use, on the other hand, is proceeding apace; and those users would argue that there is no real problem: a business need has been identified, a tool selected that addresses the problem, yielding a solved problem. What's not to like? Errr....yes, well, it does depend on your viewpoint. Was that the original intention that Zimmerman had in writing PGP? Almost certainly not. Does that make the use of PGP in a business environment any less "right" or "proper"? Not if it is to remain "free" as in speech; anybody can usu PGP, as any free software, for any purpose permitted by the license.

What's "wrong" isn't the way that the use of PGP is growing, even though that isn't in a way that necessarily enhances human freedom or liberty, or enhances the security and privacy of individual citizens, as originally intended. Rather, it is that the political and social culture has changed, to where the values of freedom and liberty are no longer widely seen as individually attainable or discernable; rather, people believe themselves to be as free as they are told that they are — and see no need for independent evaluation or confirmation. Technology can be used to aid the solution of social and political problems; it cannot, however, be a "solution" in itself. Just as the old saying goes, "you can lead a horse to water, but you can't make him drink", you can provide the people of the world, whatever their present situation, with tools to enhance that freedom and liberty — but people will only use the tool if they care about such things. If Huxley's observation is accurate, that "The victim of mind-manipulation does not know that he is a victim. To him the walls of his prison are invisible, and he believes himself to be free. That he is not free is apparent only to other people. His servitude is strictly objective" -- then the tools available don't matter. A key is useless to one who does not see she shackles on his own wrists. That, I fear, is the level that far too many Americans — and others — have fallen to.

What happened to PGP? It got better, and became as obsolete as freedom.

On the importance of keeping current

Now that PHP 6 is in the works, there is even less excuse than existed previously for Web sites (hosting providers in particular) not migrating to PHP 5 from PHP 4. We are faced with the unpleasant possibility for tool and library developers of having to support three major, necessarily incompatible, versions of PHP.

I am not yet up to speed on what PHP 6 is going to bring to the table, but PHP 5 (which will be two years old on 13 July 2006) makes PHP a much more pleasant, usable language for projects large and small. With a true object model, access control, exception handling, improved database support, improved XML support, proper security design concepts, and so on, it's a far cry from the revised-nearly-to-the-point-of-absurdity PHP 4.

Another great thing about PHP 5, if not strictly part of it, is the PHPUnit unit testing framework (see also the distribution blog). This is a wonderful tool for unit testing, refactoring, and continuous automated verification of your codebase. It will strongly encourage you to make your development process more agile, using a test first/test everything/test always mindset that, once you have crossed the chasm, will benefit a small one- or two-man shop at least as much as the large, battalion-strength corporate development teams that have to date been its most enthusiastic audience.

I have so far used this tool and technique for three customer projects: the first was delivered (admittedly barely) on time, the second was actually deliverable less than 2/3 of the scheduled calendar time into the project (allowing for further refactoring to improve performance) and delivered on time, and the third was delivered 10% ahead of time, with no heroic kill-the-last-bug all-night sessions required.

Discussing the technique with other developers regarding its use in PHP and other languages (such as Python, Ruby, C++ and of course Java; the seminal "JUnit" testing framework was written for Java), gives the impression that this experience is by no means unique or extreme (nor did I expect it to be). Given that two of my three major career interests for the last couple of decades have been rapid development of high-quality code and the advancement of practices and techniques to help our software-development craft evolve towards a true engineering discipline, this would seem a natural thing for me to get excited and evangelical about. (The third, in case you're wondering, is the pervasive use of open standards and non-proprietary technologies to help focus efforts on true innovation).

All of this may seem a truly geeky thing to rave about, and to a certain degree, I plead guilty of that. But it should also be important, or at least noteworthy, to anybody whose business or casual interests involve the use of software or software-controlled artifacts like elevators and TiVo. By understanding a little bit about how process and quality interact, clients, customers and the general-user public can help prod the industry towards continuous improvement.

Because, after all, "blue screens" don't "just happen".

Once more into the breach, dear friends; once more....

To the half-dozen or so of you reading this blog, thank you; and for those of you who wondered what's happened to me and this blog over the last several months, the answer is both "a great deal" and "not much at all".

I had been ill for a couple of months, with what the doctors insisted was just an ordinary flu, and then a cold, and then an ordinary (NOT H5N1, thank you very much) flu that has kept my close friends busy trying to spy the license tag number of the lorry that keeps running me down. I am better now, thank you.

I have also changed hosting providers for my professional Web site and email hosting; the new crew look to be a good outfit so far:

• they understand the value of responding quickly to customer enquiries, no matter how harebrained;
they understand Linux and Apache and (at least do a convincing appearance of) not just a "me-too" offering;
• their people know their way around their system (see the first comment);
• they have sensibly large limits on disk space and bandwidth; which means that
• they allow you to host a lot of tools and libraries and addons that you can manage yourself (think PEAR for you PHP types) without having to rely on the (necessarily limited) knowledge of a central administrator who may not be quite as up to scratch on version X of the FooBar publishing framework as you are.

In short, as I said, off to a good start. After getting some minor details worked out, and being on my feet again, the all-new seven-sigma.com Web site should be up within the next couple of days.

If you can't extend it, is it really an eXtensible HTML?

Arrrrrrrrrrrrrggggggghhhhhhhhhhhhhhh!!!!! So much for consistency....

As anybody who has worked with me in the last 3-4 years well knows, I have been an enthusiastic advocate of DocBook as a documentation markup vocabulary for various purposes, and by extension, XML-based tools for all manner of things (Apache Ant and so on).

One feature I use regularly in Docbook XML source documents is the internal subset, which lets you define entities and include files defining entities not part of the original DTD. So, for instance, my standard software.ent file has an entry of

<ENTITY mswindows '<ulink url="http://www.apple.com/switch">Microsoft Windows</ulink>'>;

This way, anywhere in a DocBook file that includes that entity definition, I can type &mswindows; and, when the document is transformed (into XHTML, PDF, RTF or whatever), the desired link and text will appear in place of the entity. This is an obvious lifesaver when you want to include, for instance, links to glossary definitions for unfamiliar terms scattered through a document.

Fine. But the current state of XHTML (the XML-based successor to HTML) simply doesn't support it. It does not appear to be possible to have an XHTML document with an internal subset parsed correctly by any current major browser on Windows, Linux or the Macintosh. Various Google searches such as this one produce links to pages that say, with varying levels of emphasis and literal wording, "you can't use internal subsets for XHTML that is to be rendered by a browser". It seems that in the force-fit of XML to HTML that produced XHTML, the concept of different "streams", or purposes, for documents was introduced. XHTML which is to be rendered in a Web browser has one set of limitations (including the internal subset); whereas XHTML conformant to the same definition documents which is to be processed as "pure XML" has another.

To say that this sucks is to use that colloquialism as an extreme understatement, akin to saying that tsunamis are wet. This limitation closes off an entire range of applications that would use dynamically-generated XHTML as browser-viewable data in the same spirit as XML generally (without writing an otherwise redundant app to parse and reformat the data). The benefits — and they are significant — of an XML-based browser markup language are (in my view) seriously degraded by foolishness like this.

Of course, several of you are already thinking, I could just use XSLT instead. I had previously wondered why PHP and other Web scripting languages included support for XSLT processing. Now I know, I guess.

If anybody has any corrections or other good ideas, please let me know.