Best Practices Alleged; Your Mileage May Vary

Yahoo! quite often releases interesting/useful/thought-provoking tools for people doing "serious" Web development. I add the modifier to specify that we're usually not talking about the Joe Leet three-page magnum oopus; a lot of what they do and talk about really only pays huge returns when you work with a site as large and complex as, well, Yahoo!.

Recently, they brought out a couple of nifty tools that integrate into the Firefox browser's Firebug Web-developer-Swiss-Army-knife extension. One of these, YSlow ("why [my site] slow?") does some interesting evaluations and calculations against whatever page (with secondary requests) you throw it at. Its "Performance" tab shows how your page matches up against Yahoo!'s new "Best Practices for Speeding Up Your Web Site." At first blush, a lot of these make perfect sense; "Avoid Redirects", "No 404s", and so on. YSlow, on the other hand, evaluates against a slightly different set of guidelines to those on the Best Practices Page:

1. Make fewer HTTP requests
2. Use a CDN
3. Add an Expires header
4. Gzip components
5. Put CSS at the top
6. Put JS at the bottom
7. Avoid CSS expressions
8. Make JS and CSS external
9. Reduce DNS lookups
10. Minify JS
11. Avoid redirects
12. Remove duplicate scripts
13. Configure ETags

"Huh?", our hypothetical Web pseudogod Mr Leet might well ask. "What the heck is an 'ETag'? Or a 'CDN'? Does any of this even apply to me?" Well, Joe, yes and no. For instance, content-delivery networks like Akamai or ATDN, as you might well know by hearing the names, scatter servers at strategic places around the planet with the aim of reducing the time it takes to get data from huge, media-content-heavy sites like CNN.com or the like, down to your browser at the end of a surprisingly long chain. Does everybody who puts a site up need something like this? Does the average small-to-midsize business? Usually not, unless you really are a Web Hype-Dot-Oh site that shoves exabytes out every day to wow the yokels or the investors. For the local pizza joint with a site containing maybe forty files, tops, with a couple of megabytes of images, a CDN is thermonuclear overkill. As many Web-development sites have pointed out for the last decade, there's quite a bit you can do to speed things up and lower bandwidth usage without spending the big bucks on this.

Why do I blather on about this when I started talking about best practices and YSlow? Because for practices to be "best", they first and foremost have to be appropriate for the use at hand. Buying a Lamborghini Countach to go down to the corner store for some sodas will quite likely get you yelled at by the Significant Other (followed by your bank). But if Lewis Hamilton showed up at pole position in a '72 Ford Pinto... you'd hear the laughter from St Paul to São Paolo.

Use the tools and techniques appropriate to the task at hand. There's a lot that small Website developers can learn from Google and the tools they publish. Getting an "A" score has a certain karmic appeal, and most of the optimizations required are straightforward anyway (tweaking how your Web server serves your data, for the most part). But is this worth all the geek love it's been getting?

Until someone with the developer credibility and experience of a Yahoo! stands up and explains a better set of practices for the SMB developer, the answer seems to be "yeah, probably". We who make our living (or our diversion) from the creation, care and feeding of Web sites are, for the most part, artisans posing as engineers, with inconsistent knowledge or practice of our craft; we dream of building the online equivalent of the Empire State Building but wind up with the Cologne Cathedral; a wonder, yes, but surely 600 years was well beyond the original estimated schedule! Agreed-upon standards (so that, say, a page appears identical in different browsers),; a shared, common body of knowledge; even (gasp!) widespread, vendor-neutral certifications of professional competence will eventually become common in software (including Web) development for the same reasons as, say, in architecture. The artifacts involved (skyscrapers, Web sites) have important social and policy implications, and inconsistent competence in practice poses a real and serious danger to the public at large. Sooner or later, it's going to be uneconomic for the present ad hoc system to advance the state of the art, or to meet the needs placed upon its products.

Best practices are good; best practices that actually work for the stated purposes in a broad variety of praxis are much better. But to get there, we're going to need to collaborate and communicate effectively, and to do that, we're going to have to make sure everybody involved is speaking the same language to describe the same things. If we don't, we'll continue to be stuck in pretty much the same place we are now — with a bunch of shade-tree mechanics running around in the pits at the Monaco Grand Prix...only doing a lot more damage.

Comments are welcomed, as always.

Does anybody else have a problem with this?

If you've got an ssh connection to a Debian or Ubuntu Linux box handy, and you have sudo privileges on that box, try this little experiment:

1. ssh to your box as an ordinary user;
2. sudo su to get a root prompt (you should be asked for your password - this is important);
3. as soon as you get the root prompt, exit back to normal user, then exit your ssh session entirely.

Now, here's the scary part:

1. ssh to that same box again right away, as the same user;
2. sudo su to get a root prompt again.

Why is this scary? Because the second time you ask for a root prompt, you're not prompted for a password. This means that, not only does the actual Linux box require access and user security appropriate to its function, but so does every device that can ssh into it with a rootable user! I'm sure this isn't in any way new, but in 10+ years of using Linux, I just now encountered that scenario for the very first time. As Linux is becoming more popular, and more users are marching up the 'power user' scale, this is something that should be paid attention to - especially in a business environment. Yowza!

Standard Standards Rant, Redux: Why the World-Wide Web Isn't "World-Wide" Any More

The "World Wide Web", to the degree that it was ever truly universal, has broken down dramatically over the last couple of years, and it's our mission as Web development professionals to stand up to the idiots that think that's a Good Thing. If they're inside our organization, either as managers or as non-(Web-)technical people, we should patiently explain why semantic markup, clean design, accessibility and (supporting all of the above) standards compliance are Good for Business. (As the mantra says, "Google is your most important blind customer," because your prospective customers who know what they're looking for but don't yet know who they're buying it from find you that way.) Modern design patterns also encourage more efficient use of bandwidth (that you're probably paying for), since there's less non-visible, non-semantic data in a properly designed nest of divs than in an equivalent TABLE structure. Modern design also encourages consistent design among related pages (one set of stylesheets for your entire site, one for your online product-brochure pages, and so on). Pages that look like they're related and are actually related reassure the user that he hasn't gotten lost in the bowels of your site (or strayed off into your competitor's). It's easier to make and test changes that affect a specified area within your site (and don't affect others). It's easier to add usability improvements, such as letting users control text size), when you've separated content (XHTML) from presentation (CSS and, in a pinch, JavaScript). Easier-to-use Web sites make happier users, who visit your site more often and for longer periods, and buy more of your stuff.

Experienced Web developers know all this, especially if they've been keeping up with the better design sites and blogs such as A List Apart. But marketing folks, (real) engineers and sales people don't, usually, and can't really be expected to -- any more than a typical Web guy knows about internal rate of return or plastic injection molding in manufacturing. But you should be able to have intelligent conversations with them, and show them why 1997 Web design isn't usually such a good idea any more. (For a quick Google-eye demo, try lynx).  Management, on the other hand, in the absence of PHBs and management by magazine, should at least be open to an elevator pitch. Make it a good one; use business value (that you can defend as needed after the pitch).

That's all fine, for dealing with entrenched obsolescence within your own organization. What about chauvinism outside — from sites you depend on professionally, socially or in some combination? For years, marginalized customers have quietly gone elsewhere, with at most a plaintive appeal to the offenders, pointing out that a good chunk of Windows usees don't browse with Internet Explorer anymore (check out the linked article; a major business-tech Website from 2004(!!); the arguments are much stronger now). But some companies, particularly Microsoft-sensitive media sites like CNet and its subsidiary ZDNet, still don't work right when viewed with major non-Windows browsers (even when the same browser, such as Opera or Safari, works just fine with that site from Windows). And then there are the sites for whom their Web presence is the entire company, but they haven't yet invested the resources into competent design required to take their site construction from a point-and-drool interface virtually incapable of producing standards-compliant work, and instead present a site that a) actively checks for IE and snarls at you if you're using anything else, and b) has their design so badly broken and inaccessible that people stay away in droves. (Yes, I'm looking at you — every click opens a new window).

When we encounter Web poison like this, we should take the following actions:

• Notify the site owner that we will use a better (compatible, accessible, etc.) site, with sufficient details that your problem can be reproduced (flamemail that just says "Teh site sux0rs, d00d!" is virtually guaranteed to be counterproductive);
• When you find an acceptable substitute, let that site's owners know how they earned your patronage. Send a brief thank-you note to one or two of their large advertisers (if any), as well as to the advertisers on the site you've left (if you know any). Politely thank them for supporting good Web sites, or remind them why their advertising won't be reaching you anymore (as appropriate);
• Finally, there really ought to be a site (if there isn't already) where people can leave categorized works/doesn't-work-for-me notes about sites they've visited. This sounds an awful lot like the original argument for Yahoo!; I can see where such a review site would either die of starvation or grow to consume massive resources. But praise and shame are powerful inducements in the offline world; it's long past time to wield them effectively online.

I'm sure that there are literally millions of sites with Web poison out there, and likely several "beware" sites as well. For the record, the two that wasted enough of my week this week to deserve special dishonor are ZDNet and JobStreet. Guys, even Microsoft doesn't lock people out and lock browsers up the way you do; I can browse MSDN and Hotmail just fine on my Mac, on an old PC with Linux, or on an Asus Eee. And if you need help, I and several thousand others like me are just an email away. :-)

It's easy to think there's a war going on...

(playing softly, in the background of my mind, The Beatles'Revolution)

....between the Web developers promoting nice, clean development with RESTful, semantic (X)HTML judiciously enhanced with CSS and JavaScript (henceforth often referred to as the "Army of Light") and those using "popular", "mainstream" frameworks such as CakePHP and the Zend Framework, who route everything through a Front Controller of some sort, and often seem to be in the dubious company of WS-Whatever Web services (which, I am reliably told, provide ample amounts of The Wrong Kind of job security — they know more about your app than you do, and aren't telling what they know). The sides would seem to be pretty cut-and-dried, judging from a lot of the blog activity (Google REST XML-RPC PHP to get a million and a half or so hits of light reading material). Except...

Briefly skimming through the Zend Framework documentation, for instance, and looking at the QuickStart and tutorials reinforces the idea that URL handling is routed through a front controller to an application-specific action controller, which is the C in the notorious (and some say overused) MVC (model-view-controller) framework. Originally developed to help improve desktop-application development, particularly in languages like Java and Smalltalk, it became popular for Web development because.... it seemed like a good idea at the time. Actually, for Web development in the Pleistocene (say, late-1990s), it was a good idea. Anything that cut through the estimated 27.612 interconnected details that needed to be simultaneously mastered to get a "Hello, World" EJB up and happy was, by its very existence, a Very Good Thing. And so, when shops moved to more productive, less pathologically irrational development systems than J2EE, the models and design patterns that had saved their bacon were brought over into the New World, to maintain conceptual touchstones that helped Useful Work Get Done. Happiness abounded throughout the realm, until apps started outgrowing the meager bounds of static HTML and became "Rich Internet Applications". (To the tune of "Lions and Tigers and Bears, Oh My!", you hear faint murmurs of "AJAX and WS-* and REST, Oh My!") And, to pile on the snowclones, there really be dragons there.

'Dragons' in the form of falling into a GET-centric, action-oriented, everything-just-a-click-away world of convoluted Web apps with limited (re)usability and even less understandability to those who haven't swum there in some time. The entire promise of REST is simple: by centering applications around resources, rather than actions (through the use of URIs; Universal Resource Identifiers) and following the eminently sensible notion of not putting kilobytes of state information into those URIs (necessary information is POSTed along with the URI request), many problems that become painfully visible in large systems, simply go away. (Try sending a link to a cool book you found on Amazon over an instant messenger chat.)

But a typical, outsourced-development, haven't-really-used-this-tool-and-you-want-it-WHEN?!? developer isn't going to think of those things. He's going to grab a tool that has promising-sounding Google hits, run through a tutorial or two, and then plunge into the Son of the Enhancement of the Rewrite of yehey.com, with the customer sending him an "is it done yet?" email every six minutes. Clean design? What's that? Well-guarded state transitions? Who's got time to even understand that, let alone implement it? If we don't get it done, the customer's going to pull the project and send it to Vietnam or somewhere...

Just to make one point absolutely clear: I don't mean to be picking on Zend and CakePHP as being more than simply representative of widely-used, well-reputed tools that can be used to get the unwary, rushed developer (are there any other kind earning a paycheck?). While it is entirely practical to write semantic, RESTful Web applications in both frameworks (and both document how to do so), it's like, say, RPG; a fantastic tool for solving problems in a well-defined domain, usable with significant effort outside that domain, and Zeus help you if you use it to write an MMORPG.

The real point of this rant, if it hasn't hit you like a Muhammad Ali speed-anchor punch, is another pout over the state into which we've allowed the once-honorable craft of software development (of which Web development is but a specific case) into absolute bollocks. We've allowed the pay-any-price-to-cut-costs, pinch-a-penny-until-you-can-hear-it-scream-from-Boise-to-Bangalore idiots pervert us from Muhammad Ali (or at least Sonny Liston) into Herschel Shmoikel Pinkus Yerucham Krustofski. A plurality, if not yet an overwhelming majority of those who call themselves "software development 'engineers'" have been given neither sufficient formal training in their craft nor the resources (time, money, support, etc.) to continue learning as they go. "If you can spell EJB and ERP, you're the guy for us — as long as you're young and dirt cheap. And when you're done with that, we've got some BASIC code we want in Java instead."

So at the unique moment in history when ephemeral intellectual artifacts have assumed primacy in a wide range of human affairs, the humans whose intellect is responsible for their creation and correct functioning have progressively less ability to do the job properly. The way that, if they sit back and think for a moment, they know should be possible, has to be possible in any sort of rational omniverse whatsoever. But few, if any, ever get that chance for reflection. Fewer still, having reflected, researched and enlightened themselves, are welcomed back into the paying ranks who toil away at this once-noble craft.

And my Zend Framework code still feels slimy. It's not Zend's fault, at least, not entirely. Front controllers are good; front controllers are your friends; front controllers are.... *crunch!*

It's Time to Grow Up

Adrian Kingsley-Hughes over at ZDNet has a very interesting post up, titled "Sticking with XP / Upgrading to Vista / Waiting for Windows 7 / Switching to Mac or Linux — There’s no single right answer". He puts forward the blitheringly-obvious elephant-in-the-room answer to the perennial food-fight question, "Which system is best?" Namely, "use what works for you."

As I read through the post and the comments that had been added to it, I thought about all the man-centuries (if not -millennia) that had been "invested" in the topic. Naturally, I had my own two rupiah worth to say on the topic. Following is the text of the comment I left at approximately 1645 GMT on Friday 27 June. Let me know what you think. (There aren't any links in the post reprinted below; to the best of my knowledge, ZDNet's commenting software hates links, and it definitely hates Macs; every single comment I've posted has given me the error "You must enter the text to post" — on a clean, empty comment form — after I've hit the "Add your opinion" button. Fix it, guys!

---

"Use what you like, and like what you use."

That's excellent advice for those of us who've been bouncing around in the funhouse for a while, who know which mirrors make us look weird (or, worse, are broken and likely to cut us if we're not careful)...and, granted, there are blessed few truly "new" users now; statistically, nearly everybody's used a PC with one form or another of Windows, and increasing numbers of us have used Mac and/or Linux, but...

We still do have the FNG syndrome with folks who haven't upgraded for a while, and finally they get tired of their molasses-slow Win98 box when they see this zippy new PC or Mac they've been handed at work. "Gee, we use XP at work, but I heard Microsoft isn't going to sell it anymore... what should I use?" Many of us, professionally or otherwise, are tasked with advising those people. Too often, the advice becomes "this is what I use; try it" without really understanding the (often vast) difference between the adviser and the user in question. And when "advisers" try and hash things out among themselves, it almost universally degenerates into an Animal House food fight scene — which doesn't bring any value to the discussion and actually makes us LESS able to give good advice.

Mental hands up: How many of you reading this have ever spent a month using Leopard? Vista? At least two of Ubuntu, Fedora or SuSE Linux? How many raised your hand all three times? Yeah, I see you, way back in the back.... but blessed few others.

In any other endeavour that dared call itself a craft, let alone aspire to an engineering discipline, this would be malfeasance if not negligence; you just DO NOT give advice on matters in which you are not qualified — and if you don't have experience and/or training with Technology X, you're NOT qualified to present advice as being any more valuable than used toilet paper.

In that light, Adrian has performed a major public service here. Given the reality that most people whose job relies on using and/or developing for one of the major platforms are quite unlikely to be as current and proficient on any of the others, this is the best advice that will just let us get along with our jobs without pissing in each other's lemonade each and every single day (Mike Cox and No_Axe, you know who you are).

But in the increasingly unlikely event that we're ever to make something professional out of this hobby that we are lucky enough to get paid for, the fact that this "solution" is seen as viable to any degree, let alone the MOST viable solution, is absolutely, reprehensibly unacceptable. And since computers and software have become absolutely central to nearly everything in modern life, including not least public policy, if we don't get our house in order under our own power, sooner or later some governmental organization or group thereof is going to step in and exert adult supervision. Is that what we want?

The days when Windows geeks and Mac users and Linux hackers could happily putter away, each in their own walled garden with tactical nuclear landmines guarding against any encroachment by reality, are gone as surely as the clipper ship. In the world of the Internet, where information is what's important and how it's processed/generated/visualized/stored is at best secondary, we're faced with the same choice as every biological or cultural organism at an evolutionary shift: adapt or die. Keep the lemonade clean, or drink the purple Kool-Aid. Our choice. Each and every one of us.

Browser Support: Why "Internet Explorer 6" Really Is A Typo

(Experienced Web developers know that the correct name for the program is Microsoft Internet Exploder — especially for version 6.)

Case in point: I was browsing the daringfireball.net RSS feed and came across an article on the 37signals blog talking about Apple's new MobileMe service dropping support for IE6. The blog is mostly geared towards 37signals' current and potential clients who, if not Web developers themselves are at least familiar with the major technical issues involved. Not surprisingly, virtually every one of the 65 comments left between 9 and 13 June was enthusiastic in support for the move; not because the commenters necessarily favor Apple (though, clearly, many do), but because anybody who's ever cared about Web standards knows that IE6 is an antediluvian, defiantly defective middle finger thrust violently up the nostril of the Web development community; the technological equivalent of the Chevrolet Corvair: unsafe at any speed.

The degree to which this is true, and to which this truth continues to plague the Web developer and user communities, were brought into sharp focus by three of the comments on the post. The first, from 37signals' Jason Fried, estimates that 38% of their total traffic is IE, of which 31% is IE 6.0 (giving a grand total of 11.8% of total traffic — not huge, but significant).  The second is from Josh Nichols, who points out that Microsoft published a patch to solve the problem with IE6 in January, 2007; he notes, however, that an unknowable number of users may not have applied that patch. Finally, Michael Geary points out that later versions of Internet Explorer (version 7 and possibly the now-in-beta Version 8) also have the problem of not being able to "set cookies for a 2×2 domain, i.e. a two-letter second level domain in a two-letter top level domain with no further subdomain below that," (including his own mg.to blog domain). The fact that relatively few domains fall into that category can be argued to be part of the problem; users running IE, particularly an out-of-date version of IE, are likely to be less experienced, less able to recognize and solve the problem correctly, than to blame it on "something wrong with the Internet". For those people and companies who've paid for those perfectly legitimate domains, the negligence and/or incompetence of the browser supplier and/or user mean that they're not getting their money's worth.  And ICANN, the bureaucracy "managing" the domain-name system, is now "fast-tracking" a proposal to increase the number of top-level domain names (TLDs) used. (In time-honored ICANN custom, the press release is dated 22 June 2008 and "welcome[s]" "Public Comments" "by 23 June 2008." Nothing like transparency and responsiveness in governance, eh?

Good Things are good things....aren't they?

Anybody who's worked with me over the last 20 years or so knows that I generally evangelize conforming to standards when they exist, are relevant and widely agreed on. As the famous quote from Andrew Tanenbaum (in Computer Networks, 2/e, p. 254) reminds us, "The nice thing about standards is that you have so many to choose from." When "standards" are used to promote vendor agendas (e.g., Microsoft force-feeding OOXML to a hapless ISO) or when they go against the common sense built up through hard-won experience by practitioners. And when multiple standards for a product or activity exist, and those standards are each widely used by various users (who could have chosen other alternatives), and when those standards conflict with each other in important ways that can't be amicably resolved, then those "standards" cause reasonable people to not merely question their validity, but, too often, the entire concept of "standards".

As any developer who's worked in more than one shop, or sometimes even on more than one project in a shop, knows, coding standards are sometimes arbitrary, often the prizes and products of epic bureaucratic struggle, and (in the absence of automated enforcement such as PHP CodeSniffer) often honored more in the breach than the compliance. What makes things even more "fun" is conflicting standards. It's not all that unusual for a company to contract out for development work, specifying that their coding standards be complied with (since they're the customer and they're going to maintain, or control maintenance of, the code). If the contractor has their own set of standards that conflict with the customer's, then problems arise with internal process compliance, customer involvement and final delivery. It can be — and too often is — a sorry mess. Simple code reformatting problems can be taken care of with a pretty-printer program; oftentimes, though, one sees entire programs (which have to be debugged, documented and maintained) developed just to "translate" one format to another. Many shops just give up, declare the project to be an exception or exemption from their own internal standards and processes, and try to conform to the customer's demands. "Try to", since their developers, both writing and reviewing the code, are going to be fighting against it tooth and nail because it just "feels wrong".

This whole rant was inspired by reading through yet another coding-standard document; this one the Zend Framework PHP Coding Standard. One item in particular struck me as counter-intuitive. In Item B.2.1.1, PHP File Formatting — General, it says:

For files that contain only PHP code, the closing tag ("?>") is never permitted. It is not required by PHP. Not including it prevents trailing whitespace from being accidentally injected into the output.

Experienced PHP developers are quite likely to have problems with this, not least because it conflicts with earlier behavior of the PHP interpreter and with tools that expect well-formed code. This is one of the oddities which tools like the aforementioned PHP CodeSniffer need to take into account. (There are other, more blatant "yellow flags"

If you're in a shop which takes standards seriously, uses PEAR code and uses the Zend Framework, your code review meetings are likely quite interesting.

• "OK, we're going to look at foodb.class.php first, and then the others I mentioned in the email yesterday."
• "Which standard does it use?"
• "Well, it ties in with PDO, so it ought to follow the PEAR standard, right?"
• "OK, that sounds reasonable."

As the meeting continues...

• "Hey, what's this at the end of omnibar.class.php? There's no 'close-PHP' tag! If we start using the code-search Wiki plugin that the Bronx Project folks keep raving about, it's not going to like that...."
• "Oh, yeah, but that's because  it uses all this Zend Framework stuff, so we use Zend's coding conventions... see that comment at the top about how to run CodeSniffer?"
• "Riiiiight...."

and so on. Weren't process and standards supposed to make development easier and more reliable?

I agree with the sentiment, apocryphally attributed to one or another of numerous software gurus, that, in the presence of otherwise adequate and sufficient standards, we shouldn't be so "egotistical" as to think developing a "better" standard than others already have is worth our time; take what's already out there, adapt as necessary, and move forward. The trick, of course, is in evaluating that condition, "otherwise adequate and sufficient." Also, since our craft is (hopefully) continuing to advance and adopt standard patterns for things done before, striking out on your own (after careful consideration) demands that the question be revisited from time to time. What are other development groups using (broadly) similar techniques to solve (broadly) similar problems using? Is a consensus forming, and do we have anything useful to say about it? Or has a single standard already taken hold, and we can take advantage of it (at least for new or reworked code)?

Code analyzers like lint and PHP CodeSniffer can be amazingly useful. But for them to function as standard/policy enforcement tools, there must be a standard, or a small group of similar standards for them to enforce. When development teams have to juggle between incompatible standards, it discourages them from following any standards. And in that direction lie... the 1970s.

g++ != gcc (arrrrrrgh!)

Coming back up to speed on Mac programming, now that I've finally got a shiny new iMac. Their XCode IDE looks like a great tool (Objective C, C++, C, etc., etc.), but I was hacking around building some simple test code. Being a fully-certified Unix operating system, of course that's an easy way to get something done while minimizing the number of known and unknown unknowns that need to be dealt with.

This mostly transpired between 2300 Sunday and 0130 Monday (last night/this morning). I installed CppUnit (Boost was already on the system), and kept running into the same problem.

jeffs-imac:Foo jeff$ gcc -I /opt/local/include -I /opt/local/include/cppunit foo.cpp -L /usr/local/lib  -lcppunit  -o foo
ld: in /usr/local/lib, can't map file, errno=22
collect2: ld returned 1 exit status

Hmmm. Maybe the library that's on there is screwed up somehow? Go to Sourceforge, pull down the library source, build it, install it, and try again.

Same thing. Fiddle with the code, fiddle with the command line, nothing fixes it. Go out and Google for help. The very first hit, from MacOSXHints, had a silly-sounding but tantalizing "clue":

You're right. I figured out the problem is that I was missing the -c switch when building the .o file with gcc. For some reason the linker doesn't complain about it, but when I try to link the shared lib with my main program I get the obscure can't map file error. Now it is working. Thanks.

Hmmm again. Go fiddle some more, this time compiling and linking my trivial proof-of-concept in separate gcc command lines. Still no joy.

I go back and play with building libcppunit again, wondering if I've missed some funky option to configure. Nope. It's pushing 0130, I need to be up at 6-something, and my brain is fried, so I shut down for the night.

(Later) in the morning, something's niggling at the back of my mind, saying I missed something while watching libcppunit compile, so I do it again. Yep, cue the "you dumb palooka!" moment: it's not using gcc to compile; it's using g++. For those who've never had the (dubious) pleasure - gcc is the "general-purpose" front-end to the GNU Compiler Collection, a set of (numerous) language systems, of which g++ is the C++-specific toolchain (and standalone front end). All compilers in the Collection produce object code in compatible format (the same back-end is used for almost everything), so usually all you have to do is invoke the One Command to automagically compile your Ada, FORTRAN, Java, PL/I, whatever. And, to be honest, it had been a few months since I'd dealt with C++ on gcc/g++ from the command line. (Thank you, Eclipse!) But I remembered a bit of wisdom lost in the Mists of Time...

They can both compile your C++ (in a single step). But: They're. Not. The. Same.

So, I re-do my (now-) two-step process, substituting g++ for gcc:

jeffs-imac:Foo jeff$ g++ -c -I /opt/local/include/ -I /usr/local/include/cppunit/ foo.cpp
jeffs-imac:Foo jeff$ g++ foo.o -L/opt/local/lib -lcppunit -o foo
jeffs-imac:Foo jeff$

Ta-daaaaa!  OK, can we go back to a single step? That is, compiling and linking (using g++) in one go:

jeffs-imac:Foo jeff$ g++ -I /opt/local/include -I /opt/local/include/cppunit foo.cpp -L /usr/local/lib -lcppunit  -o foo
ld: in /usr/local/lib, can't map file, errno=22
collect2: ld returned 1 exit status
jeffs-imac:Foo jeff$ 

Nope. This is where the MacOSXHints commenter was right on the money. But why? I used to be knee-deep in the (FORTRAN-specific) code, a feeling akin to being knee-deep in the dead on occasion, and the answer doesn't come immediately to mind. Any ideas?

ANFSD: starting a series to scratch an itch

(And Now For Something Different, for the 5LA-challenged amangst you...)

I've made my living, for about half my career, on the proposition that if I stayed (at least) three to six months ahead of (what would become) the popular mean in software technology, I'd be well-positioned to help out when Joe Businessman or Acme Corporation came along and hit the same technology — with the effect of "Refrigerator" Perry hitting a reinforced-concrete wall. This went reasonably well as "the market" started using PCs, then GUIs, then object-oriented programming, and then "that Internet thingy" (Shameless plug: résumé here or in PDF format).

In other ways, I've been a staunch traditionalist. I've used IDEs from time to time, because I was working as part of a team that had a standard tool set, or because I was programming for Microsoft Windows and the Collective essentially requires that that be done in their (seventh-rate) IDE unless you want to decrease productivity by several dozen orders of magnitude.

Otherwise, just give me KATE or BBEdit and a command-line compiler and I'm happy. This continued for a significant chunk of the history of PCs, until I decided that, for the Java work I was doing, I really needed some of the whiz-bang refactoring and other tie-ins supported by Eclipse and NetBeans. Then I started hacking around on a couple of open-source C++ packages and thought I'd give the Eclipse C/C++ Development Tooling a try. Now I'm coming up to speed on wxWidgets development in C++.

During this learning-curve week, I spent a lot of time browsing the Web for samples, tutorials and so on. To call most of them execrable is to give them unwarranted praise. Having recently resumed work on a Web development book dealing with useful standards and helpful process, and since I've been doing C++ off and on since the mid-80s, I thought I'd start a series of blog entries that would:

• Document some of the traps and tricks I hit to get a simple wxWidgets program into Eclipse;
• Illustrate some early, very simple refactoring of the simple program to get a bit more sanity;
• Get Subversion and Eclipse playing well together;
• Explain why I think parts of teh Agile method are simulataneously nothing new and the best new idea to hit development in a very long time.
• Start using an automated-testing tool to build confidence during debugging and refactoring; and
• Using a code-documentation tool in the spirit of JavaDoc to produce nice technical/API docs.

At the end of the series, you'll have a pretty good idea of how I feel most projects (regardless of underlying technology and specific tools) "should" be done. You'll have seen a very simple walk-through of the process, demonstrated using Linux, Eclipse, C++ and wxWidgets, but actually quite broadly applicable well beyond those bounds.

Please send comments, reactions, job offers, etc., to my email. Death threats, religious pamphlets, and other ignorance can, as always, go to /dev/null. Thanks!

Oh. Mah. Gawwwwwwwwd.

You no longer need to reboot a running Linux system to apply security patches to it.

Check it out.

Excuse me whilst I pick my jaw up from the sub-sub-sub-basement floor. If this checks out in the field, on multiple distros, then a lot of sysadmins are going to be able to get a lot more sleep at night. And the comment by one David Pottage:

This should be good for distro kernels.

Just think if you can prepare a special kernel module that will apply security patches to a running kernel, then so can your favorite distro. In future when there is a security update, instead of downloading a ~20Mb kernel package from security.debian.org or the like, and then waiting until a suitable time to install it and reboot the system, you can download a small package containing patching modules for the standard kernels from that distro, and install it immediately...

The mind reels a bit. Security patches are the most gotta-do-it-right-NOW things that come down the pipe for any system. Open-source systems that are widely audited, like Linux, tend to get patches a lot quicker than Windows (which had attacks in the wild with no fixes available for some 271 days in 2007), or even Mac OS X. Any closed system that depends on a single organization to secure it will always have slower reaction time than an open system with enough (mutually independent, distributed) resources to throw at it. As Eric S. Raymond wrote in The Cathedral and The Bazaar, "given enough eyeballs, all bugs are shallow". As long as there is some meritocratic control over the "official" patch-submission process - and there is - it's now easier than ever to keep critical systems up and secure, in ways and at speeds that simply can't be matched in the Microsoft world. Remember, even if your uptime is 99.999%, (the so-called "five nines gold standard"), you're still down five minutes and fifteen seconds every year. Murphy's Law says at least five minutes of that time will be when it was really, truly important that the system not be down.

You can't repeal Murphy's Law, but I think it does give us a big step towards an insurance policy.

Rant - can people at least engage brains before asking stupid questions? (Or: Paging Andy Rooney)

I read several different development blogs and message boards, such as those associated with phpclasses.org, codeproject.com, IBM DeveloperWorks, and so on. Usually pretty useful both for learning new techniques and keeping an eye on what other people are doing. Several of the boards, particularly on CodeProject, have been getting hot and bothered lately about the declining quality of questions being asked on the public lists; one large category of these is "affectionately" known as "homework questions'. These usually aren't for actual classwork. A more typical scenario seems to go like this: Sanjay is brought into an outsourced project because his agency assures the client that he's a hotshot — fully qualified in J2EE, PHP, XML, RPC and LSMFT (obviously a key qualification, but since the client HR person is neither technical nor over 40, it just appears to be "tech jargon.")

The client manager thinks, "I'll have to let a couple of my other guys go to stay in budget, but if this guy can save our bacon, it's worth it." Sanjay starts one bright Monday morning at 9.00,, gets the usual here's-where-things-are spiel, sits around (billing time) waiting for his computer to be set up and connected to the network (these things almost never happen before the warm body shows up), and by 1 PM is browsing the codebase for the project. By 1.30, he's on the Web, posting questions on sites that make it absolutely, crystal clear that he wouldn't know his ear from a hole in the ground if you gave him a flashlight, a map and six hours' head start.

The most extreme example of this I've personally witnessed was when I was part of a team consulting to a major Southeast Asian telecoms firm, working on integrating the homegrown billing system that one division used into the (telecom-)industry-standard one used by most of the rest of the company. This benefited greatly from knowledge of Java, of both Linux and Windows, and especially of the commercial system which was the target for the migration. The firm which produces this system, in true Java-ecosystem fashion, offers their own training which leads to certification in various aspects of the system. The team had a good mix of knowledge and experience, with the single yellow flag being that the major new-system expert was a member of the client's staff. After being encouraged to bring our own domain expert in (apparently so the client could reassign the existing one if desired), our headquarters (in India) found the "perfect guy". He had Java and J2EE certificates. He had a BSCS from "one of the top universities" in India. He had all the certifications the vendor offered. Oops, they were all for the previous version of the system....but this guy could, on paper, walk on water as far as our project's needs were concerned. So we flew him out from Bangalore. We had, as I recall, six weeks before major schedule slips would hit the fan.

A month later, my team manager and I sat watching this guy type in sample programs from the manual, try to build them, watch them fail, erase everything and start over. The manager said to me, "well, why don't you and (the junior guy on the team) start trying to pick up the pieces? Maybe you can pull something out." Then we noticed something else that was interesting. The client manager kept saying nice things about the "expert", how diligent and hardworking he was. He started taking the expert out for lunch and so on. My manager and I were in shock: this guy had already cut into the project for thousands of dollars and had yet to produce a single project artifact. Worse, in our view, his "experience" and "qualifications" were obviously complete fabrications. Headquarters (back in India), however, wouldn't send out a replacement because they had been told that the client was happy with the guy. We started looking around for brown-projectile-proof mackintoshes, anticipating the storm sure to come. And it did, eventually — shortly after the "expert" failed to follow explicit, idiot-proof instructions on how to extend his visa so he wouldn't have to go back home. Instead of buying a round-trip bus ticket to Singapore, he bought a one-way ticket, later claiming he wasn't sure which bus he'd want to take back up. The Singapore immigration officials blocked him from entering Singapore as they had no reasonable assurance of when he'd leave. The Malaysians wouldn't let him back in because his (tourist) visa had expired that day and he'd need to enter another country before reentering Malaysia. After several frantic midnight telephone calls and (I was later told) negotiation and pleas from our firm and the client, he was unceremoniously dumped on a plane back home, and our firm was billed for a full-fare coach ticket. The client manager became upset because his buddy, the "expert" was nowhere to be found and on no notice, at that.

Why am I blathering about all this now, several years after the event? My ire was raised by one of these "homework questions" I'd mentioned earlier, this time posted on the WeberDev general PHP forum. Entitled "Which is compatible PHP or Java with SQL SERVER 2005", the writer, using the nick "kumarsudu", starts out with...

Hi All, I would like to know which one is have compatible 1. PHP with SQL SERVER 2005 2. Java with SQL SERVER 2005 as i have to develop a project, ...

As another popular forum's members often ask, "how many WTFs are there in this message?" The mind reels. Asking totally nonsensical, absolutely open-ended questions on technical sites and lists is now at a flood stage not seen since the AOL infection of the Internet back in the early 1990s. The writer, besides the sterling specificity of the question as mentioned earlier, shows little knowledge of or interest in proper use of the English language (despite ample materials and information available online as well as off).

It pains me that:

• a person of such deliberate ignorance and lack of brilliance would not only choose to waste my and other readers' time with the request to do his work for him;
• any contracting agency would have such miserably low non-standards that this guy would even get the time of day, let alone a job for which there are an ample supply of other (by definition more qualified) candidates for (granted, they may have to raise their pay to a level higher than a fast-food burger-flipper);
• any client company would not only pay money for such an individual, but continue to do business with the "recruiter" that brought him in;
• there are no project-local, competent individuals whom this writer could ask for help, who would apply appropriate informational and organizational responses to the question; and finally
• that the prevailing business contempt of software development has sunk so low that this type of thing is more unusual for the fact that anyone bothered to notice it than that it happens at all. Would you want to fly with an airline whose pilots were the cheapest available, using forged certificates and qualifications to highlight (non-existent) experience? If your answer to that question is anything less emphatic than "hell no!," please inform me of your flying plans so that I may ensure that I am neither flying nor anywhere on the ground along your route during such a flight.

The "project" mentioned by the initial writer, if performed by individuals of this apparent calibre, is highly likely to fail — wasting the client's time and money, leaving the client wih a problem that still needs to be solved, and continuing to erode the opinions of the client and the client's associates of the business value of software, since, obviously "IT projects always fail."

Buzz About the Code Buzzards

Scott Hackett over at SlickEdit has a blog entry where he talks about code scavenging as "a new software development methodology." The point being, of course, that code scavenging isn't new at all; it's at least as old as the UNIVAC I. Every new, and not so new, software developer has used it at one time or another, usually for quick bootstraps to get a specific part of the software under development working in spite of limited knowledge of the language and/or domain in question by the programmer, or limited resources (time/budget). Most times, it's at least felt to be some combination of the two.

What's changed recently, and what makes five-finger coding a screaming, begging candidate for formalization, as Mr. Hackett notes, are two distinct phenomena: the rise of new repositories on Web sites like Koders, Krugle, Google Code Search, among others, that at least hold the promise of finding "what you need right now", right now. Also, development professionals face ever-increasing resource constraints ("can you get that for us yesterday?") in a craft that is continually expanding in scope and detail.

What I found interesting, having visited sites like Krugle many times in the past, was the effect that open-source licensing like the GNU General Public License and the BSD License are having on one of the main limiting factors that Mr. Hackett identifies: trust. Many developers, for various reasons, don't trust the work of someone they don't know when their own jobs or reputations are on the line. What open source and the new search engines bring to the table is the idea that now, you don't always have to just accept a code fragment "on faith". Many of the more experienced developers have a track record of other code that they've written, often accessible via the same search engines that helped you find the one you're looking at. Open source and free licensing mean that you can look at other stuff that the guy in question has written, get a feel for how his style and competence match your own, and use that information as additional basis for evaluating the code fragment you're scavenging. The trick, obviously, is to keep the time and effort required for all this significantly below what it would take for you to write your own implementation from a clean sheet of paper. That's why search engines and good indexing are important.

Of course, none of this will help you make the changes that always have to be made to put a piece of "foreign" code into your handiwork. That's still going to take some work; hopefully not on the order of putting a Porsche engine into your Ford pick-up. We'll have to wait for — and work for — serious improvements in the state of the software development craft to have any effect on that. Development methods, languages, and so on are still in the cathedral at Rheims. I've been waiting 30 years for the binary Renaissance to hit my professional life.... anybody need a virtual stonemason?

Improvement as opposed to Change

A link on the Agile CMMi blog had some very interesting things to say about a gentleman named Brian Lyons, the CEO (as well as CTO and founder) of Number Six, a Vienna, Virginia (Washington, DC area) software technology services/business consulting firm. Number Six quite obviously 'get' the concepts behind Agile CMMi, Hillel Glazer of the blog wrote. Interested, I flipped over to their site to hopefully learn a bit more, maybe drop off a CV.

Mr. Glazer wrote the blog entry in question on 25 July. The first thing you notice when viewing the Number Six site is the home-page obituary of and tribute to Brian Lyons, who died on (Monday) 3 September 2007 in a motorcycle accident. There are various links to family and tribute sites, a press release, and a request that "in lieu of flowers, the family has requested that donations be made to" a scholarship fund at the University of Maryland (excellent, particularly for those too far or too late to attend the funeral). I wish the family and company all the best in their times of grief and trouble.

The point I originally started writing this entry to make, however, was inspired by one of the bullets on Number Six' careers page, under the heading "Why Six?": "We are committed to consistent improvement, not continual drastic change."

Think about that distinction for a moment. Many of us, particularly those in the software-geek persuasion, start our careers trying to remake everything; not necessarily because it's broken, but so that we can do it (whatever it is), make it ours, introduce new techniques or technologies, and hopefully (but improbably) provide a better solution to the problem at hand than what was there before.

The forces arrayed against that impulse are formidable. Playing the role of (and often in actual fact) those who are by nature suspicious of any radical approach to a problem, too often dismissing out of hand any alleged improvement or innovation without "giving it a fair chance", in the eyes of the wet-behind-the-ears Young Turk. Occasionally, of course, improvements and innovations too beneficial to ignore do come out of this process.

As the developer matures (which may take days, decades or eternities), the Young Turk recognizes that not all of the hindrances were traditionalist per se; rather, they were operating from a different (usually business-oriented) set of priorities. Learning to understand and appreciate those priorities is one of the fundamental aspects of any successful transformation from ivory-tower Geek to journeyman software craftsman (or -woman), able to make a contribution to customers and to the craft of software development.

What becomes obvious, to all thoughtful participants and stakeholders in the process of developing and maintaining any software system is that a natural tension exists between three apparently conflicting ideas:

• Don't change what works well enough when there are more pressing needs to attend to;
• New capabilities, chosen and implemented properly, can have strongly beneficial effects (efficiency, productivity, wider customer scope, 'better' according to various aspects of quality);
• Any change to an existing system involves direct and indirect costs which must be carefully evaluated and compared against the expected improvements.

Balancing and managing the conflict between those concepts throughout the lifetime of a piece of software is an art that has yet to be thoroughly and reliably mastered with a level of reliability and cost widely acceptable to business customers. Several philosophies and techniques (often marketed as technologies) have been developed and marketed as (attempted) solutions to that problem. Three that I have found extremely useful, in complementary ways, over the last few decades are agile development, refactoring and the CMMI. (Those already familiar with the concepts may skip down to the paragraph which begins "As argued by practitioners such as Hillel...".)

The CMMI (previously more widely and less precisely known as the CMM, for Capability Maturity Model) has been around for quite some time as a formal means of evaluating the maturity and coherence of an organization's development efforts (among other activities). It requires its users (organizations) to acquire and document ever-increasing detail of precisely how they go about the process of development and how they are required to be able to prove (in an internal or external audit) that those processes are being followed and any discrepancies noted and accounted for. This suffers as a standalone policy framework for software development on two grounds:

• the "what" and "why" of development are completely ignored. CMMI fundamentally doesn't care if you're building a pile of junk that has serious practical problems, as long as you do it using the process you've defined for yourself.
• The CMMI, along with US DOD Standard 2167A, have been blamed for the logging of more trees (to make paper) than virtually any other industry business practice. This is largely because both are seen (in CMMI's case, somewhat unfairly) as pushing "hidden mandates" for the rigid , never-far-from-obsolete "waterfall" model of development.

Agile development, on the other hand, is a survival response to both the (largely management-driven) mantra that "Real Artists Ship", and on the other hand, to seemingly interminable periods between "milestones" (particularly in the waterfall model) where nothing of intrinsic value is visible to an outside observer (e.g., management or customers). Its main criticism from opponents is that it produces too little documentation (the product is the documenatation, to a large degree).

An aspect of, or adjunct to, agile development is refactoring, pioneered and popularized in the Martin Fowler-written book. Refactoring is all about how you can make (sometimes radical) changes to the internals of a software system, provided you keep the external (customer-facing, revenue-generating) interfaces constant. Wrapping one's mind around this as a formalized process, as opposed to the "don't-break-what-works" mentality common to experienced developers (and managers), is a threshold experience in a software craftsman's progress.

As argued by practitioners such as Hillel, the combination of the process-centric CMMI and the solution-centric, visible-progress Agile set of processes (including, particularly, refactoring) gives the "best of both worlds" for development — a focus on artifacts and products created through a survivable process, matched with a demonstrated and documented knowledge of exactly what is meant by "that process" under current circumstances. This also implies demonstrating understanding of the specifics of "current circumstances" that drive the decision-making and artifact-development processes — which brings us back full circle to understanding why Agile is a good fit to begin with.

Unlike the Rational Unified Process (RUP), which grew out of a waterfall-driven management-oriented mentality (and the opportunity to sell software tools and consulting services to any shop using the heavily-marketed process), Agile and CMMI development can be successfully started after passing around a few books among the development and management staff. Tools and training are, chosen thoughtfully, extremely helpful, and the appraisal process within CMMI (formally SCAMPI, the well-known "Level 1" to "Level 5" assessment of process maturity) does eventually require an outside assessor, or registrar, to formally certify the organization. But to take a typical small development group from the customary initial chaos to a finely-tuned, market-leading, customer-satisfying machine, it has been my experience and observation that it is far easier (and more cost-effective) to implement CMMI in an Agile fashion than to go down the (vendor-specific) RUP route. Imposing either on a large organization would require an unlikely combination of managerial brilliance, sadism and masochism; revolutions boiling up from below and winning eventual management sanction have proven much more likely to be successful in that type of environment.

So what's a Young Turk (or a more seasoned craftsman) to make of all this blather? The simple point: Consistent improvement is practical, and without life- or career-threatening implications, achievable on a repeatable basis. Continual drastic change, in contrast and almost by definition, will lead the development group to many sleepless nights trying to get that last show-stopper bug fixed, management to wonder why those bozos in development can't be trusted to ship anything on time, and customers to wonder what they're really getting for their money beyond the hype and bling of the marketing materials. Which team would you rather be on?

Crap Is Not a Professional Goal

I recently, very briefly, worked with a Web startup based in Beijing (the "Startup"). The CEO of this company, an apparently very intelligent, focussed individual with great talent motivating sales and marketing people, takes as his guiding principle a quote from Guy Kawasaki, "Don't worry, be crappy".

The problem with that approach for the Startup, as I see it, is two-fold. To start, Kawasaki makes clear in his commentary that he's referring strictly to products that are truly innovative breakthroughs, exemplars of a whole new way of looking at some part of the world. Very few products or companies meet that standard (and even if yours does, Kawasaki declares, you should eliminate the crappiness with all possible speed). No matter how simultaneously useful and geeky the service offered by the Startup's site is, it is, at best, a novel and useful twist resting on several existing, innovative-in-their-day technologies. (A friend who I explained the company to commented that "it sounds as innovative as if Amazon.com only sold romance novels within the city limits of Boston" — hardly a breakthrough concept). Indeed, Kawasaki makes clear that he's talking about order-of-magnitude innovation; the examples he cites are the jump from daisy-wheel to laser printing and the Apple Macintosh.

The second, more insidious, problem with the approach, and the trap that many early-1990s Silicon Valley startups fell into, is that you take crappiness as a given, without even trying to deliver the one-two punch of true innovation and a sublimely well-engineered product that immediately raises the bar for would-be "me-too" copycats. (Sony, for instance, has traditionally excelled at this, as has the Apple iPod (learning from the mistakes Kawasaki cites for the earliest Macintosh). Deliver crap, and anybody can compete with you once they understand the basics of your product. The wireless mouse is a good example of this.

If you tell yourself from the get-go that you'll be satisfied if you ship a 70%-quality product, what will happen is that, as time goes by, that magical 70% becomes 50%, then 30%, then whatever it takes to meet the date you told the investors. And if management doesn't trust engineering to give honest, realistic estimates (as is typical in software and pandemic in startups), you have a recipe for disaster: engineering takes a month to come back with an estimate that development will take 12-18 months; management hears '12' and automatically cuts that to 6 and pushes to have a "beta" out in 4. The problem is that, if you're dealing with an even marginally innovative product, things are not cut-and-dried; the engineers will have misunderstood some aspects of the situation, underestimated certain risks, and been completely blind to others. This was pithily summed up, in another field entirely, by Donald Rumsfeld:

There are known "knowns." There are things we know that we know. There are known unknowns. That is to say there are things that we now know we don't know. But there are also unknown unknowns. There are things we don't know we don't know. So when we do the best we can and we pull all this information together, and we then say well that's basically what we see as the situation, that is really only the known knowns and the known unknowns. And each year, we discover a few more of those unknown unknowns.

Companies that simultaneously forget the ramifications of this while taking too puffed-up a view of themselves are leaving themselves vulnerable to delivering nothing more useful or profitable than the old pets.com (not the current PetSmart) sock puppet — and probably nothing that memorable, either.

And that further assumes that they don't fall prey to preventable disasters like losing the only hard drive running a production server built with non-production, undocumented software. If Web presence defines a "Web 2.0" company in the eyes of its customers and investors, going dark could be very costly indeed.

Back in the Saddle, Again

....with abject apologies to Gene Autry...

I haven't posted here for a while (about three months — gak!). For the six or eight of you still hanging on, humble apologies and my deepest appreciation (sympathies?). Some have publicly wondered (offline) whether I am merely offline or have flatlined. Actually, I've been in hospital twice during that time, and my personal and professional lives have undergone more than the usual random fluctuations. Be that as it may...

As with roughly half of the Linux-aware folks out there, I've been playing with Ubuntu Linux for a while now. The job I just started — as Principal Technologist and alleged future CTO for FoneVillage.com in Beijing — is with an Ubuntu shop, so that's one motivation. I've been a Debian evangelist for a few years now — formerly a Kanotix (now Sidux refugee, now with Ubuntu and Mepis installed and happy on laptops and a desktop (and lusting after a Mac Pro (but that's another blog entry)...

Half of me LOVES Ubuntu. Point-and-click everything; all the applications (except mainstream violent Windows games) that a user could want immediately available, name-brand Big Applications for the enterprise; more-solid-than-most-rocks Debian under the hood; regularly updated Live CDs (but get the better Live DVD); what's not to like?

The other half of me, the guy who's been intimate with the care and feeding of Unix systems for almost 30 years, has an easy answer for that; sudo (as superuser/SystemGod, do) everything, but in particular, sudo bash (as superuser, open up a shell [terminal] and let me run arbitrary commands with no restrictions).If you read Ubuntu guides and Web pages, almost everything a user does from a command shell that affects the system is done as sudo command, while logged in as an ordinary user. A bit of poking around with Google led me to a page on About.com's Ubuntu Desktop Guide that put things into better perspective:

The first user account you created on your system during installation will, by default, have access to sudo. You can restrict and enable sudo access to users with the Users and Groups application.

My knee-jerk reaction having subsided, I'm back to generally liking what I see in Ubuntu. It's intended to achieve — and generally succeeds at — being "easy enough for anybody to use", not just "geeks", as Linux has heretofore been viewed by Windows usees. It's another answer to the classic "what's the difference between a Windows usee and a Mac user?" question: The Windows usee talks about everything he had to do to get his work done; the Mac user (or, generally, the Ubuntu user) talks about all the great work she got done.

For the technophobes out there who still want to join the modern world, definitely worth a spin.

If You Can't, It Doesn't

If you can't measure something, can you prove that it exists? Or at least that you understand it? If we're having a theological discussion, you might have one answer, but in the material world, and more specifically in technology development, the answer to both questions is "probably not". To quote what several nurses and doctors tell me was pounded into them during their training, "if you don't write it down, it didn't happen" — because there's no record that it happened the way you think you remember it. Memory's a funny thing — even about events that actually happened.

This less-than-breathtakingly original, but relevant, train of thought occurred to me after I spent an hour kicking the tires of the PHPUnit code coverage analysis features. For you PHP coders out there who aspire to professionalism, you will almost certainly find that rigorous, formal testing procedures will save you time and grief in the not-very-long run. If you've been "test infected" or you're working in an XP (the development method, not the massively defective software), you already know this; otherwise, you're likely to be shocked by how good your code is likely to become.

Automated unit tests (using tools such as JUnit (for Java), NUnit (for Microsoft .NET) and PHPUnit are good for proving a) that your code works the way you expect and b) that it keeps working as you make changes to it. Automating the process (so that it runs all your tests whenever you make changes) ensures that you get continuous feedback that all the tests still work (reducing the likelihood of hidden dependencies breaking). Code coverage testing, as the phrase implies, tracks which lines of your code are exercised. Code that has been tested extensively and found to work as expected can be differentiated from untested code and "dead" code. Dead code is code that can't be executed because no logical path through the program exists that will execute that code. A minimal number of lines of code will be marked as dead code in many languages and test systems due to the textual structure of the language itself (e.g., PHPUnit marks a line of code consisting solely of the brace ending an if-block as dead code). Code identified as dead should be eliminated except for this structural detritus; why maintain code that will never be executed?

Many beginning developers (and not so beginning) tend to assume that their code works if it doesn't contain any obvious syntax errors flagged by the interpreter or compiler. These language systems are (generally) reasonably competent at interpreting what you wrote; they have significant constraints in discerning what you intended beyond what you wrote. Hence, just because the code "compiles cleanly" doesn't mean it is free from defects. That's the job of testing — starting with you, the developer, running unit tests.

If you're doing unit tests without any specialised tools ("rolling your own" tests), or your testing tool does not provide code coverage analysis, it may often be difficult to determine whether specific blocks of your code have or have not been tested successfully. Thus, the assumptions that you made while writing the code are unlikely to be challenged during testing — the same assumptions will guide an ad hoc testing regime as the original coding.

One of the benefits of test-driven development is the ability to cut out unnecessary code and other development artifacts; the development team is very confident that exactly and only what is part of the desired system is actually in that system and can expand/refactor the code to adapt to changing requirements; "see a need, fill a need". You can get a lot more done, more quickly, when you have total confidence that your code will continue to work as features are added or changed, and that anything that breaks will be immediately and obviously detected. You can't do that — really — without code coverage analysis.

For you plinkers out there who aren't doing sufficient (or sufficiently organised) testing yet — your competition is, and if you spend any time at all in this craft, you will bang up against "difficult" bugs that wouldn't have been so difficult with pervasive testing. Those of you who have been writing unit-test cases shouldn't automatically get too comfortable, however... how do you know how much of your code is being tested? If you're testing the same block of code eight different ways and other sections of your code don't get tested at all, can you ship a quality product? Coverage analysis will save you time (by not writing redundant test cases), grief (by prodding you to test areas of code you thought you tested but hadn't), and money (from #1 and #2). If you're trying to run a completely instrumented shop — where everything that can be measured for a lower cost than the failure of that thing, is being measured — then I (should be) preaching to the choir.

To sum up:

• If you don't write it down (in a way that "it" can be found again), it never happened. If you've never tested your code, it's broken until proven otherwise.
• If you can't repeat the test at will, it hasn't been tested.
• If you don't or can't know how much of your code has been tested, your users will.

We can't all be Microsoft and expect our paying customers to find our problems!

XHTML Is (Nearly) Useless

EDITED 2010/11/02: If you saw this post as a single massive block "paragraph," my apologies. The definition of "what is a paragraph?" changed after this was originally posted, and it had scrolled far enough back that I didn't notice the carnage. Let me know if there are still any problems. Thanks.

---

If you've written any Web pages in the last five years (at least), you've at some point bumped into the difference (schism?) between "original" HTML and "new, improved - now based on XML!" XHTML. If you don't write Web "content" (thanks for reading my blog, but why are you here?), or deal professionally with those who do, you may not know the difference, or care that there is a difference. There is, and people should care about it if they care about the Web.

(Briefly, for those who care but don't know; the rest of you can skip this and the next paragraph.) HTML is often known to developers as "tag soup", because very, very many sites don't follow the strict interpretation of the standard, and are "broken" in all sorts of ways. This was initially justified as working around the myriad bugs in grossly defective browsers such as Microsoft Internet Explorer. XHTML was different and better because it was HTML reformulated as XML, which could then be "validated" (checked) by any validating XML parser. HTML-as-XML also (should have) driven the development and use of all sorts of nifty techniques and tools that are only practical when assumptions can safely be made about the structure and format of the document - which would be true in XML/XHTML but not necessarily in "classic" HTML.

The problem, of course, is Microsoft's Internet Explorer browser, affectionately known to Web professionals as "Internet Exploder". Among the many "quirks" (defects) that has unknowingly afflicted usees of that browser, all versions up to and including the current Version 7 fail to understand XHTML as XHTML. The "conversation" that takes place between a browser and a server when the browser requests a Web page is defined by the open standard known as HyperText Transfer Protocol, or HTTP. Part of that conversation involves the server informing the browser what type of data it will be sending. This is done using what is called a "content type" "header".

All together now? Good. When a server wants to send a browser a page of "tag soup" HTML, the correct content type is "text/html". A properly-formatted and -served XHTML page will instead use "application/xhtml+xml". This will inform the browser that, in fact, the page being transferred is a proper XHTML page (per the open standard defining it), so the browser will kick in the assumptions and processing that works for XHTML but not for "tag soup".

Of course, Internet Explorer is now the only major browser that gets this wrong (as indicated by this vintage-2005 blog entry). As far as I am aware, every other major graphical browser in the world - Firefox, Opera, Konqueror, Galeon and the rest - all support The Right Thing. Unfortunately, IE is still the 300-pound gorilla in the china shop; the majority of Windows usees still browse the Web using IE, and though the trend is improving steadily, that will likely continue to be true for the next couple of years (say, 2009-2010 barring unforeseen circumstances).

What kinds of things would a properly XHTML 1.0-compliant browser let us do with our site? One trivial example: let's say you're writing a political-commentary site that is geared towards an upcoming election, and you want to consistently name your candidate as "The Honorable Senator Francis X. Snort (email senator@senatorsnort.org)". When your guy goes down to defeat (one too many campaign-finance scandals, mayhap) you want to change the blurb to "The Honorable former Senator F. X. Snort (email snort@somefreemail.com)". Trivial to do with whatever CMS or scripting system you're using, right? But by using an XML entity, you can simply say "&snort;" in your document, and an entity declaration in your document's header will tell the parser what you really mean. Change the declaration, and every instance of that entity expands to your new meaning. People who use other XML-based markup systems, such as DocBook, have been using this technique for years. Using XML entities in pages shown in correct (non-IE) browsers will do exactly what you tell it to. In IE, or, to be fair, several text-based browsers, the entity name will be displayed exactly as it is in the document - in our case, as &snort;. This is unlikely to have the desired effects on the folks "back home" for the Senator.

Web developers have, as I mentioned, several well-known workarounds for this type of thing, using their authoring tools rather than the document itself. It is, however, a reasonably easy example for people to understand. Given the increasing popularity of systems such as PHP Smarty that let you use large chunks of "raw" (X)HTML along with the scripting goodies, it would come in handy too.

So how does all of this make XHTML "nearly useless?" Because most developers developing pages for the general public (as opposed to corporate intranets), knowing that Microsoft IE doesn't support the correct content type, will either "not bother" developing "correct" XHTML or at best will serve it to all comers as "tag soup" HTML.

This also has the "benefit" of completely stifling further innovation (as far as the end user is concerned) based on XHTML. All of the comments I've made so far are only germane to the initial version of XHTML, designated 1.0. The newer versions, XHTML 1.1 and XHTML 2.0, provide new features and support new technologies that greatly expand the usefulness of the Web - or would, if Microsoft weren't, as usual, dragging the Web down for competitive lock-in purposes. By doing everything in their considerable power to ensure that IE browsers and sites aren't fully, completely interoperable with other browsers, they discourage Windows usees from using "rival" browsers to browse sites labeled "Best viewed with Microsoft Internet Explorer". There's nothing preventing Web designers from writing standards-compliant sites that also work well with IE; in a well-designed site, it's not particularly onerous to support both standards and Microsoft. If you're using Microsoft tools, of course, it will take quite a bit more work and knowledge to create valid sites. It can be done - several sites and mailing lists describe the techniques and mind-set required - but Microsoft do not go out of their way to make it easy to do so.

Of course, this also applies only to the public Internet. If you're fortunate enough to be developing "real Web apps" for your company's intranet, and your company understands the value of open standards, then you're not going to be subjugating yourself to IE and none of this really applies to you. Go enjoy all the things that new tech lets you implement that can really stomp on your non-standards-using competition!

For the rest of us, until the Web gets out of this proprietary funk it's in now, and IE either falls into a long-deserved oblivion (improving Windows security dramatically, but that's another post) or actually complying with the same standards every other serious browser in the world does, then we're going to have problems. One of the more annoying and frustrating ones, as we've discussed, is that XHTML is (nearly) useless." So much for innovation.

It's the End of the Net as we know it, and we feel fine....

John C. Dvorak has an interesting post on his pcmag.com column blog, entitled "Will the Internet Collapse?" He doesn't think it will, obviously, and he's got some pretty impressive trends to back up his contention. Example: 140,000 terabytes of backbone traffic in 2002 — at a "conservative" 60% annual growth through 2007, that's roughly 25 KB for each of the six billion or so people on the planet. Most of whom (still) wouldn't know what a byte was if it bit them; they've got more pressing concerns, like safe food, clean water, housing... But I digress.

I don't think the Net per se will "collapse", either. What's going to happen — what's already happening — is both more subtle and dangerous. The "Internet craze" that gave rise to Bubbles 1.0 (1990s) and 2.0 (now) and has driven the Net from a quirky research project into a cultural touchstone, has done two things that, by comparison, would make an every-Friday-from-4-to-10-PM crash seem benign in comparison (and "4-t0-10-PM" where? On the Internet, it's always "now".)

The first problem is the Baby's Spoon in the Waterfall. There's so much information (wrapped up in even more "content", which isn't the same thing) that no person, government, entity or corporation can ever comprehend. People who spend large amounts of time surfing the Web and using various tools to pull information off the Net in other ways, soon exhibit a behavior akin to being "punch drunk". Late in te 12th round, The Champ has connected so many times with Joe Palooka's jaw, and we in the crowd can see Joe staggering around, unsure of even from which direction the merciless pummeling is coming, let alone able to control the situation. The Champ, in this analogy, is the onslaught of data/information/"content" from the Net, primarily email and the Web; Joe is standing in for the typical, non-technical ("you mean Yahoo and the Web aren't synonyms?") user. As the user's eyes glaze over and the cognitive mind enters vapor-lock, he is essentially unable (and psychologically unwilling) to refine his usage patterns or seek out new experiences that he wouldn't find in "offline" life (what in an earlier age was called the "You Are There" effect). So, for instance, the stereotypical North American user goes back to the "safe, familiar" online equivalents of his offline television shows — the "news" sites owned by the same multinational corporations that own American media, and YouTube, which can be viewed as a worldwide online version of "America's 'Funniest' Home Videos": another vehicle for peddling the same tired corporate products in the commercials.

The other problem, of course, is that organizing all this "stuff" has become more difficult, and the rate that it becomes more difficult is at least as rapid as the rate of growth itself. While the Net, and the Web in particular, have enabled new ways to express individual personalities (e.g., MySpace) and alowed ordinary citizens of many countries to amass much ore detailed information about what their government is doing, for them or to them (e.g., YouGov and Thomas), if you don't know about YouGov or Thomas (or any similar site set up by your own country's government), then the old Bruce Springsteen song, "57 Channels and Nothing's On" seems quaint and manageable in comparison. People know there's all sorts of stuff out there — they can Google for it, "it must be real" — but, unable to come to grips with how things are organized (they aren't, on purpose) or how to use the available information to achieve a personally important goal, they fall back on the sites that organize and package and sanitize the content, accepting loss of control as the price of freedom from thinking too much. (E.g., AOL — a subsidiary of Time-Warner, and Fox "News".com, a wholly-owned subsidiary of AIPAC.) As people sink safely back into their easy chairs, content to absorb the anti-intellectual pablum that bombards them, they lose touch with the idea, let alone the possible reality, of an energized populace using the new, revolutionary technology at its disposal to improve their own lot in life and that of the world at large. Instead of a medium which challenges the status quo, the Net has devolved into a tool which reinforces it.

A collapse of the Internet? You're right, John; it will never happen. But a collapse of the promise and meaning of the Internet? It's already here, folks; we're just standing around watching streaming video of the rubble bouncing.