IE9 and Standards? Nothing to see here, folks; move along...

This is a re-post of a comment I left to the post "HTML5 Support in Internet Explorer 9 on Louis Lazaris' Impressive Webs blog (which I highly recommend for people working in or interested in Web development). I've slightly edited a few places for clarity. The comments that had been left by people were almost universally complaining about Microsoft in general and IE's continuing history of imperfect-at-best compliance with either-you-do-or-you-might-as-well-not-try standards. Having worked in and with Microsoft on several occasions, and being active in a number of open-source projects besides, I have a slightly different view, as I state below. Thanks for reading.

---

Actually, I'm amazed that it's progressed as much as it has.

Let me be clear; I'm not in any way praising IE; the industry as a whole needs to bury it — decisively, irretrievably and imminently. We as development professionals need to have the professional self-respect to tell current and potential clients, "We develop to current Web standards that are supported across various browsers and platforms. To the degree that any particular release of Microsoft Internet Exploder supports the markup, styling and behaviour of any particular site, that's well and good. However, without monumental additional time and budget resources, no attempt will be made to provide IE support when those resources could instead be used to improve the experience for users of modern browsers."

I firmly believe that IE hasn't progressed as far as, say, Chromium, can be laid firmly at the feet of Microsoft's existing code base. Microsoft's developers and managers are fully experienced with the reality that complete green-field rewrites of existing projects almost never succeed. They've got a code base where the list of major Windows modules and subsystems that do not have dependency relationships with IE could be read aloud comfortably in a single breath. That was done initially by choice; now, it doesn't matter how well-meaning the intentions or competent the team are, they have to live with the codebase they have. It's all legacy code. The developers at Microsoft are (with rare exception) not morons, but living in the sewer while they're trying to make something that can stand next to the shiny new browsers next door has to be a psychologically toxic exercise. Their baby is blowing up the Web left and right; they know it, and they know they can't do a damned thing about it without staging a coup d'état, replacing dozens of levels of management and senior executives and fundamentally changing the culture of the organisation. Don't hold your breath waiting for that to happen.

That isn't sour grapes or a diss against the IE developers; it's simple reality. Microsoft do some amazing things — just not so much for (or on) Windows. Unfortunately for all of us, Windows and Office for Windows are the herds of cash cows for Microsoft, and anything that could be seen as even potentially disrupting that would get shot down faster than you can say S-E-C; the investors would never stand for it. And, with the system and the rules the way they are, they'd be perfectly right. Innovation isn't as important to the bean-counters (or the regulators) as "maintaining and enhancing shareholder value," and MSFT have had enough problems with that lately. (Just compare their share values to, say, AAPL over the last ten years.) Doing anything "risky" is simply politically (and legally, really) impossible.

So, no matter how many times the IE team jump up and down and say "look at all our neat new features," without mentioning the standard features left unimplemented because they pulled off numerous miracles just making what they have work, the response has to be "nothing to see here, folks; move along."

And move along we shall.

Those were the days, my friend...but they're OVER.

No, I'm not talking about Facebook and the idea that people should have some control over how their own information makes money for people they never imagined. That's another post. This is a shout out to the software folk out there, and the wannabe software folk, who wonder why nobody outside their own self-selecting circles seems to get excited about software anymore.

Back in the early days of personal computers, from 1977 to roughly 2000, hardware capabilities imposed a real upper limit on software performance and capabilities. More and more people bought more and more new computers so they could do more and more stuff that simply wasn't practical on the earlier models. Likewise, whenever any software package managed something revolutionary, or did something in a new way that goosed performance beyond what any competitor could deliver, that got attention...until the next generation of hardware meant that even middle-of-the-road software was "better" (by whatever measure) than the best of what could happen before.

After about 2000, the speed curve that the CPUs had been climbing flattened out quite a lot. CPU vendors like Intel and AMD changed the competition to more efficient designs and multi-core processors. Operating systems – Linux, Mac OS X and even Microsoft Windows – found ways to at least begin to take advantage of the new architectures by farming tasks out to different cores. Development tools went through changes as well; new or "rediscovered" languages like Python and Forth that could handle multiprocessing (usually by multithreading) or parallel processing became popular, and "legacy" languages like C++ and even COBOL now have multithreading libraries and/or extensions.

With a new (or at least revamped) set of tools in the toolbox, we software developers were ready to go forth and deliver great new multi-everything applications. Except, as we found out, there was one stubborn problem.

Most people (and firms) involved in software application development don't have the foggiest idea how to do multithreading efficiently, and even fewer can really wrap their heads around parallel processing. So, to draw an analogy, we have these nice, shiny new Porsche 997 GT3s being driven all over, but very few people know how to get the vehicle out of first gear.

I started thinking about this earlier this evening, as I read an article on Lloyd Chambers' Mac Performance Guide for Digital Photographers & Performance Addicts (pointed to in turn by Jason O'Grady and David Morgenstern's Speedtest article on ZDNet's Apple blog. One bit in Chambers' article particularly caught my eye:

With Photoshop CS4/CS5, there is also increased overhead with 8 cores due to CS5 implementation weakness; it’s just not very smart about knowing how many threads are useful, so it wastes time and memory allocating too many threads for tasks that won’t even benefit from them.

In case you've been stuck in Microsoft Office for Windows for the last 15 years, I'll translate for you: The crown-jewels application for Adobe, one of the oldest and largest non-mainframe software companies in history, has an architecture on its largest-selling target hardware platform that is either too primitive or defective to make efficient use of that platform. This is on a newly released version of a very mature product, that probably sells a similar proportion of Macs to the proportion of Windows PCs justified by Microsoft Office (which is even better on the Mac, by the way). Adobe are also infamous among Mac users and developers for dragging their feet horribly in supporting current technologies; CS5 is the first version of Adobe Creative Suite that even begins to support OS X natively – an OS which has been around since 2001.

I've known some of the guys developing for Adobe, and they're not stupid...even if they believe that their management could give codinghorror.com enough material to take it "all the way to CS50." But I don't recall even Dilbert's Pointy Haired Boss never actually told his team to code stupid, even if he did resort to bribery (PHB announces he'll pay $10 for every bug fix. Wally says, "Hooray! I'm gonna code me a minivan!"...i.e., several thousand [fixed] bugs).

No, Adobe folk aren't stupid, per se; they're just held back by the same forces that hold back too many in our craft...chiefly inertia and consistently being pushed to cut corners. Even (especially?) if you fire the existing team and ship the work off halfway around the world to a bunch of cheaper guys, that's a non-solution to only part of the problem. Consider, as a counterexample, medicine; in particular, open-heart surgery. Let's say that Dr. X develops a refinement to standard technique that dramatically improves success rates and survival lifetimes for heart patients he operates on. He, or a colleague under his direct instruction, will write up a paper or six that get published in professional journals – that nearly everybody connected with his profession reads. More research gets done that confirms the efficacy of his technique, and within a fairly short period of time, that knowledge becomes widespread among heart surgeons. Not (only) because the new kids coming up learned it in school, but because the experienced professionals got trained up as part of their continuing education, required to keep their certification and so on.

What does software development have in common with that – or in fact, with any profession? To qualify as a profession, on the level of the law, accounting, medicine, the military, or so on, a profession is generally seen as

• Individuals who maintain high standards of ethics and practice;

• who collectively act as gatekeepers by disciplining or disqualifying individuals who fail to maintain those standards;

• who have been found through impartial examination to hold sufficient mastery of a defined body of knowledge common to all qualified practitioners, including recent advances;

• that mastery being retained and renewed by regular continuing education of at least a minimum duration and scope throughout each segment of their careers;

• the cost of that continuing education being knowingly directly or indirectly subsidised by the professional's clients and/or employer;

• such that these professionals are recognised by the public at large as being exclusively capable of performing their duties with an assuredly high level of competence, diligence and care;

• and, in exchange for that exclusivity, give solemn, binding assurances that their professional activities will not in any way contravene the public good.

Whether every doctor (or accountant, or military officer, or lawyer) actually functions at that level at all times is less of a point than that that is what is expected and required; what defines them as being "a breed apart" from the casual hobbyist. For instance, in the US and most First World countries, an electrician or gas-fitter is required to be properly educated and licensed. This didn't happen just because some "activists" decided it was a good idea; it was a direct response to events like the New London School explosion. By contrast, in many other countries (such as Singapore, where I live now), there is no such requirement; it seems that anybody can go buy himself the equipment, buy a business license (so that the government is reassured they'll get their taxes), and he can start plastering stickers all over town with his name, cell phone number and "Electrical Repair" or whatever, and wait for customers to call.

Bringing this back to software, my point is this: there is currently no method to ensure that new knowledge is reliably disseminated among practitioners of what is now the craft of software development. And while true professionals have the legal right to refuse to do something in an unsafe or unethical manner (a civil engineer, for example, cannot sign off on a design that he has reason to believe would endanger public users/bystanders, and without properly-executed declarations by such an engineer, the project may not legally be built). Software developers have no such right. Even as software has a greater role in people's life from one day to the next, the process by which that software is designed, developed and maintained is just as ad hoc as the guy walking around Tampines stuffing advertisements for his brother's aircon-repair service under every door. (We get an amazing amount of such litter here.)

Adobe may eventually get Creative Suite fixed. Perhaps CS10 or even CS7 will deliver twice the performance on an 8-core processor than on a 4-core one. But they won't do it unless they can cost-justify the improvement, and the costs are going to be much higher than they might be in a world where efficient, modern software-development techniques were a professionally-enforced norm.

That world is not the one we live in, at least not today. My recurring fear is that it will take at least one New London fire with loss of life, or a software-induced BP Deep Horizon-class environmental catastrophe, for that to start to happen. And, as always, the first (several) iterations won't get it right – they'll be driven by politics and public opinion, two sources of unshakable opinion which have been proven time and again to be hostile to professional realities.

We need a better way. Now, if not sooner.

Companies Lose their Minds, Then their Partners, Then their Customers, Then...

Following is a comment which I posted to Jason Kincaid's article on TechCrunch, "Why Apple's New Ban Against Sexy Apps is Scary". I don't know why Apple seem to be deliberately shooting themselves in so many ways on the iPhone recently; I am sure that they are leaving golden opportunities for Palm, Android and anybody else who isn't Apple or Microsoft.

Even if you're not developing for the iPhone or even for the Mac, this whole drift should concern you — because its most likely effect is going to be that you have fewer choices in what should be a rapidly-expanding marketplace.

Thanks for reading.

---

Exactly; they're pulling a Singapore here. They're saying "we're better than anybody else" because they've got this App Store with hundreds of thousands of titles, and hundreds of useful apps. Then they turn around and say "we're the only game in town; we're not going to let you sell your apps to customers any way except through us — and oh, yeah, we can be completely arbitrary and capricious before, during and after the fact."

Let me be clear: up until very recently, I've been an unalloyed Apple fan; the only sensible response to 25+ years of Windows development and user support and 10 years hitting similar but different walls in Linux. I'm typing this on one of the two Macs sitting on my desk. I've got logs and statistics that prove I'm far more productive on my worst Mac days than I ever was on my best Windows days. And I've had several Switcher clients over the past few years who say the same thing.

I can write and sell any app I want on the Mac; Apple even give me all the (quite good) tools I need right in the box. I can use any app I want to on my Mac; the average quality level is so far above Windows and Linux apps it's not even funny. In neither of those do I need the permission of Apple or anyone else outside the parties to the transaction involved. Apple do have good support for publicising Mac apps; browse http://www.apple.com/downloads/ to see a (far earlier) way they've done it right. But developers don't have to use their advertising platform.

With the iPhone, and soon the iPad, they're doing things in a very untraditionally-Apple way: they're going far out of their way to alienate and harm developers. You know, those people who create the things that make people want to use the iPhone in the first place. And a lot of us are either leaving the platform or never getting into iPhone development in the first place.

And that can't be healthy for the long-term success of the Apple mobile platform (iPhone/iPad/iWhatComesNext). As a user, as a developer, as a shareholder, that disturbs me, as I believe it should disturb anyone who cares about this industry.

Virtual Real Work

(As posted in a comment on a very detailed Ars Technica review:

After giving PD5 and Fusion 3 full evaluations, I'm going back to Fusion for another cycle.

A lot of what I do is testing software in odd developer builds of various BSDs, Linuxes and OpenSolaris; Fusion gives me the least trouble when venturing away from the Microsoft megalith. I have XP and Win7 VMs too; I just don't fire up either more than once or twice a month. I typically have 2-3 other VMs up and on my network at any given time.

I also did take a good look at OpenBox; it's the best of the bunch for running OpenSolaris (wonder why?), but for anything else, particularly the oddball Linuxes, it gave me as much trouble as PD 5. Where Parallels would crash like a drunken teenager in a Maserati, OpenBox just would not install several important systems. So much for that.

VMWare isn't as slick in some areas as PD5 (LOVE that one-click 'exclude from TM' feature, and the VM list is nice), but it DOES have one feature that's absolutely non-negotiable: It tends to work better/more often. That's worth the extra US$10 or so for the upgrade. (with an additional new license for my new MBP).

I've put about 20 hours over the past 2 months into formal evaluation of the hypervisors mentioned (along with a couple of hundred hours of just using them to run VMs with which I was working, and not trying to think "is this better than...?"). The pain level with Parallels has been fairly consistent; VMWare only surprised me a couple of times. Parallels also gives you only half the review time that VMWare does before you have to pony up. That tells me that they aren't as confident in their product (and rightfully so); there are problems that merely suggest themselves in 15 calendar days of varied use that you could really get a handle on in 30. Not having confidence borne of experience in whether I can solve subtle issues as they arise wasn't the only reason for rejecting Parallels Desktop 5, but it certainly didn't help.

The impression I get from using both products, from reading what has been written and from chatting with other users, is very consistent. VMWare portray themselves as a company that tries, and often succeeds, to do things well - starting with a truly usable, stable, versatile virtualisation platform. Parallels, on the other hand... by bringing out releases well before they're really ready, and with a shorter evaluation time, give the impression that they're desperate for your money. Give it to the company that actually earn it.

Improvement as opposed to Change

A link on the Agile CMMi blog had some very interesting things to say about a gentleman named Brian Lyons, the CEO (as well as CTO and founder) of Number Six, a Vienna, Virginia (Washington, DC area) software technology services/business consulting firm. Number Six quite obviously 'get' the concepts behind Agile CMMi, Hillel Glazer of the blog wrote. Interested, I flipped over to their site to hopefully learn a bit more, maybe drop off a CV.

Mr. Glazer wrote the blog entry in question on 25 July. The first thing you notice when viewing the Number Six site is the home-page obituary of and tribute to Brian Lyons, who died on (Monday) 3 September 2007 in a motorcycle accident. There are various links to family and tribute sites, a press release, and a request that "in lieu of flowers, the family has requested that donations be made to" a scholarship fund at the University of Maryland (excellent, particularly for those too far or too late to attend the funeral). I wish the family and company all the best in their times of grief and trouble.

The point I originally started writing this entry to make, however, was inspired by one of the bullets on Number Six' careers page, under the heading "Why Six?": "We are committed to consistent improvement, not continual drastic change."

Think about that distinction for a moment. Many of us, particularly those in the software-geek persuasion, start our careers trying to remake everything; not necessarily because it's broken, but so that we can do it (whatever it is), make it ours, introduce new techniques or technologies, and hopefully (but improbably) provide a better solution to the problem at hand than what was there before.

The forces arrayed against that impulse are formidable. Playing the role of (and often in actual fact) those who are by nature suspicious of any radical approach to a problem, too often dismissing out of hand any alleged improvement or innovation without "giving it a fair chance", in the eyes of the wet-behind-the-ears Young Turk. Occasionally, of course, improvements and innovations too beneficial to ignore do come out of this process.

As the developer matures (which may take days, decades or eternities), the Young Turk recognizes that not all of the hindrances were traditionalist per se; rather, they were operating from a different (usually business-oriented) set of priorities. Learning to understand and appreciate those priorities is one of the fundamental aspects of any successful transformation from ivory-tower Geek to journeyman software craftsman (or -woman), able to make a contribution to customers and to the craft of software development.

What becomes obvious, to all thoughtful participants and stakeholders in the process of developing and maintaining any software system is that a natural tension exists between three apparently conflicting ideas:

• Don't change what works well enough when there are more pressing needs to attend to;
• New capabilities, chosen and implemented properly, can have strongly beneficial effects (efficiency, productivity, wider customer scope, 'better' according to various aspects of quality);
• Any change to an existing system involves direct and indirect costs which must be carefully evaluated and compared against the expected improvements.

Balancing and managing the conflict between those concepts throughout the lifetime of a piece of software is an art that has yet to be thoroughly and reliably mastered with a level of reliability and cost widely acceptable to business customers. Several philosophies and techniques (often marketed as technologies) have been developed and marketed as (attempted) solutions to that problem. Three that I have found extremely useful, in complementary ways, over the last few decades are agile development, refactoring and the CMMI. (Those already familiar with the concepts may skip down to the paragraph which begins "As argued by practitioners such as Hillel...".)

The CMMI (previously more widely and less precisely known as the CMM, for Capability Maturity Model) has been around for quite some time as a formal means of evaluating the maturity and coherence of an organization's development efforts (among other activities). It requires its users (organizations) to acquire and document ever-increasing detail of precisely how they go about the process of development and how they are required to be able to prove (in an internal or external audit) that those processes are being followed and any discrepancies noted and accounted for. This suffers as a standalone policy framework for software development on two grounds:

• the "what" and "why" of development are completely ignored. CMMI fundamentally doesn't care if you're building a pile of junk that has serious practical problems, as long as you do it using the process you've defined for yourself.
• The CMMI, along with US DOD Standard 2167A, have been blamed for the logging of more trees (to make paper) than virtually any other industry business practice. This is largely because both are seen (in CMMI's case, somewhat unfairly) as pushing "hidden mandates" for the rigid , never-far-from-obsolete "waterfall" model of development.

Agile development, on the other hand, is a survival response to both the (largely management-driven) mantra that "Real Artists Ship", and on the other hand, to seemingly interminable periods between "milestones" (particularly in the waterfall model) where nothing of intrinsic value is visible to an outside observer (e.g., management or customers). Its main criticism from opponents is that it produces too little documentation (the product is the documenatation, to a large degree).

An aspect of, or adjunct to, agile development is refactoring, pioneered and popularized in the Martin Fowler-written book. Refactoring is all about how you can make (sometimes radical) changes to the internals of a software system, provided you keep the external (customer-facing, revenue-generating) interfaces constant. Wrapping one's mind around this as a formalized process, as opposed to the "don't-break-what-works" mentality common to experienced developers (and managers), is a threshold experience in a software craftsman's progress.

As argued by practitioners such as Hillel, the combination of the process-centric CMMI and the solution-centric, visible-progress Agile set of processes (including, particularly, refactoring) gives the "best of both worlds" for development — a focus on artifacts and products created through a survivable process, matched with a demonstrated and documented knowledge of exactly what is meant by "that process" under current circumstances. This also implies demonstrating understanding of the specifics of "current circumstances" that drive the decision-making and artifact-development processes — which brings us back full circle to understanding why Agile is a good fit to begin with.

Unlike the Rational Unified Process (RUP), which grew out of a waterfall-driven management-oriented mentality (and the opportunity to sell software tools and consulting services to any shop using the heavily-marketed process), Agile and CMMI development can be successfully started after passing around a few books among the development and management staff. Tools and training are, chosen thoughtfully, extremely helpful, and the appraisal process within CMMI (formally SCAMPI, the well-known "Level 1" to "Level 5" assessment of process maturity) does eventually require an outside assessor, or registrar, to formally certify the organization. But to take a typical small development group from the customary initial chaos to a finely-tuned, market-leading, customer-satisfying machine, it has been my experience and observation that it is far easier (and more cost-effective) to implement CMMI in an Agile fashion than to go down the (vendor-specific) RUP route. Imposing either on a large organization would require an unlikely combination of managerial brilliance, sadism and masochism; revolutions boiling up from below and winning eventual management sanction have proven much more likely to be successful in that type of environment.

So what's a Young Turk (or a more seasoned craftsman) to make of all this blather? The simple point: Consistent improvement is practical, and without life- or career-threatening implications, achievable on a repeatable basis. Continual drastic change, in contrast and almost by definition, will lead the development group to many sleepless nights trying to get that last show-stopper bug fixed, management to wonder why those bozos in development can't be trusted to ship anything on time, and customers to wonder what they're really getting for their money beyond the hype and bling of the marketing materials. Which team would you rather be on?