Redundancy is Repetitive, Inefficient and Counterproductive

But you already know that.

I've long been a supporter of free and open source software, even contributing to a few projects. However, my enthusiasm has cooled a bit in the last several years, as the time I've been able to devote to such projects has both dwindled and been divided among more projects.

I'm continually flabbergasted by the number of open-source projects that cover the same ground, ad infinitum, often with little to no apparent differentiation between them. By continually "reinventing the same wheel," these small projects usually manage to put out a small number of alpha- or beta-quality releases, and then cease to make visible progress. This could be because the original developer(s) became frustrated or distracted; because the project succumbed to internal politics (a leading cause of FLOSS project death); or merely because the developer(s) realised that they were deep in the rut of a path that has been travelled many, many times before.

This flash of obviousness came about as I've been researching and evaluating PHP frameworks for Web development. I've been developing and maintaining sites using a couple of these, notably Kohana 2.3, but for various reasons have been reluctant to make the switch to the now-current 3.0 release. I decided to do a relatively formal, structured evaluation of several of the frameworks out there, rather than just pounce on the first I came across that looked "reasonable." (Silly me, but oh well.) Of course, as my list of frameworks to look at grew from six, to twenty, to considerably more, I became uncomfortable; torn between the reality that I'm doing this all myself (and devoting significantly less than 100% time to the endeavour), but at the same time wanting to remain open to pleasant surprises. After all, "pleasant surprises" were how I stumbled upon Kohana in the first place, and though I am strongly considering using another tool in its place (again, for non-technical reasons), contributing to a megalithic monoculture is not my idea of "adding value."

Today I did something that I should have done a week or two ago: I went out to SourceForge and got a list of all the PHP frameworks registered on the site.

Oh, my effing $DEITY! "Searching gives 1316 results."

Obviously, not all of those are appropriate (there are numerous reinventions of the CMS wheel, for instance); many can be filtered simply because they haven't published updates recently enough to meet my criteria, and so on, but that's all rather beside my current point.

Open-source projects are (in)famously both insular and prolific, leading to the kind of nonsense I ran into on SourceForge. Commercial products have a built-in inhibitor of this particular pathology: if a market gets divided among so many competitors that nobody can cover their costs, then in very short order the existing products will either consolidate into fewer, more sustainable alternatives or die altogether. Open-source projects, particularly those started by people inexperienced in such projects, fail to realise that they too are competing for resources just as much as their commercial brethren. In the FLOSS case, the finite, competed-for resource is time, both of (potential) contributors and of passive users (what the commercial world calls "customers").

Some open-source projects recognise this. In the Linux and Unix/BSD desktop worlds, the "market" for desktop managers (that provide the windows, icons, menus, and so on that are the UI to the user) has been dominated by two projects, KDE and GNOME. Though there are other choices with solid offerings and active communities, a developer can choose to develop for either KDE or GNOME with the confidence that virtually anyone in his target audience will be able to run his software on their system of choice. Consequently, most projects that have to make a choice, choose KDE or GNOME.

The Web is a different beast entirely, and while standards are for the most part very well-defined, the "barrier to entry" for those seeking to do things (allegedly) differently is very low. Some in the PHP Web development community advise new developers to tackle creating their own framework, as a learning tool. I suspect that that's how many of the SourceForge listings originated. I also happen to think that the advice itself is naïve, akin to a new auto mechanic being advised to build his own engine to gain a better understanding of those from Toyota or Ford. What I would recommend instead is that a new developer learn one or two of the more widely-used frameworks, browse around for another two or three that seem interesting, and start contributing to the project(s) of her choice. Virtually all such projects recognise their need for more help, as well as the fact that "we were all newbies once."

Consolidation in the PHP-framework field, while maintaining the low barrier to entry, should be a Good Thing™: to a greater degree than commercial products, FLOSS projects should succeed on their merits, technical and otherwise. As more developers join a well-run project community, the project's software should continue to both improve and find wider user/customer use. The ability to "do your own thing" should be kept, if only as a check against the use of marketing hype over technical merit.

But I believe that relatively small, informal "A- and B-List" groups of selections would be good for projects and developers (and developers' clients); a lot of what I've come across in the last few days would be on my "P-List," if not "ZZZZ."

Oh yeah, by the way: this sort of maturation and shared effort would do wonders for our Craft's progress towards true professional status. Any of us who've had our schedules and budgets tinkered with by people who admit they know nothing of what we do should understand the desirability of that goal. Given the degree to which software permeates public life and policy, that "desirability" has become a dire necessity; it's just more expedient for people to pretend that isn't really the case. Stuxnet, anyone?

In Praise of Robust Tools and of The Future™

One of the best points about doing Web development in PHP is that it's so widely used; several respectable estimates by organizations that get paid to find these things out say that some varying number north of 50% of all sites on the World Wide Web use PHP as their implementation language. This includes numerous content management systems, or CMS, such as WordPress, Joomla! and Drupal.

One of the far-less-good points about doing Web development in PHP is that it's so widely used; the (evolving) "standard" set of tools, in true open-source fashion, are assembled and maintained by an ad-hoc band of individual and small-corporate luminaries, on infrastructure that worked just fine back when any given server was hit a couple of thousand times a year; by continuing to rely on (what to the outside Net appear to be) individual servers without terribly huge pipes connecting them to the larger Internet, the infrastructure completely fails when that's scaled up by a factor of a thousand or ten.

Web developers using PHP are highly dependent on an extension architecture/platform called PEAR, the "PHP Extension and Application Repository." Singular. Well, for any given extension or application, it's singular. So, as the user base scales upwards, popular tools that go through a vigorous release cycle, the servers they're hosted on (and network choke points between large subsets of users and those servers) start having reliability problems; transfers slow to a crawl, or fizzle out entirely. (One such transfer this evening proceeded at a sedate 600 bytes per second. Not kilobytes: bytes. That's 1980s dial-up speed.)

Developers in other languages have similar tools; Rubyists have gems available from sites like rubygems.org; Pythonistas have eggs. But somehow I hear a lot less grumbling, and do less of my own, when the subjects of gems or eggs come up; they seem to Just Work™ – which indicates that the network infrastructure is a lot more robust; either larger "pipes" and/or more, distributed servers such as with a CDN.

Trying to access servers like this from Second World cities like Singapore that apparently devote more resources to content filtering/monitoring than easing congestion doesn't help either. Hey, SingNet and M"D"A, how come it takes ten hops to get off an island that's barely that many miles across? If we want to plug Singapore into the "new global economy," having grotesquely under-resourced connections is not helpful.

But back to the main subject: If PHP is going to continue its phenomenally successful growth, then the infrastructure is going to have to decentralise, aggressively. Having essential tools like PEAR and the PHPUnit repository available only from a single point ensures that single points of failure will continue to seriously compromise the PHP ecosystem. No access to servers, no access to tools. No access to tools, then development of PHP artifacts with reasonable efficiency and economy is severely degraded.

In other words, the status quo is a limiting factor on future growth and success. However, as many will be quick to point out, building infrastructure costs money – for hardware, for connectivity, for the paid labour of skilled craftspeople and professionals to create, install, operate and maintain this enhanced infrastructure. No explicit means of funding such an endeavour presently exists, at least not to my knowledge.

So where do we go from here, PHP community?>

Time v. Money v. Risk v. Frustration

I just spent three hours beating my head against The Joys of Wi-Fi Networking. No doubt because it saved $50 or so, the HDB public-housing flat I live in here in Singapore (which are "affectionately" known as 'chicken coops' for their quality and structural integrity) didn't put telephone jacks in every room. Nor did they run conduit between rooms so that suchlike could be added later. The upshot of this is that my DSL modem, in the living/dining room, is 15 meters and two concrete-slab walls away from my study/office, in a bedroom (with no phone jack and two power outlets).

Up to now, this hasn't been an insurmountable problem, because the DSL modem/802.11g router the government-linked telco sells you when you open a DSL subscription could punch a (barely) usable signal through those two concrete walls; desktop and laptop computers strewn around the study were on the LAN, with quite serviceable WPA2 encryption to keep private bits private. The router (in the living room) claimed to have a 14 Mbps connection to The Net; the aggregate total bandwidth available in the study ranged from 3 to 5 Mbps; not fantastic but usable. The world was in a survivable state of chaotic flux.

Then, yesterday morning, FedEx delivered a bright, shiny (actually used, in a Target bag rather than original box, no longer in the Polycom US catalog but still serviceable) Polycom SoundPoint 501 SIP phone, courtesy of my newest client (thanks, Nathan, Matt and Margaret - I'm not kvetching, honest!). An hour of work with jackhammer and flamethrower cleared sufficient space on the desk for the new trophy. Now to plug it in and fire it up.

Plug it in? To what? My ear?

Just to make sure I had my head straight, I looked at what it would take to run line from point P to R. Two solid concrete/rebar walls, routing around various doors/windows, across the span of the living room.... no, that wasn't going to happen, certainly not by Thursday. Put the phone in the living room? Not a chance. Wait a minute - they have these things called wireless Ethernet bridges; I should be able to get one of those, stick it in the study, connect the phone to it and it to the existing wireless LAN, and I'm good to go.

I went out to the local "IT super-mall", Funan. It's the best place in Singapore to buy electronic gear of whatever variety from reasonably reputable dealers. After visiting some 25 different shops, large and small, I was firmly reacquainted with one of the basic facts of Singapore life.

This is a firmly Stalinist economy in the areas that matter. All imports (which is to say, anything of value) are brought in through a small number of generally well-connected exclusive distributors. When one shop says "finished, already!" (the de facto motto of the city), everybody is going to say the same thing. That is, if you're fortunate enough to even run into some salesperson who actually understands what you're looking for. I could find things easier (and often cheaper!) in Vietnam.

Then I happened to pop into one hole-in-the-wall one-man shop, Crystal Systems. The "one man" said "say, I understand what you're trying to do; I did exactly that for a firm here recently; all you need are two routers." The archetypal light-bulb moment. (Remember, I'm a software craftsman, not a hardware/network engineer.) This is where the "time v. money v. ..." of the title comes in. What I should have done was to buy two identical brand-new 802.11n routers. That would have cost me about S$190 (~US$136 or so), and would have given me a perfect excuse to replace my aging 2Wire 2700HGV-2, But this has already been a bit of an expensive month - new hard drive, various software - so after some consideration I decided to try to salvage the existing router (apparent POS though it may be) and just spend the minimum necessary. So I went home with one S$69 TP-LINK TL-WR340G router.

It features the now-customary browser-based setup - just remember to use a hard-wired link rather than wireless (d'oh!), and it's almost as easy as falling off your seat. Bridging mode is pretty obvious; the TP-LINK wants to know the MAC address of the router you're connecting to, along with its encryption setup.

In bridge mode, the TL-WR340G only supports the obsolescent WEP encryption, not the current and generally superior WPA2. The 2Wire - along with every device currently connected to it, obviously - uses WPA2, and does not support a mixed WPA2/WEP environment.

Back to the store I go tomorrow; the only question now is: one new Netgear N router (which can fall back to 802.11g) or two?

We Interrupt This Program...

We interrupt this tutorial to interject an observation about goals, methods and promises. Goals we have for ourselves as people and as professionals; the method we use to pursue those dreams; perhaps most importantly, the promises we make, both to ourselves and to our customers, about what we're doing and why.

I consider this after reading the Website for some consultants who've done some (relatively nice, relatively low-key) link-dropping on LinkedIn. I'm not naming them here, not because I don't want to draw attention to them (their own site is very clean and well-done), but because the point that I'm going to be making here isn't just limited to them - we, as a craft and an industry of Web development, have some Serious Problems™.

The "problem" is nicely summarized by this group's mission statement:

Our mission is to produce the perfect implementation of your dreams.

What could possibly be wrong with that?

As a goal, implied but left unspoken, absolutely nothing; both as practitioners and as clients, we tend to set ever-higher goals for ourselves. Indeed, that's the only way the "state of the art" - any "art" - can advance. But we who practice the art and craft of software (including Web) development (as opposed to the engineering discipline of hardware development) have a history of slashed-beyond-reality schedules and budgets coupled with a tendency for stakeholders not to hear "if all goes well" as a condition to our latest schedule estimate. We have a history, perceived and actual, of promising more than we can deliver. Far more attention is paid by non-technical people to the "failures" and "broken promises" of software than to things done right. For a craft whose work is accruing increasing public-policy and -safety implications, the effect of unrealistic expectations, brought about by poor communication and technical decisions being made by people who aren't just technically ignorant but proud of the fact, is disturbing. What started as a slow-motion train wreck has now achieved hypersonic speeds, and represents a clear and present danger to the organisational health and safety of all stakeholders.

I don't mean to say that projects always fail, but an alarming number of them do. If, say, dams or aircraft were built with the same overall lack of care and measurable engineering precision that is the norm in commercial software development, we'd have a lot more catastrophic floods, and a lot few survivors fleeing the deluge by air. When I entered this craft thirty years ago (last May), I was soon led to believe that we were thirty to fifty years away from seeing a true profession of "software engineering". As a time frame beginning now, in 2009, I now think that is almost laughably optimistic.

Why have things gotten worse when we as a tool-building and -using society need them to get better? Some people blame "The Microsoft Effect" - shipping software of usually-dubious quality to consumers (as opposed to 'customers') who have bought into the (false) idea that they have no realistic choice.

It's more pervasive than that; commercial software development merely reflects the fashion of the business "community" that supports it, which has bought into one of the mantras of Guy Kawasaki's "The Art of Innovation", namely "don't worry, be crappy." Not that Kawasaki is giving bad advice, but his precondition is being ignored just as those of other software people have been: the key sentence in his "don't worry, be crappy" paragraph is "An innovator doesn't worry about shipping an innovative product with elements of crappiness if it's truly innovative" (emphasis mine). In other words, if you really are going to change the world, nobody will notice if your Deus ex Machina 1.0 has clay feet as long as you follow up quickly with a 1.1 that doesn't...and follow that with a 2.0 that changes the game again. But that space between 1.0 and 1.1 has to be fast, Kawasaki argues (in the next paragraph, titled "Churn, Baby, Churn"), and the version after that has to come along before people (like possible competitors) start saying things like "well, he just brought out 1.1 to fix the clay feet in 1.0." If the customers see that you're bringing out new versions as fast as they can adapt to the previous ones, but that each new version is a vastly superior, revelatory experience compared to the earlier release that they were already delighted by, they'll keep giving you enough money for you to finish scaling the "revolutionary" cliff and take a (brief) rest with "evolutionary" versions. Business has not only forgotten how important that whole process is to their continued survival, but they've removed the capability for their bespoke software (and Web) infrastructure to use and reuse that model. All that remains is "it's ok if we ship crap; so does everybody else." That's the kind of thinking that made General Motors the world-bestriding Goliath it is today - as opposed to the wimpy also-ran it was (emphatically NOT) half a century ago. We really don't need any more businesses going over that sort of cliff.

What we do need, and urgently, are two complementary, mutually dependent things. We need a sea change in the attitude of (most) businesses, even technology businesses, towards software - to realise and acknowledge that the Pointy-Haired Boss is not merely a common occurrence in the way business manages software, but actively threatens the success of any project (and business) so infested. Just as businesses at some point realise that "paying any price to cut costs" is an active threat to their own survival, they need to apply that reality to their view of and dealings with the technical infrastructure that increasingly enables their business to function at all.

Both dependent on that and as an enabler of that change, the software and Web development industry really needs to get its house in order. We need to get away from the haphazard by-guess-and-by-golly estimation and monitoring procedures in use by the majority of projects (whose elaborate Microsoft Project plans and PowerPoint decks bear less and less resemblance to reality as the project progresses) and enforce use of the tools and techniques that have been proven to work, and have an organised, structured quest to research improvements and New Things.. Despite what millions of business cards and thousands of job advertisements the world over proclaim, there is no true discipline of "software engineering", any more than there was "oilfield engineering" in widespread use before the New London School explosion of 1937. Over 295 people died in that blast; we have software-controlled systems that, should they fail, could in fact hurt or kill many more - or cause significant, company- or industry-ruinous physical damages. We should not wait for such an event before "someone" (at that point, almost certainly an outside governmental or trans-governmental entity) says "These are the rules." While I understand and agree with the widespread assertion that certification tests in their present form merely demonstrate an individual's capability to do well on such tests, we do need a practical, experiential system - probably one modelled on the existing systems for engineering, law or medicine. Not that people should work 72-hour shifts; there's enough of that already. But rather that there should be a progression of steps from raw beginner to fully-trusted professional, with a mix of educational and experiential ingredients to ascend that progression, and continuing educational and certificating processes throughout one's entire career. The cost for this is going to have to be accepted as "part of the system" by business; if business wants properly competent engineers, and not just the latest boatload of unknowns with mimeographed vendor certs, then they're going to have to realize that that benefit does not come without cost to all sides. The free ride is over - for all the stakeholders at the table.

Jaw-Droppers - Blast from the Past

Just when you thought it was safe to forget that the 1970s ever existed... this gem shows up on the XML Daily Newslink, a mailing list I follow intermittently. (Actually, this was included in the XMLDN from Wed 11 Feb — an indication of how "closely" I've been following lately.)

Developing a CICS-Based Web Service
G. Subrahmanyam, G. Mokhasi, S. Kusumanchi; SOA World Magazine

Web services have opened opportunities to integrate the applications at an enterprise level irrespective of the technology they have been implemented in. IBM's CICS transaction server for z/OS v3.1 can support web services. It can help expose existing applications as web services or develop new functionality to invoke web services. One of the commonly used protocols for CICS web services is SOAP for CICS. It enables the communication of applications through XML. It supports as a service provider and service consumer independent of platform and language. SOAP for CICS enables CICS applications to be integrated with the enterprise via web services as part of lowering the cost of integration and retaining the value of the legacy application. SOAP for CICS also comes along with the implementation encoder and decoder.

"SOAP for CICS"? Give it a REST, guys. On the other hand.... preserving this by-now more-solid-than-most-rocks 1969-era technology IS a signal achievement; an example of engineering stability on par with Soyuz.

On the other hand... support for legacy technologies like this looks set to become increasingly expensive and risky over time, since apparently:

1. Numerous surveys published in the last few years indicate that the sizable majority of "old mainframe" tech folk will have retired by 2010 (do the math on the years), and

2. partly as a result of (1), users risk become increasingly dependent on outsourced Indian providers — hardly conducive to effective project control.

In my own professional view, technological preservation activities like this are mainly useful for one thing: they can serve as the pattern against which a re-implementation of the business process using more currently-supportable tech can be verified. And once an organization has done this for its most valuable legacy systems (which wouldn't have been preserved this long if they weren't so valuable), the actual and perceived risk of migrating to new technologies as conditions warrant drops dramatically. After all, if you (or your internal colleagues) have successfully brought your line-of-business systems from CICS via REST to, say, PHP or Java, you're a lot more comfortable with the idea of migrating to whatever the mid-to-trailing technology is in ten years' time — while the support costs of the (then) existing system are still manageable.

Are any of you actually involved in any technological archaeology like this? When I was younger, I used to brag that half the systems I'd worked with were older than I was, but that stopped being true about 1995. (AFAIK, I was one of the last guys to professionally touch a working IBM 360 mainframe, in about 1989.)

(Original article at this entry's title link, which is also here.)