Redundancy is Repetitive, Inefficient and Counterproductive

But you already know that.

I've long been a supporter of free and open source software, even contributing to a few projects. However, my enthusiasm has cooled a bit in the last several years, as the time I've been able to devote to such projects has both dwindled and been divided among more projects.

I'm continually flabbergasted by the number of open-source projects that cover the same ground, ad infinitum, often with little to no apparent differentiation between them. By continually "reinventing the same wheel," these small projects usually manage to put out a small number of alpha- or beta-quality releases, and then cease to make visible progress. This could be because the original developer(s) became frustrated or distracted; because the project succumbed to internal politics (a leading cause of FLOSS project death); or merely because the developer(s) realised that they were deep in the rut of a path that has been travelled many, many times before.

This flash of obviousness came about as I've been researching and evaluating PHP frameworks for Web development. I've been developing and maintaining sites using a couple of these, notably Kohana 2.3, but for various reasons have been reluctant to make the switch to the now-current 3.0 release. I decided to do a relatively formal, structured evaluation of several of the frameworks out there, rather than just pounce on the first I came across that looked "reasonable." (Silly me, but oh well.) Of course, as my list of frameworks to look at grew from six, to twenty, to considerably more, I became uncomfortable; torn between the reality that I'm doing this all myself (and devoting significantly less than 100% time to the endeavour), but at the same time wanting to remain open to pleasant surprises. After all, "pleasant surprises" were how I stumbled upon Kohana in the first place, and though I am strongly considering using another tool in its place (again, for non-technical reasons), contributing to a megalithic monoculture is not my idea of "adding value."

Today I did something that I should have done a week or two ago: I went out to SourceForge and got a list of all the PHP frameworks registered on the site.

Oh, my effing $DEITY! "Searching gives 1316 results."

Obviously, not all of those are appropriate (there are numerous reinventions of the CMS wheel, for instance); many can be filtered simply because they haven't published updates recently enough to meet my criteria, and so on, but that's all rather beside my current point.

Open-source projects are (in)famously both insular and prolific, leading to the kind of nonsense I ran into on SourceForge. Commercial products have a built-in inhibitor of this particular pathology: if a market gets divided among so many competitors that nobody can cover their costs, then in very short order the existing products will either consolidate into fewer, more sustainable alternatives or die altogether. Open-source projects, particularly those started by people inexperienced in such projects, fail to realise that they too are competing for resources just as much as their commercial brethren. In the FLOSS case, the finite, competed-for resource is time, both of (potential) contributors and of passive users (what the commercial world calls "customers").

Some open-source projects recognise this. In the Linux and Unix/BSD desktop worlds, the "market" for desktop managers (that provide the windows, icons, menus, and so on that are the UI to the user) has been dominated by two projects, KDE and GNOME. Though there are other choices with solid offerings and active communities, a developer can choose to develop for either KDE or GNOME with the confidence that virtually anyone in his target audience will be able to run his software on their system of choice. Consequently, most projects that have to make a choice, choose KDE or GNOME.

The Web is a different beast entirely, and while standards are for the most part very well-defined, the "barrier to entry" for those seeking to do things (allegedly) differently is very low. Some in the PHP Web development community advise new developers to tackle creating their own framework, as a learning tool. I suspect that that's how many of the SourceForge listings originated. I also happen to think that the advice itself is naïve, akin to a new auto mechanic being advised to build his own engine to gain a better understanding of those from Toyota or Ford. What I would recommend instead is that a new developer learn one or two of the more widely-used frameworks, browse around for another two or three that seem interesting, and start contributing to the project(s) of her choice. Virtually all such projects recognise their need for more help, as well as the fact that "we were all newbies once."

Consolidation in the PHP-framework field, while maintaining the low barrier to entry, should be a Good Thing™: to a greater degree than commercial products, FLOSS projects should succeed on their merits, technical and otherwise. As more developers join a well-run project community, the project's software should continue to both improve and find wider user/customer use. The ability to "do your own thing" should be kept, if only as a check against the use of marketing hype over technical merit.

But I believe that relatively small, informal "A- and B-List" groups of selections would be good for projects and developers (and developers' clients); a lot of what I've come across in the last few days would be on my "P-List," if not "ZZZZ."

Oh yeah, by the way: this sort of maturation and shared effort would do wonders for our Craft's progress towards true professional status. Any of us who've had our schedules and budgets tinkered with by people who admit they know nothing of what we do should understand the desirability of that goal. Given the degree to which software permeates public life and policy, that "desirability" has become a dire necessity; it's just more expedient for people to pretend that isn't really the case. Stuxnet, anyone?

She's Putting Me Through Changes...

...they're even likely to turn out to be good ones.

As you may recall, I've been using and recommending the Kohana PHP application framework for some time. Kohana now offer two versions of their framework:

• the 2.x series is an MVC framework, with the upcoming 2.4 release to be the last in that series; and

• the 3.0 series, which is an HMVC framework.

Until quite recently, the difference between the two has been positioned as largely structural/philosophical; if you wished to develop with the 'traditional' model-view-controller architecture, then 2.x (currently 2.3.4) is what you're after; with great documentation and tutorials, any reasonably decent PHP developer should be able to get Real Work™ done quickly and efficiently. Oh the other hand, the 3.0 (now 3.0.4.2) offering is a hierarchical MVC framework. While HMVC via 3.0 offers some tantalising capabilities, especially in large-scale or extended sequential development, there remains an enthusiastic, solid community built around the 2.3 releases.

One of the long-time problems with 2.3 has been how to do unit testing? Although vestigial support for both a home-grown testing system and the standard PHPUnit framework exists in the 2.3 code, neither is officially documented or supported. What this leads to is a separation between non-UI classes, which are mocked appropriately and tested from the 'traditional' PHPUnit command line, and UI testing using tools like FitNesse. This encourages the developer to create as thin a UI layer as practical over the standalone (and more readily testable) PHP classes which that UI layer makes use of. While this is (generally) a desirable development pattern, encouraging and enabling wider reuse of the underlying components, it's quite a chore to get an automated testing/CI rig built around this.

But when I came across a couple of pages like this one on LinkedIn (free membership required). This thread started out asking how to integrate PHPUnit with Kohana 2.3.4, and then described moving to 3.0 as

I grabbed Kohana 3, plugged in PHPUnit, tested it, works a treat! So we're biting the bullet and moving to K3! :)

I've done a half-dozen sites in Kohana 2.3, as I'd alluded to earlier. I've just downloaded KO3 and started poking at it, with the expectation to move my own site over shortly and, in all probability, moving 3.0 to the top of my "recommended tools" list for PHP.

Like the original poster, Mark Rowntree, I would be interested to know if and how anybody got PHPUnit working properly in 2.3.4.

Thanks for reading.

NIH v. An Embarrassment of Riches

One thing most good developers learn early on is not to "reinvent" basic technology for each new project they work on, The common, often corporate, antithesis to this is NIH, or "Not Invented Here." But sometimes, it's hard to decide which "giants" one wants to "stand on the shoulders of."

I've recently done a couple of mid-sized Web projects using PHP and the Kohana framework. A framework, as most readers know, is useful a) by helping you work faster b) by including a lot of usually-good code you don't have to write and maintain (but you should understand!). Good frameworks encourage you to write your own code in a style that encourages reuse by other projects that use the same framework.

One task supported by many frameworks is logging. There have also been many "standalone" (i.e., not integrated into larger systems) logging packages. The most well-known of these, and the source of many derivatives, is the Apache log4j package for Java. This has been ported, also as an Apache project, is log4php.

Log4php has saved me countless hours of exploratory debugging. I stand firmly with the growing group of serious developers who assert that if you use a reasonably agile process (with iterative, red, green, refactor unit testing) and make good use of logging, you'll very rarely, if ever, need a traditional debugger.

What does this have to do with Kohana? Well, Kohana includes its own relatively minimalist, straightforward logging facility (implemented as static methods in the core class, grumble, grumble). There's a standard place for such logs to be written to disk, and a nice little 'debug toolbar' add-on module that lets you see logging output while you're viewing the page that generated it.

So I ought to just ditch log4php in favor of the inbuilt logging system when I'm developing Kohana apps, right? Not so fast...

Log4php, as does log4j, has far more flexibility. I can log output from different sources to different places (file, system log, console, database, etc.), have messages written to more than one place (e.g., console and file), and so on. Kohana's logging API is too simple for that.

With log4php, I have total control over the logging output based on configuration information stored in an external file, not in the code itself. That means I can fiddle with the configuration during development, even deploy the application, without having to make any code changes to control logging output. The fewer times I have to touch my code, the less likely I am to inadvertently break something. Kohana? I only have one logging stream that has to be controlled within my code, by making Kohana method calls.

Many experienced developers of object-oriented software are uncomfortable with putting more than one logical feature into a class (or closely-related set of classes). Why carry around overhead you don't use, especially when your framework offers a nice extension capability via "modules" and "helpers"?. While there may sometimes be arguments for doing so (the PHP interpreter is notoriously slow, especially using dynamic features like reflection), I have always failed to understand how aggregating large chunks of your omniverse into a Grand Unified God Object™ pays dividends over the life of the project.

So, for now, I'll continue using log4php as a standalone tool in my various PHP development projects (including those based on Kohana). One thing that just went onto my "nice to do when I get around to it" list is to implement a module or similar add-on that would more cleanly integrate log4php into the surrounding Kohana framework.

This whole episode has raised my metaphorical eyebrow a bit. There are "best practices" for developing in OO (object-oriented) languages; PHP borrows many of these from Java (along with tools like log4php and PHPUnit, the de facto standard unit-test framework). I did a fairly exhaustive survey of the available PHP frameworks before starting to use Kohana. I chose it because it wasn't a "everything including several kitchen sinks" tool like Zend, it wasn't bending over backwards to support obsolete language misfeatures left over from PHP 4, and it has what looks to be a pretty healthy "community" ecosystem (unlike some once-heavily-flogged "small" frameworks like Ulysses). I'm not likely to stop using Kohana very soon. I may well have to make time to participate in that community I mentioned earlier, if for no other reason that to better understand why things are the way they are.

But that's the beauty of open source, community-driven development, surely?

Reuse, Renew, Recycle - Data Structures Edition

Everything gets recycled these days, including acronyms...

Whether you're developing Web pages in your language of choice, writing a new game to take on the world, or (insert your project here), at some point, you're going to deal with structured data.

You're quite likely, at some point, to deal with fairly complex, nested structured data - for configuration, to preserve the state of objects within your software, or whatever. In recent years, "complex, nested structured data" has often been coded up in XML. This can save your coding bacon; there are quite a few nice tools out there to work with XML, insulating you from having to deal with the "raw" <tag ...="" attribute="value">data<nested_tag>more data</nested_tag></tag> soup.

But at some point you may well end up having to hand-edit the thing — you may want to pull something out for reuse, or you suspect that one of the tools you're using is messing up in some way... it happens. And even during that bright, wonderful time when you're building a POC (proof of concept/ possibly overly c***tastic; see, I said we could reuse), you wind up humping a lot of "extraneous" data around that just really wraps a few small things. Ugh.

Even with compression, you're still taking a hit; you've just moved it from the pipe to the CPU. Twice.

I got handed a client project a little over a year ago that used this nifty notation called YAML. While YAML was apparently originally developed for Ruby apps, it's evolved; there are now support tools/libraries for everything from PHP to Visual Basic to Haskell. "YAML Ain't Markup Language," but it can express very complex data structures; nesting is defined by indentation, item types by decorations. (See the spec or this PHP implementation for more detail.)

For projects which have been using binary data storage for non-BLOB data, simply because XML was seen as overkill, this might well simplify things. Likewise, if you're pushing data around between different apps/platforms, thinking "I maybe oughta look at JSON", this is for you.

I've been using it primarily in PHP to encode various data and content models, and it has allowed me to dramatically simplify my code in several places - reducing complexity, inefficiency, and possible bugs. The Ruby and Python crowds have been using this for a while; PHP folk have tended to look at it suspiciously, since it came from Over in Those Languages.

---

Good tools work in any language.

---

Reuse? I was talking about recycled acronyms earlier?

At about the same time I found YAML the non-markup-language markup language, I also came across a nifty CSS framework with the highly original name of YAML. This code, which makes it pretty easy to style sites using Yet Another Multicolumn Layout, is a set of CSS (2.x) stylesheets, with minimal graphics and JavaScript thrown into the box. What it gives you is a nice, consistent layout on just about any browser (including hideously-broken variants of MS Internet Exploder) which you can slice and dice pretty much at will. It even degrades sensibly for non-CSS, non-JS browsers — like the Googlebot. If you know how to read CSS reasonably well and this takes you more than an hour to grok, you're working too hard. Browse the docs, the samples, the tutorials, or the online community rumblings, and you'll be productive in no time.

And yes, you can use both together on the same Web dev project. Just make sure you keep your tools straight. After all, YAML is not at all the same thing as YAML.

OMFG, or Holy Deforestation, Batman!

As some of you know, I'm working on a book on Web development, using off-the-shelf tools (frameworks, template engines, JavaScript libraries, etc.) to leverage semantic, standards-compliant, accessible, search-friendly Websites. (That's more a matter of adjusting your development philosophy and workflow than anything else, but I digress). As part of that, I'e been doing a (reasonably) comprehensive review of PHP 5 application frameworks. You might have heard of ezComponents or CakePHP, but the 900-kg elephant in the room is definitely the Zend Framework. It ships with the Dojo JavaScript toolkit, but doesn't make it excessively difficult to mix and match others (Scriptaculous, Prototype, jQuery, etc.) if desired.

And here's another reason to call ZF the '900-kg elephant' — the programmer's reference guide (for version 1.7) weighs in at a <sarcasm>svelte</sarcasm> 1170 pages. Don't print this at home, folks. Better yet, just don't print it... either browse it online or download the PDF. Save a forest or six. For you old-timers, this will remind you quite a bit of US DOD Standard 2167A, "fondly" remembered as "documentation by the boxcar load".

It's easy to think there's a war going on...

(playing softly, in the background of my mind, The Beatles'Revolution)

....between the Web developers promoting nice, clean development with RESTful, semantic (X)HTML judiciously enhanced with CSS and JavaScript (henceforth often referred to as the "Army of Light") and those using "popular", "mainstream" frameworks such as CakePHP and the Zend Framework, who route everything through a Front Controller of some sort, and often seem to be in the dubious company of WS-Whatever Web services (which, I am reliably told, provide ample amounts of The Wrong Kind of job security — they know more about your app than you do, and aren't telling what they know). The sides would seem to be pretty cut-and-dried, judging from a lot of the blog activity (Google REST XML-RPC PHP to get a million and a half or so hits of light reading material). Except...

Briefly skimming through the Zend Framework documentation, for instance, and looking at the QuickStart and tutorials reinforces the idea that URL handling is routed through a front controller to an application-specific action controller, which is the C in the notorious (and some say overused) MVC (model-view-controller) framework. Originally developed to help improve desktop-application development, particularly in languages like Java and Smalltalk, it became popular for Web development because.... it seemed like a good idea at the time. Actually, for Web development in the Pleistocene (say, late-1990s), it was a good idea. Anything that cut through the estimated 27.612 interconnected details that needed to be simultaneously mastered to get a "Hello, World" EJB up and happy was, by its very existence, a Very Good Thing. And so, when shops moved to more productive, less pathologically irrational development systems than J2EE, the models and design patterns that had saved their bacon were brought over into the New World, to maintain conceptual touchstones that helped Useful Work Get Done. Happiness abounded throughout the realm, until apps started outgrowing the meager bounds of static HTML and became "Rich Internet Applications". (To the tune of "Lions and Tigers and Bears, Oh My!", you hear faint murmurs of "AJAX and WS-* and REST, Oh My!") And, to pile on the snowclones, there really be dragons there.

'Dragons' in the form of falling into a GET-centric, action-oriented, everything-just-a-click-away world of convoluted Web apps with limited (re)usability and even less understandability to those who haven't swum there in some time. The entire promise of REST is simple: by centering applications around resources, rather than actions (through the use of URIs; Universal Resource Identifiers) and following the eminently sensible notion of not putting kilobytes of state information into those URIs (necessary information is POSTed along with the URI request), many problems that become painfully visible in large systems, simply go away. (Try sending a link to a cool book you found on Amazon over an instant messenger chat.)

But a typical, outsourced-development, haven't-really-used-this-tool-and-you-want-it-WHEN?!? developer isn't going to think of those things. He's going to grab a tool that has promising-sounding Google hits, run through a tutorial or two, and then plunge into the Son of the Enhancement of the Rewrite of yehey.com, with the customer sending him an "is it done yet?" email every six minutes. Clean design? What's that? Well-guarded state transitions? Who's got time to even understand that, let alone implement it? If we don't get it done, the customer's going to pull the project and send it to Vietnam or somewhere...

Just to make one point absolutely clear: I don't mean to be picking on Zend and CakePHP as being more than simply representative of widely-used, well-reputed tools that can be used to get the unwary, rushed developer (are there any other kind earning a paycheck?). While it is entirely practical to write semantic, RESTful Web applications in both frameworks (and both document how to do so), it's like, say, RPG; a fantastic tool for solving problems in a well-defined domain, usable with significant effort outside that domain, and Zeus help you if you use it to write an MMORPG.

The real point of this rant, if it hasn't hit you like a Muhammad Ali speed-anchor punch, is another pout over the state into which we've allowed the once-honorable craft of software development (of which Web development is but a specific case) into absolute bollocks. We've allowed the pay-any-price-to-cut-costs, pinch-a-penny-until-you-can-hear-it-scream-from-Boise-to-Bangalore idiots pervert us from Muhammad Ali (or at least Sonny Liston) into Herschel Shmoikel Pinkus Yerucham Krustofski. A plurality, if not yet an overwhelming majority of those who call themselves "software development 'engineers'" have been given neither sufficient formal training in their craft nor the resources (time, money, support, etc.) to continue learning as they go. "If you can spell EJB and ERP, you're the guy for us — as long as you're young and dirt cheap. And when you're done with that, we've got some BASIC code we want in Java instead."

So at the unique moment in history when ephemeral intellectual artifacts have assumed primacy in a wide range of human affairs, the humans whose intellect is responsible for their creation and correct functioning have progressively less ability to do the job properly. The way that, if they sit back and think for a moment, they know should be possible, has to be possible in any sort of rational omniverse whatsoever. But few, if any, ever get that chance for reflection. Fewer still, having reflected, researched and enlightened themselves, are welcomed back into the paying ranks who toil away at this once-noble craft.

And my Zend Framework code still feels slimy. It's not Zend's fault, at least, not entirely. Front controllers are good; front controllers are your friends; front controllers are.... *crunch!*