Going Forward; Avoiding the Cliff While Looking for Cliff's Notes

Over on the Ruby sub-Reddit, I got into a discussion (mostly with Jelly_Jim, the OP, and with realnti) about the conflation in far too many minds of Ruby, the programming language, with Ruby on Rails, the famously-"opinionated" Web application framework. While pontificating on Jelly_Jim's original question, I described my frustration-bordering-on-antipathy with the traditional (known as "The Rails Way") application structure; listed a few presentations, books and blog posts that have influenced me greatly, and described my own take on the Kübler-Ross model for Ruby-based (and at least initially, largely Rails-based) Web development.

That discussion, over the course of roughly a day, helped me figure out how our app is going to survive the expected Slashdot effect, transitioning from a pretty traditional, megalithic Rails app to an app that uses Rails components in a sharply different, more recognisably "object-oriented", service-based architecture.

Leaving aside my more prosaic experiential difficulties with Rails (I really loathe ActiveRecord as commonly used), and taking into account the "Uncle" Bob Martin keynote, the various blog posts I referenced in the original Reddit post, and a couple of books I mentioned in a later reply, I think I've possibly hit upon a way to get where I want us to go.

That's the easy part, of course; it only took me never-mind-how-many months. The "interesting" part is going to be getting there — understaffed, on a shoestring budget for the moment even by startup standards (another problem I'm working), and working a schedule for two and a half years that would land a healthy twenty-year-old in hospital by now. Comments/suggestions are, as always, greatly appreciated.

---

Basic Architectural Principles

• Loosely-coupled, granular architecture FTW. That means, inter alia,
  • Nearly the entire app to be packaged as Gems used by an otherwise minimal application;
  • Rails mountable engines to package assets (particularly CoffeeScript/JavaScript) and major components;
  • Plan and design for updating and provisioning components which communicate across natural architectural seams.
• Hexagonal, "ports-and-adapters", or "clean" conceptual architecture; all dependencies point inward. Your domain entities, at the centre of the system, will change less than details like the delivery mechanism (Web UI, phone app, etc) or the database (which the domain shouldn't even be able to prove exists as such).
• By adopting a heavily job-story-oriented view of the workflows; with the use of online tools like Trello, WebSequenceDiagrams.com; and with supporting code tools like ActiveInteraction and so on, we should be a lot more productive and a lot more Agile than we have been during our forty (calendar time: two) years lost in the wilderness.
• And oh, yeah, just to keep things interesting: we've got to keep regularly demonstrating forward user-visible progress, for all the obvious reasons and a couple I really can't go into here.

---

How We Get There, v. 28.0

First, since the major focus of late has been on some unique (and not-so-unique) CoffeeScript code, start there. Separate the existing Script code out into four engine-based Gems, following/adapting Derek Prior's example. These levels are, from conceptual bottom to top:

1. A "gateway" CoffeeScript class we've written as a poor man's RequireJS, that gives us just enough pseudo-dynamic referencing of data/code without changing anything the Rails asset pipeline assumes;
2. Our directly DOM-aware Script code, as one or possibly two Gems (one is a major crown jewel; we might decide to split off utility/support code into a separate Gem, so as not to touch one when maintaining the other);
3. The code proxying our internally-used message queue (decoupling "FTW", remember?) and the Ajax code to communicate with the Rails back end; and
4. The code implementing various domain/service objects sitting atop the Script stack; one Gem per service.

Next, adapt the relatively-straightforward (as yet) Ruby/Rails code to honour the architectural principles listed earlier. This is where tools like ActiveInteraction pull their weight (and solve several existing pain points in the process).

Just to make sure we really do understand where we're going with this, take our first user story-specific code (including interactor) and package it up as a Gem that should then Just Work as in the previous step. Start thinking about whether we really want traditionally-packaged Gems or unbuilt dependencies that remain part of the single app repository.

Proceed to reimplement and repackage the (again, relatively few) job stories already existing as was done in the preceding step.

Start knocking down the remaining Trello cards as we complete our app.

We've been saying since the Very Beginning that anything that we were building that wasn't inextricably tied to proprietary, commercially-valuable code (or that gives too much of a hint into any unique details it may have) "should be" openly released, but hadn't yet figured out a feasible, cleanly reusable way to do that. If we've done our jobs properly up to this point, we have that way now.

Push the "deploy and launch" Big Green Button. We've bloody well earned it by now.

---

Any Suggestions or Comments?

"You use the tools you have, not the tools you wish you had."

(with apologies to Donald Rumsfeld, and noting that this is frustrating precisely because we're so asymptotically close!

I forget who first said this, but "Experience is as much a catalogue of what doesn't work as well as you'd hoped as what does" might have had my career in mind. Or at least the 2013 segment of it.

A few short months ago, I came across Trello, by Joel Spolsky's Fog Creek Software. As with all great Web products, it started as a brilliantly simple idea (a Web version of the classic stick-user-story-index-cards-to-it whiteboard that every Agile project centres around) that's slowly sinking under a tsunami of well-intentioned customer feature requests.

So excuse me if I seem to be a)piling on in b)an arguably hypocritical manner when I say that, like most things, it's both a blessing and a curse. You use it for a while, you start thinking in terms of cards and such… and then you realise that it's really not going to help you as much as you expected. Partly because of features, alas, but more because of philosophy, or communication of same.

Some features that I would gladly pay to be able to use:

1. There doesn't appear to be a "show me a list of all cards on this board, sorted by card number (creation date), last modification, and so on" feature. Apparently, the workflow Trello expects you to adopt is to have your to-be-done and in-work cards in various lists, and your "done" cards (and lists) should be archived. If, instead, you prefer to have a "Done" list, you'll quickly find that it grows into a mess that you're going to be searching through using the (very handy in itself) filtering feature, but that requires you to remember keywords exactly and, more importantly, prevents the "aha!" moments that often come when you can look at a list of not-very-obviously-related information and draw new insights from what you're seeing.

2. Trello also have (generally deservedly) made a Big Deal out of their support for "Markdown Everywhere", but their parser doesn't support the feature that future-proofs Markdown itself; the ability to embed HTML inline seamlessly. This is especially noticeable for things like tables, which have no Markdown (or Trello) direct implementation.

   This lack means, for instance, that "rolling your own" card that contains a list of all cards in the board looks pretty cluttered and hard-to-read:

   

3. Checklists are almost optimally useful and easy to use. However, they suffer from one major misfeature: converting a checklist item to a separate card "removes the item and creates a card with the title of the item beneath the current card". It would be very useful to have Trello, instead of deleting the checklist item after the card has been created, replace the checklist item's text with a link to the new card. Many (most?) of our cards start out as blue-sky, high-level description of a functional feature. For example, "Any User, or any Guest, may view a list of all Articles with public status." That card had a six-item checklist on it, several of which turned out to be "worth" breaking out to separate cards. For each of those checklist-items-turned-cards, I went back and re-added an item to the checklist with a one-sentence summary of the card and a link (e.g., "Update the `Ability` model; see Card #47."), followed by editing the new "item" card so that its description started out like "This is a checklist item from Card #44."

   Why? In our process, each card (most granular tracked activity) should take a reasonably consistent amount of time, generally under a man-day of effort. When a single card instead takes 4-5 days (as has happened), it's easy for customer reps and investors to get the idea that we've slacked off because they understand our estimating process/goals. Being able to drill down (and up) from any given card, without the level of effort presently required on my part, helps avoid misunderstandings and time lost responding to panicked emails.

4. Trello allows you to "subscribe" to cards, lists and boards. This adds items to a list viewable when you're logged into trello. It would be much more useful if these "feeds" were available via RSS/Atom as well.

tl;dr: Trello is a great tool; so great, that many users (including me) are either pulling it either in directions it wasn't designed to go, or are demanding features that don't exist in the way we'd like them. I'll keep using it and recommending it for now, even though the time involved in running a project with it is scaling up prodigiously. That time, however, is less than the time it would take to work without Trello or something quite similar to it.

Stubs Aren't Mocks; BDD Isn't TDD; Which Side(s) Are You On?

I just finished re-reading Martin Fowler's Mocks Aren't Stubs from 2007. I wasn't as experienced then in the various forms of agile development as I am now, so couldn't quite appreciate his perspective until somebody (and I'm sorry I can't find whom) brought up the paper again in a tweet a month or two ago. (Yes, that's how far behind I am; how do you do when you're working 15- to 18-hour days, 6 or 7 days a week for 6 months?)

In particular, the distinctions he draws between "classical" and "mockist" test-driven development (TDD), and then between mockist TDD and behaviour-driven development (BDD) are particularly useful given the successes and challenges of the last dozen or so projects I've been involved with. I wouldn't quite say that many teams are doing it wrong. They/we have been, however, operating on intuition, local folklore and nebulously-understood principles gained through trial-and-error experience. Having a systematic, non-evangelistic, nuts-and-bolts differentiation and exploration of various techniques and processes is (and should be) a basic building block in any practitioner's understanding of his craft.

Put (perhaps too simply), the major distinction between classic and mockist TDD is that one focuses on state while the other focuses on specific, per-entity function; projects that mix the two too freely often come to grief. I believe that projects, especially midsize, greenfield development projects by small or inexperienced teams should pick one approach (classic or mockist TDD, or BDD) and stick with it throughout a single major-release cycle. You may credibly say "we made the wrong choice for this product" after getting an initial, complete version out the door, and you should be able to switch the next full release cycle to a different approach. But if you don't know why you're doing what you're doing, and what the coarse- and fine-grained alternatives are to your current approach, you can't benefit from having made a conscious, rational decision and your project thus can't benefit from that choice.

Anything that gives your team better understanding of what you're doing, why and how will enhance the likelihood of successfully delivering your project and delighting, or at least satisfying, your customers. Even on a hobby project where your customer is…you yourself. Because, after all, your time is worth something to you, isn't it?

Rules can be broken, but there are Consequences. Beware.

If you're in an agile shop (of whatever persuasion), you're quite familiar with a basic, easily justifiable rule:

No line of production code shall be created, modified or deleted in the absence of a failing test case, for which the change is required to make the test case pass.

Various people add extra conditionals to the dictum (my personal favourite is "…to make the test case pass with the minimal, most domain-consistent code changes currently practicable") but, if your shop is (striving towards being) agile, it's very hard to imagine that you're not honouring that basic dictum. Usually in the breach at first, but everybody starts in perfect ignorance, yes?

I (very) recently was working on a bit of code that, for nearly its entire existence over several major iterations, had 100% test coverage (technically, C0 or line coverage) and no known defects in implemented code. It then underwent a short (less than one man-week) burst of "rush" coding aimed at demonstrating a new feature, without the supporting tests having been done beforehand. It then was to be used as the basis for implementing a related set of new features, that would affect and be affected by the state of several software components.

That induced some serious second-guessing. Do we continue mad-hatter hacking, trusting experience and heroic effort to somehow save the day? Do we go back and backfill test coverage, to prove that we understand exactly what we're dealing with and that it works as intended before jumping off into the new features (with or without proper specs/tests up front)? Or do we try to take a middle route, marking the missing test coverage as tech debt that will have to be paid off sometime in The Glorious Future To Come™? The most masochistic of cultists (or perhaps the most serenely confident of infinite schedule and other resources) would pick the first; the "agile" cargo-cultist with an irate manager breathing fire down his neck the second; but the third is the only pragmatic hope for a way forward… as long as development has and trusts in assurances that the debt will be paid in full immediately after the current project-delivery arc (at which time increased revenue should be coming in to pay for the developer time).

The moral of the story is well-known, and has been summed up as "Murphy " (of Murphy's Law fame) "always gets his cut", "payback's a b*tch" and other, more "colourful" metaphors. I prefer the dictum that started this post, perhaps alternately summed up as

Don't point firearms at bits of anatomy that you (or their owner) would mind losing. And, especially, never, ever do it more than once".

Because while even the most dilettante of managers is likely to have heard Fred Brooks' famous "adding people to a late project only makes it later" or his rephrasing as "nine women cannot have a baby in one month", too few know of Steve McConnell's observation (from 1995!) that aggressive schedules are at least equally delay-inducing. If your technical people say that, with the scope currently defined, that something will take N man-weeks, pushing for N/4 is an absolutely fantastic way to turn delivery in N*4 into a major challenge.

Remember, "close" only counts in horseshoes and hand grenades and, even then, only if it's close enough to have the intended effect. Software, like the computers it runs on, is a binary affair; either it works or it doesn't.

Cover Yourself: Toolchains Are Agile, Too

As people who know me professionally and/or read my blog know well, I have been a (raucously) loud evangelist for test-first development (TDD, BDD, Scrum, whatever your flavour) for years now. If I write even an exploratory bit of code and don't have tests in place first, I get very uncomfortable. As complexity increases, without tests (preferably automated, repeatable tests), I argue that I simply can't know what's really going on, because I can't prove it.

A major corollary to this is test coverage reporting. If I can't see what's been tested and what hasn't, then in a very real sense nothing has been, since I can't document/prove what has been and what hasn't. And the better (more productive) teams I've worked in have established, and regularly hit, coverage testing better than 95%, with 100% being a common (and commonly attained) goal. (Edit: Note that this is for C0 and C1 test coverage; tools that cover C3 and C4 are rare to nonexistent in most languages, such as Ruby.)

As you may also know, I've been getting (back) into Ruby development, using Rails 3 on Ruby 1.9. Ruby's long-time de facto standard coverage tool for many years was rcov, which generally worked well. However, Rob Sanheim has stated that "RCov does not, and will not support C based Ruby 1.9.x implementations due to significant changes between 1.8 and 1.9.". He recommends either SimpleCov or CoverMe. Ripping out RCov and replacing it with CoverMeSimpleCov on a test project took all of five minutes and left me with attractive, functional, (so far apparently) quite accurate reports.

One of the basic principles of agile development is that the team must actively embrace constructive change as their project evolves. It's often easy for harried, hurried people to forget that that applies to their tools as much as it does to what they produce using those tools.

Just a thought as my evening winds down.

Know when to walk away; know when to run

This started out as a reply to a comment on the LinkedPHPers group on LinkedIn; once I started writing, of course, it quickly grew beyond what was appropriate as a conversationally-inline comment. So I brought it over here. It's something I've been thinking about for a couple of weeks now, so let me get this anvil off my chest.

---

To R Matthew Songer1: I'd advise adding Ruby on Rails to that list of alternatives. I've been writing PHP for fun and profit since PHP 4 was still a future hope wrapped in hyperbole. Now that we've finally got a (mostly-)decent language in 5.3 for some serious software development, I'm burning out. Part of that burnout is due to what I see in the PHP community, part to geography, and part to other factors that approach "fit for use".

I've recently taken a post as Chief Engineer at a little startup nobody's heard of yet. Our prototype was done in PHP; it got the idea across well enough for our initial investors and customers to bang on our door. So the CEO and I thought, great, we'll find another senior2 PHP guy, write a real app (the prototype doesn't even have testable seams), and we're off.

If I were looking to hire a battalion of deputy junior assistant coders whose main qualification as PHP devs was being able to spell it whilst taking endless trips down the waterfall, I could do that. I was aiming higher: I wanted someone who knew his way around current best practices; who realises that being experienced is no excuse to stop learning aggressively; who understands how to build web apps that can evolve and scale; who looks for the best feasible way to do something instead of just the first one that pops into mind. I especially needed to find someone who was as BDD/TDD-infected as I am, and recognised the value of building tools and automation early (but incrementally) so that we'd be a lean, agile development machine when the rubber really needed to hit the road, a (very) few short weeks from now.

In the States, or in much of Europe, or basically anyplace outside Singapore, I'd probably have a decent chance of finding such a person. Here, not so much. I was especially concerned by the way PHP usage seems to be devolving in these parts. There are lots of folks who think that it's a good idea to reinvent Every. Single. Wheel. themselves, without really knowing all that much about what has gone before. Frameworks are a good example; if you aren't intensely aware of how good software, sites and Web apps get built; if you don't even bother with encapsulation or MVC or SOLID or on and on, how can you expect anybody who does know his Craft from a hole in the ground to take you or your "tool" seriously? It might wow the management rubes who proudly admit they don't know squat — but in the world as it is, or at least as it's going to exist in the very near future, those will (continue to) be a vanishing breed even here. Even in a Second World Potemkin village of a city-state where what you are and who you know is far more important in most situations than what you've done and what you know, you're still running a risk that somebody is going to come along who actually gets up in the morning and applies herself or himself to writing better stuff than they wrote yesterday. And, eventually, they're going to wipe the floor with you — either you as an individual or you as a society that remains stubbornly top-down-at-any-cost.3

In contrast, my experience talking with Ruby and Rails people here, even people with absolute minimal experience in Rails, is chalk-and-Friday different. For one illustrative example: one book that more than half the yes-I've-done-a-bit-of-Rails folk I've talked to here is Russ Olsen's Eloquent Ruby (ISBN 978-0-321-58410-6). Your average PHP dev — or C++ dev or Java dev — is fighting the dragons (schedule, complexity, communication, etc.) far too much to get out of fire-drill-coding mode and into writing. If you believe, as I do, that all truly competent software is written as a basis for and means of conversation between humans, and incidentally to be executed by a computer, then you know what I'm driving at here. Knuth's dictum that programming is a creative, literary act is too easily lost when you're just throwing yourself against the wall every day trying to see which of you will break first. (It's very rarely the wall.)

If you write software that can be read and understood, and intelligently expanded on or borrowed from, and your whole view of the omniverse changes. Your stress goes down, your written works' quality goes up, and you enjoy what you do a lot more. Automating away the repeated, detailed work and having a sensible process so you don't give yourself enough rope to shoot yourself in the foot regularly (with apologies to Alan Holub) is the most reliable way yet found to get your project schedule under your control instead of vice versa.

All this does however have one pre-supposition, which I have received numerous complaints on here locally: you and your team must be fully literate and fluent in a shared human language4. The vast majority of software to date seems to be associated with four such: Business Standard English, Russian, Japanese and Chinese. If your team shares a different language at a (near-)native level, with one or more of you having similar skills in the earlier four, you should be able to make do for yourself rather nicely. Having others build on your work, if desired, is going to be a bit more problematic. Poor communication has been at least a strong contributing cause, if not the root cause, of every failed project I have seen in my career. If you can't speak the customer's language effectively enough, with nuance, subtlety and clarity as needed, then your chances of project success are somewhat worse than my chances of winning the next six consecutive Toto jackpots — and I don't plan on buying any tickets.

Footnotes:

1. "I am sitting here tossing the options around for a business application, browser based, and trying to decide...PHP, Java, Python, .NET?" (Return)

2. Why insist on a senior guy when they're so rare here (at least the PHP variety)? Because I figure that with a senior dev coming on a month from now, we'd spend roughly half the time between now and our first drop-deadline writing magnificent code, and the other half building infrastructure and process to make building, testing and deploying the magnificent code we write straightforward enough not to distract the team from the creative acts of development. There's not enough time to wipe anybody's backside. (Return)

3. The phrase "at any cost" always reminds me of a company I contracted to back in the late '80s. My boss there had a (large) sign above his desk, "We will pay any price to cut costs." The company eventually paid the ultimate price — bankruptcy. (Return)

4. I'm describing what used to be called "college- or university-level language skills", but as any university teach will readily tell you, the skills exhibited by students have dropped precipitously and measurably in the last three decades or so. (Return)

Patterns and Anti-Patterns: Like Matter and Anti-Matter

Well, that's a few hours I'd like to have over again.

As both my regular readers know, I've long been a proponent of agile software development, particularly with respect to my current focus on Web development using PHP.

One tool that I, and frankly any PHP developer worth their salt, use is PHPUnit for unit testing, a central practice in what's called test-driven development or TDD. Essentially, TDD is just a bit of discipline that requires you to determine how you can prove that each new bit of code you write works properly — before writing that new code. By running the test before you write the new code, you can prove that the test fails... so that, when you write only the new code you intend, a passing test indicates that you've (quite likely) got it right.

At one point, I had a class I was writing (called Table) and its associated test class (TableTest). Once I got started, I could see that I would be writing a rather large series of tests in TableTest. If they remained joined in a single class, they would quickly grow quite long and repetitive, as several tests would verify small but crucial variations on common themes. So, I decided to do the "proper" thing and decompose the test class into smaller, more focused pieces, and have a common "parent" class manage all the things that were shared or common between them. Again, as anyone who's developed software knows, this has been a standard practice for several decades; it's ordinarily the matter of a few minutes' thought about how to go about it, and then a relatively uneventful series of test/code/revise iterations to make it happen.

What happened this afternoon was not "ordinary." I made an initial rewrite of the existing test class, making a base (or "parent") class which did most of the housekeeping detail and left the new subclass (or "child" class) with just the tests that had already been written and proven to work. (That's the key point here; I knew the tests passed when I'd been using a single test class, and no changes whatever were made to the code being tested. It couldn't have found new ways to fail.)

Every single test produced an error. "OK," I thought, "let's make the simplest possible two-class test code and poke around with that." Ten minutes later, a simplified parent class and a child class with a single test were producing the same error.

The simplified parent class can be seen on this page, and the simplified child class here. Anybody who knows PHP will likely look at the code and ask, "what's so hard about that?" The answer is, nothing — as far as the code itself goes.

What's happening, as the updated comments on pastebin make clear, is that there is a name collision between the ''data'' item declared as part of my TableTest class and an item of the same name declared as part of the parent of that class, PHPUnit's PHPUnit_Framework_TestCase.

In many programming languages, conflicts like this are detected and at least warned about by the interpreter or compiler (the program responsible for turning your source code into something the computer can understand). PHP doesn't do this, at least not as of the current version. There are occasions when being able to "clobber" existing data is a desirable thing; the PHPUnit manual even documents instances where that behaviour is necessary to test certain types of code. (I'd seen that in the manual before; but the significance didn't immediately strike me today.)

This has inspired me to write up a standard issue-resolution procedure to add to my own personal Wiki documenting such things. It will probably make it into the book I'm writing, too. Basically, whenever I run into a problem like this with PHPUnit or any other similar interpreted-PHP tool, I'll write tests which do nothing more than define, write to and read from any data items that I define in the code that has problems. Had I done that in the beginning today, I would have saved myself quite a lot of time.

Namely, the three hours it did take me to solve the problem, and the hour I've spent here venting about it.

Thanks for your patience. I'll have another, more intelligent, post along shortly. (That "more intelligent" part shouldn't be too difficult now, should it?)

Process: Still 'garbage in, garbage out,', but...

...you can protect yourself and your team. Even if we're talking about topics that everybody's rehashed since the Pleistocene (or at least since the UNIVAC I).

Traditional, command-and-control, bureaucratic/structured/waterfall development process managed to get (quite?) a few things right (especially given the circumstances). One of these was code review.

Done right, a formal code review process can help the team improve a software project more quickly and effectively than ad-hoc "exploration and discovery" by individual team members. Many projects, including essentially all continuing open-source projects that I've seen, use review as a tool to (among other things) help new teammates get up to speed with the project. While it can certainly be argued that pair programming provides a more effective means to that particular end, they (and honestly, most agile processes) tend to focus on the immediate, detail-level view of a project. Good reviews (including but not limited to group code reviews) can identify and evaluate issues that are not as visibly obvious "down on the ground." (Cédric Beust, of TestNG and Android fame, has a nice discussion on his blog about why code reviews are good for you.

Done wrong, and 'wrong' here often means "as a means of control by non-technical managers, either so that they can honour arbitrary standards in the breach or so that they can call out and publicly humiliate selected victims," code reviews are nearly Pure Evil™, good mostly for causing incalculable harm and driving sentient developers in search of more humane tools – which tend (nowadays) to be identified with agile development. Many individuals prominent in developer punditry regularly badmouth reviews altogether, declaring that if you adopt the currently-trendy process, you won't ever have to do those eeeeeeeeevil code reviews ever again. Honest. Well, unless.... (check the fine print carefully, friends!)

Which brings us to the point of why I'm bloviating today:

1. Code reviews, done right, are quite useful;

2. Traditional, "camp-out-in-the-conference-room" code reviews are impractical in today's distributed, virtual-team environment (as well as being spectacularly inefficient), and

3. That latter problem has been sorted, in several different ways.

This topic came up after some tortuous spelunking following an essentially unrelated tweet, eventually leading me to Marc Hedlund's Code Review Redux... post on O'Reilly Radar (and then to his earlier review of Review Board and to numerous other similar projects.

The thinking goes something like, Hey, we've got all these "dashboards" for CRM, ERP, LSMFT and the like; why not build a workflow around one that's actually useful to project teams. And these tools fit the bill – helping teams integrate a managed approach to (any of several different flavours of) code review into their development workflow. This generally gets placed either immediately before or immediately after a new, or newly-modified, project artifact is checked into the project's SCM. Many people, including Beust in the link above, prefer to review code after it's been checked in; others, including me, prefer reviews to take place before checkin, so as to not risk breaking any builds that pull directly from the SCM.

We've been using collaborative tools like Wikis for enough years now that any self-respecting project has one. They've proven very useful for capturing and organising collective knowledge, but they are not at their best for tracking changes to external resources, like files in an SCM. (Trac mostly finesses this, by blurring the lines between a wiki, an SCM and an issue tracker.) So, a consensus seems to be forming, across several different projects, that argues for

• a "review dashboard," showing a drillable snapshot of the project's code, including completed, in-process and pending reviews;

• a discussion system, supporting topics related to individual reviews, groups of reviews based on topics such as features, or the project as a whole; these discussions can be searched and referenced/linked to; and

• integration support for widely-used SCM and issue-tracking systems like Subversion and Mantis.

Effective use of such a tool, whatever your process, will help you create better software by tying reviews into the collaborative process. The Web-based versions in particular remove physical location as a condition for review. Having such a tool that works together with your existing (you do have these, yes?) source-code management and issue-tracking systems makes it much harder to have code in an unknown, unknowable state in your project. In an agile development group, this will be one of the first places you look for insight into the cause of problems discovered during automated build or testing, along with your SCM history.

And if you're in a shop that doesn't use these processes, why not?

---

On a personal note, this represents my return to blogging after far, far too long buried under Other Stuff™. The spectacularly imminent crises are now (mostly, hopefully) put out of our misery now; you should see me posting more regularly here for a while. As always, your comments are most welcome; this should be a discussion, not a broadcast!

NIH v. An Embarrassment of Riches

One thing most good developers learn early on is not to "reinvent" basic technology for each new project they work on, The common, often corporate, antithesis to this is NIH, or "Not Invented Here." But sometimes, it's hard to decide which "giants" one wants to "stand on the shoulders of."

I've recently done a couple of mid-sized Web projects using PHP and the Kohana framework. A framework, as most readers know, is useful a) by helping you work faster b) by including a lot of usually-good code you don't have to write and maintain (but you should understand!). Good frameworks encourage you to write your own code in a style that encourages reuse by other projects that use the same framework.

One task supported by many frameworks is logging. There have also been many "standalone" (i.e., not integrated into larger systems) logging packages. The most well-known of these, and the source of many derivatives, is the Apache log4j package for Java. This has been ported, also as an Apache project, is log4php.

Log4php has saved me countless hours of exploratory debugging. I stand firmly with the growing group of serious developers who assert that if you use a reasonably agile process (with iterative, red, green, refactor unit testing) and make good use of logging, you'll very rarely, if ever, need a traditional debugger.

What does this have to do with Kohana? Well, Kohana includes its own relatively minimalist, straightforward logging facility (implemented as static methods in the core class, grumble, grumble). There's a standard place for such logs to be written to disk, and a nice little 'debug toolbar' add-on module that lets you see logging output while you're viewing the page that generated it.

So I ought to just ditch log4php in favor of the inbuilt logging system when I'm developing Kohana apps, right? Not so fast...

Log4php, as does log4j, has far more flexibility. I can log output from different sources to different places (file, system log, console, database, etc.), have messages written to more than one place (e.g., console and file), and so on. Kohana's logging API is too simple for that.

With log4php, I have total control over the logging output based on configuration information stored in an external file, not in the code itself. That means I can fiddle with the configuration during development, even deploy the application, without having to make any code changes to control logging output. The fewer times I have to touch my code, the less likely I am to inadvertently break something. Kohana? I only have one logging stream that has to be controlled within my code, by making Kohana method calls.

Many experienced developers of object-oriented software are uncomfortable with putting more than one logical feature into a class (or closely-related set of classes). Why carry around overhead you don't use, especially when your framework offers a nice extension capability via "modules" and "helpers"?. While there may sometimes be arguments for doing so (the PHP interpreter is notoriously slow, especially using dynamic features like reflection), I have always failed to understand how aggregating large chunks of your omniverse into a Grand Unified God Object™ pays dividends over the life of the project.

So, for now, I'll continue using log4php as a standalone tool in my various PHP development projects (including those based on Kohana). One thing that just went onto my "nice to do when I get around to it" list is to implement a module or similar add-on that would more cleanly integrate log4php into the surrounding Kohana framework.

This whole episode has raised my metaphorical eyebrow a bit. There are "best practices" for developing in OO (object-oriented) languages; PHP borrows many of these from Java (along with tools like log4php and PHPUnit, the de facto standard unit-test framework). I did a fairly exhaustive survey of the available PHP frameworks before starting to use Kohana. I chose it because it wasn't a "everything including several kitchen sinks" tool like Zend, it wasn't bending over backwards to support obsolete language misfeatures left over from PHP 4, and it has what looks to be a pretty healthy "community" ecosystem (unlike some once-heavily-flogged "small" frameworks like Ulysses). I'm not likely to stop using Kohana very soon. I may well have to make time to participate in that community I mentioned earlier, if for no other reason that to better understand why things are the way they are.

But that's the beauty of open source, community-driven development, surely?

Remember to test your testing tools!

I've been doing some PHP development lately that involves a lot of SPL, or Standard PHP Library exceptions. I do test-driven development for all the usual reasons, and so make heavy use of the PHPUnit framework. One great idea that the developer of PHPUnit had was to add a test-case method called setExpectedException(), which should eliminate the need for you (the person writing the test code) to do an explicit try/catch block yourself. Tell PHPUnit what you expect to see thrown in the very near future, and it will handle the details.

But, as the saying says, every blessing comes with a curse (and vice versa). The architecture of PHPUnit pretty well seems to dictate that there can only be one such caught exception in a test method. In other words, you can't set up a loop that will repeatedly call a method and pass it parameters that you expect it to throw on; the first time PHPUnit's behind-the-scenes exception-catcher catches the exception you told it was coming, it terminates the test case.

Oops. But if you think about it, pretty expectable (pardon the pun). For PHPUnit to catch the exception, the exception has to get thrown and unwind the call stack past your test-case method. That makes it very difficult (read: probably impossible to do reliably inside PHPUnit's current architecture) to resume your test-case code after the call that caused the exception to be thrown — which is what you'd want if you were looping through these things.

This leaves you, of course, with the option of writing try/catch blocks yourself — which you were hoping to avoid but which still works precisely as expected.

Moral of the story: Beware magic bullets. They tend to blow up in your face when you least expect it.

Test Infection Lab Notes

In a continuing series... As current and former colleagues and clients are well aware, I have been using and evangelizing test-driven development in one flavor or another since at least 2001 (the earliest notes I can find where I write about "100% test coverage" of code). To use the current Agile terminology, I've been "test-infected". My main Web development language is PHP 5.2 (and anxiously awaiting the goodness to come in 5.3), using Sebastian Bergmann's excellent PHPUnit testing framework. PHPUnit uses a well-documented convention for naming test classes and methods. One mistake often made by people in a hurry (novices or otherwise) is to neglect those conventions and then wonder why "perfectly innocuous" tests break. I fell victim to this for about ten minutes tonight, flipping back and forth between test and subject classes to understand why PHPUnit was giving this complaint:

There was 1 failure:
1) Warning(PHPUnit_Framework_Warning)
   No tests found in class "SSPFPageConfigurationTest".

FAILURES!
Tests: 1, Failures: 1.

about this code:

class SSPFPageConfigurationTest extends PHPUnit_Framework_TestCase
    public function canConstruct()
    {
        $Config = new SSPFPageConfiguration();
        $this->assertTrue( $Config instanceof SSPFPageConfiguration );
    }
};

which was "obviously" too simple to fail. The wise programmer is not afraid to admit his errors, particularly those arising from haste. The novice developer proceeds farther on the path to enlightenment; the sage chuckles in sympathy, thinking "been there, done that; nice to be reminded that other people have, too". May you do a better job of keeping your koans in a nice, neat cone.

ANFSD: starting a series to scratch an itch

(And Now For Something Different, for the 5LA-challenged amangst you...)

I've made my living, for about half my career, on the proposition that if I stayed (at least) three to six months ahead of (what would become) the popular mean in software technology, I'd be well-positioned to help out when Joe Businessman or Acme Corporation came along and hit the same technology — with the effect of "Refrigerator" Perry hitting a reinforced-concrete wall. This went reasonably well as "the market" started using PCs, then GUIs, then object-oriented programming, and then "that Internet thingy" (Shameless plug: résumé here or in PDF format).

In other ways, I've been a staunch traditionalist. I've used IDEs from time to time, because I was working as part of a team that had a standard tool set, or because I was programming for Microsoft Windows and the Collective essentially requires that that be done in their (seventh-rate) IDE unless you want to decrease productivity by several dozen orders of magnitude.

Otherwise, just give me KATE or BBEdit and a command-line compiler and I'm happy. This continued for a significant chunk of the history of PCs, until I decided that, for the Java work I was doing, I really needed some of the whiz-bang refactoring and other tie-ins supported by Eclipse and NetBeans. Then I started hacking around on a couple of open-source C++ packages and thought I'd give the Eclipse C/C++ Development Tooling a try. Now I'm coming up to speed on wxWidgets development in C++.

During this learning-curve week, I spent a lot of time browsing the Web for samples, tutorials and so on. To call most of them execrable is to give them unwarranted praise. Having recently resumed work on a Web development book dealing with useful standards and helpful process, and since I've been doing C++ off and on since the mid-80s, I thought I'd start a series of blog entries that would:

• Document some of the traps and tricks I hit to get a simple wxWidgets program into Eclipse;
• Illustrate some early, very simple refactoring of the simple program to get a bit more sanity;
• Get Subversion and Eclipse playing well together;
• Explain why I think parts of teh Agile method are simulataneously nothing new and the best new idea to hit development in a very long time.
• Start using an automated-testing tool to build confidence during debugging and refactoring; and
• Using a code-documentation tool in the spirit of JavaDoc to produce nice technical/API docs.

At the end of the series, you'll have a pretty good idea of how I feel most projects (regardless of underlying technology and specific tools) "should" be done. You'll have seen a very simple walk-through of the process, demonstrated using Linux, Eclipse, C++ and wxWidgets, but actually quite broadly applicable well beyond those bounds.

Please send comments, reactions, job offers, etc., to my email. Death threats, religious pamphlets, and other ignorance can, as always, go to /dev/null. Thanks!

Improvement as opposed to Change

A link on the Agile CMMi blog had some very interesting things to say about a gentleman named Brian Lyons, the CEO (as well as CTO and founder) of Number Six, a Vienna, Virginia (Washington, DC area) software technology services/business consulting firm. Number Six quite obviously 'get' the concepts behind Agile CMMi, Hillel Glazer of the blog wrote. Interested, I flipped over to their site to hopefully learn a bit more, maybe drop off a CV.

Mr. Glazer wrote the blog entry in question on 25 July. The first thing you notice when viewing the Number Six site is the home-page obituary of and tribute to Brian Lyons, who died on (Monday) 3 September 2007 in a motorcycle accident. There are various links to family and tribute sites, a press release, and a request that "in lieu of flowers, the family has requested that donations be made to" a scholarship fund at the University of Maryland (excellent, particularly for those too far or too late to attend the funeral). I wish the family and company all the best in their times of grief and trouble.

The point I originally started writing this entry to make, however, was inspired by one of the bullets on Number Six' careers page, under the heading "Why Six?": "We are committed to consistent improvement, not continual drastic change."

Think about that distinction for a moment. Many of us, particularly those in the software-geek persuasion, start our careers trying to remake everything; not necessarily because it's broken, but so that we can do it (whatever it is), make it ours, introduce new techniques or technologies, and hopefully (but improbably) provide a better solution to the problem at hand than what was there before.

The forces arrayed against that impulse are formidable. Playing the role of (and often in actual fact) those who are by nature suspicious of any radical approach to a problem, too often dismissing out of hand any alleged improvement or innovation without "giving it a fair chance", in the eyes of the wet-behind-the-ears Young Turk. Occasionally, of course, improvements and innovations too beneficial to ignore do come out of this process.

As the developer matures (which may take days, decades or eternities), the Young Turk recognizes that not all of the hindrances were traditionalist per se; rather, they were operating from a different (usually business-oriented) set of priorities. Learning to understand and appreciate those priorities is one of the fundamental aspects of any successful transformation from ivory-tower Geek to journeyman software craftsman (or -woman), able to make a contribution to customers and to the craft of software development.

What becomes obvious, to all thoughtful participants and stakeholders in the process of developing and maintaining any software system is that a natural tension exists between three apparently conflicting ideas:

• Don't change what works well enough when there are more pressing needs to attend to;
• New capabilities, chosen and implemented properly, can have strongly beneficial effects (efficiency, productivity, wider customer scope, 'better' according to various aspects of quality);
• Any change to an existing system involves direct and indirect costs which must be carefully evaluated and compared against the expected improvements.

Balancing and managing the conflict between those concepts throughout the lifetime of a piece of software is an art that has yet to be thoroughly and reliably mastered with a level of reliability and cost widely acceptable to business customers. Several philosophies and techniques (often marketed as technologies) have been developed and marketed as (attempted) solutions to that problem. Three that I have found extremely useful, in complementary ways, over the last few decades are agile development, refactoring and the CMMI. (Those already familiar with the concepts may skip down to the paragraph which begins "As argued by practitioners such as Hillel...".)

The CMMI (previously more widely and less precisely known as the CMM, for Capability Maturity Model) has been around for quite some time as a formal means of evaluating the maturity and coherence of an organization's development efforts (among other activities). It requires its users (organizations) to acquire and document ever-increasing detail of precisely how they go about the process of development and how they are required to be able to prove (in an internal or external audit) that those processes are being followed and any discrepancies noted and accounted for. This suffers as a standalone policy framework for software development on two grounds:

• the "what" and "why" of development are completely ignored. CMMI fundamentally doesn't care if you're building a pile of junk that has serious practical problems, as long as you do it using the process you've defined for yourself.
• The CMMI, along with US DOD Standard 2167A, have been blamed for the logging of more trees (to make paper) than virtually any other industry business practice. This is largely because both are seen (in CMMI's case, somewhat unfairly) as pushing "hidden mandates" for the rigid , never-far-from-obsolete "waterfall" model of development.

Agile development, on the other hand, is a survival response to both the (largely management-driven) mantra that "Real Artists Ship", and on the other hand, to seemingly interminable periods between "milestones" (particularly in the waterfall model) where nothing of intrinsic value is visible to an outside observer (e.g., management or customers). Its main criticism from opponents is that it produces too little documentation (the product is the documenatation, to a large degree).

An aspect of, or adjunct to, agile development is refactoring, pioneered and popularized in the Martin Fowler-written book. Refactoring is all about how you can make (sometimes radical) changes to the internals of a software system, provided you keep the external (customer-facing, revenue-generating) interfaces constant. Wrapping one's mind around this as a formalized process, as opposed to the "don't-break-what-works" mentality common to experienced developers (and managers), is a threshold experience in a software craftsman's progress.

As argued by practitioners such as Hillel, the combination of the process-centric CMMI and the solution-centric, visible-progress Agile set of processes (including, particularly, refactoring) gives the "best of both worlds" for development — a focus on artifacts and products created through a survivable process, matched with a demonstrated and documented knowledge of exactly what is meant by "that process" under current circumstances. This also implies demonstrating understanding of the specifics of "current circumstances" that drive the decision-making and artifact-development processes — which brings us back full circle to understanding why Agile is a good fit to begin with.

Unlike the Rational Unified Process (RUP), which grew out of a waterfall-driven management-oriented mentality (and the opportunity to sell software tools and consulting services to any shop using the heavily-marketed process), Agile and CMMI development can be successfully started after passing around a few books among the development and management staff. Tools and training are, chosen thoughtfully, extremely helpful, and the appraisal process within CMMI (formally SCAMPI, the well-known "Level 1" to "Level 5" assessment of process maturity) does eventually require an outside assessor, or registrar, to formally certify the organization. But to take a typical small development group from the customary initial chaos to a finely-tuned, market-leading, customer-satisfying machine, it has been my experience and observation that it is far easier (and more cost-effective) to implement CMMI in an Agile fashion than to go down the (vendor-specific) RUP route. Imposing either on a large organization would require an unlikely combination of managerial brilliance, sadism and masochism; revolutions boiling up from below and winning eventual management sanction have proven much more likely to be successful in that type of environment.

So what's a Young Turk (or a more seasoned craftsman) to make of all this blather? The simple point: Consistent improvement is practical, and without life- or career-threatening implications, achievable on a repeatable basis. Continual drastic change, in contrast and almost by definition, will lead the development group to many sleepless nights trying to get that last show-stopper bug fixed, management to wonder why those bozos in development can't be trusted to ship anything on time, and customers to wonder what they're really getting for their money beyond the hype and bling of the marketing materials. Which team would you rather be on?

The Vision, Forward through the Rear-View Mirror

The vision I'm trying to promote here, which has been used successfully many times before, is that of a very flexible, highly iterative, highly automated development process, where a small team (like ours) can produce high-quality code rapidly and reliably, without burning anybody out in the process. (Think Agile, as a pervasive, commoditized process.) Having just returned (17 October) from being in hospital due to a series of small strokes, I'm rather highly motivated to do this personally. It's also the best way I can see for our small team to honour our commitments.

To do this, we need to be able to:

• have development artifacts (code/Web pages) integrated into their own documentation, using something like Javadoc;
• have automated build tools like Ant regularly pull all development artifacts from our software configuration management tool (which for now is subversion)
• run an automated testing system (like TestNG) on the newly built artifacts;
• add issue reports documenting any failed tests to our issue tracking system, which then crunches various reports and
• automatically emails relevant reports to the various stakeholders.

The whole point of this is to have everybody be able to come into work in the morning and/or back from lunch in the afternoon and know exactly what the status of development is, and to be able to track that over time. This can dramatically reduce delays and bottlenecks from traditional flailing about in a more ad hoc development style.

Obviously, one interest is test-driven development, where, as in most so-called Extreme Programming methods, all development artifacts (such as code) are fully tested at least as often as the state of the system changes. What this means in practice is that a developer would include code for testing each artifact integrated with that artifact. Then, an automated test tool would run those tests and report to the Quality Engineering team any results. This would not eliminate the need for a QE team; it would make the team more effective by helping to separate the things which need further exploration from the things that are, provably, working properly.

Why does this matter? For example, there was an article on test-driven development in the September 2005 issue of IEEE Computer (reported on here) that showed one of three development groups reducing defect density by 50% after adopting TDD, and another similar group enjoying a 40% improvement.

All this becomes interesting to us at Cilix when we start looking at tools like:

• Cobertura for evaluating test coverage (the percentage of code accessed by tests)
• TestNG, one successor to the venerable JUnit automated Java testing framework. TestNG is an improvement for a whole variety of reasons, including being less intrusive in the code under test and having multiple ways to group tests in a way that makes it much harder for you to forget to test something;
• Ant, the Apache-developed, Java-based build tool. This is, being Java-based and not interactive per se, easy to automate;
• and so on, as mentioned earlier.