Why is "doing things right" a "thing"? Whither professionalism?

Over on the Reddit r/programming subreddit, /u/jfalvarez posted a link to Tom Stuart's great presentation at the Scottish Ruby Conference 2014, entitled "The DHH Problem".

If you're involved with Ruby or Web development in any way, it's well worth three and a half minutes of your time.

On the Reddit post where Mr Alvarez posted the link to the video, another Redditor, dventimi, asked

Why is this a thing?

I disagree with the later poster who snarked "People like gossip." It's much, much more important than that.

What follows is the content of my reply to "why is this a thing?" I'd love to have a discussion with you about it, either here or on the original post.

---

Because people are finally waking up to the fact that the coding style and philosophy espoused by DHH are disastrous in non-toy applications maintained over any significant period of time.

That this should have been obvious to all, but wasn't discussed as such, once Twitter, after having conflated Rails with Ruby as a whole, left Ruby because they couldn't make Rails do what they needed to no matter how many times DHH came onsite and insulted their intelligence, speaks eloquently to how broken things are.

Most master software developers I've met and worked with have been mediocre-at-best promoters, including self-promoters. DHH has the opposite problem: his initial extraction of Rails from Basecamp's codebase "worked", for apps very much like Basecamp at the time. His fixed, what some have called "trolling", attitudes and the personality they describe, prevent him from seeing things from any other perspective or from growing his skills beyond a certain point. The echo chamber of acolytes and zealots his sublime self-promotional talents have allowed him to accrete around himself "ensure" that he never has to.

That's a problem for the rest of us, who have deadlines, work to finish within those deadlines, and paycheques we'd like to be able to cash that we won't get unless we meet those deadlines with that work. Especially for those of us who wish to improve our craft and, in whatever way we can, nudge ourselves and others towards helping the craft of software development more closely approximate a true engineering discipline.

Paying excessive heed to the opinions-masquerading-as-principles brayed by DHH, and twisting ourselves to fit inside The Rails Way, actively impede all of the above tasks. Tom Stuart was absolutely right to thank DHH for producing Rails and promoting it sufficiently that the rest of us get to write Ruby for a living…just before tearing DHH a new one, in the polite, discreet way only a well-educated Brit can, for getting so much of Rails and how software should be written so disastrously wrong.

Thank you, Tom Stuart. An entire industry may well be in your debt.

Trust, Safety, Sandi Metz and A Proverb Walked Into A Bar...

Rereading POODR again. I love the bit in Chapter 6 where she talks about using a post_initialize method to decouple subclasses from having to remember to call super in an initialize method override. That's the sort of thing I remember making work in UCSD Pascal back when it wanted to be the Next Hot Language™, and in a few languages since.

But one question I never found an acceptable answer to was, how do I prevent myself from forgetting to call super (or its equivalent) in a hierarchy with multiple levels? Sure, the base class implements a do-nothing post_initialize-type method, so that every single subclass anywhere in the hierarchy can "kick it upstairs" and things will Just Work.

But at some point, you have to trust yourself, your team, your successors who grapple with your code long after you are gone. Comprehensive tests will catch the fallout from a forgotten upstairs kick and give you a lovely red bar — or will show you that the parent's chained behaviour you thought you needed was either frivolous or incorrectly verified. (Or both. Shtuff happens; nobody ever went broke underestimating the degree to which developers can fool ourselves.) Whatever twisty little maze of keystrokes leads to you that point, you've discovered a refactoring opportunity. Enjoy your Homer Simpson Moment, make the appropriate changes, verify that the tests still pass, and move on.

Доверяй, но проверяй.

Going Forward; Avoiding the Cliff While Looking for Cliff's Notes

Over on the Ruby sub-Reddit, I got into a discussion (mostly with Jelly_Jim, the OP, and with realnti) about the conflation in far too many minds of Ruby, the programming language, with Ruby on Rails, the famously-"opinionated" Web application framework. While pontificating on Jelly_Jim's original question, I described my frustration-bordering-on-antipathy with the traditional (known as "The Rails Way") application structure; listed a few presentations, books and blog posts that have influenced me greatly, and described my own take on the Kübler-Ross model for Ruby-based (and at least initially, largely Rails-based) Web development.

That discussion, over the course of roughly a day, helped me figure out how our app is going to survive the expected Slashdot effect, transitioning from a pretty traditional, megalithic Rails app to an app that uses Rails components in a sharply different, more recognisably "object-oriented", service-based architecture.

Leaving aside my more prosaic experiential difficulties with Rails (I really loathe ActiveRecord as commonly used), and taking into account the "Uncle" Bob Martin keynote, the various blog posts I referenced in the original Reddit post, and a couple of books I mentioned in a later reply, I think I've possibly hit upon a way to get where I want us to go.

That's the easy part, of course; it only took me never-mind-how-many months. The "interesting" part is going to be getting there — understaffed, on a shoestring budget for the moment even by startup standards (another problem I'm working), and working a schedule for two and a half years that would land a healthy twenty-year-old in hospital by now. Comments/suggestions are, as always, greatly appreciated.

---

Basic Architectural Principles

• Loosely-coupled, granular architecture FTW. That means, inter alia,
  • Nearly the entire app to be packaged as Gems used by an otherwise minimal application;
  • Rails mountable engines to package assets (particularly CoffeeScript/JavaScript) and major components;
  • Plan and design for updating and provisioning components which communicate across natural architectural seams.
• Hexagonal, "ports-and-adapters", or "clean" conceptual architecture; all dependencies point inward. Your domain entities, at the centre of the system, will change less than details like the delivery mechanism (Web UI, phone app, etc) or the database (which the domain shouldn't even be able to prove exists as such).
• By adopting a heavily job-story-oriented view of the workflows; with the use of online tools like Trello, WebSequenceDiagrams.com; and with supporting code tools like ActiveInteraction and so on, we should be a lot more productive and a lot more Agile than we have been during our forty (calendar time: two) years lost in the wilderness.
• And oh, yeah, just to keep things interesting: we've got to keep regularly demonstrating forward user-visible progress, for all the obvious reasons and a couple I really can't go into here.

---

How We Get There, v. 28.0

First, since the major focus of late has been on some unique (and not-so-unique) CoffeeScript code, start there. Separate the existing Script code out into four engine-based Gems, following/adapting Derek Prior's example. These levels are, from conceptual bottom to top:

1. A "gateway" CoffeeScript class we've written as a poor man's RequireJS, that gives us just enough pseudo-dynamic referencing of data/code without changing anything the Rails asset pipeline assumes;
2. Our directly DOM-aware Script code, as one or possibly two Gems (one is a major crown jewel; we might decide to split off utility/support code into a separate Gem, so as not to touch one when maintaining the other);
3. The code proxying our internally-used message queue (decoupling "FTW", remember?) and the Ajax code to communicate with the Rails back end; and
4. The code implementing various domain/service objects sitting atop the Script stack; one Gem per service.

Next, adapt the relatively-straightforward (as yet) Ruby/Rails code to honour the architectural principles listed earlier. This is where tools like ActiveInteraction pull their weight (and solve several existing pain points in the process).

Just to make sure we really do understand where we're going with this, take our first user story-specific code (including interactor) and package it up as a Gem that should then Just Work as in the previous step. Start thinking about whether we really want traditionally-packaged Gems or unbuilt dependencies that remain part of the single app repository.

Proceed to reimplement and repackage the (again, relatively few) job stories already existing as was done in the preceding step.

Start knocking down the remaining Trello cards as we complete our app.

We've been saying since the Very Beginning that anything that we were building that wasn't inextricably tied to proprietary, commercially-valuable code (or that gives too much of a hint into any unique details it may have) "should be" openly released, but hadn't yet figured out a feasible, cleanly reusable way to do that. If we've done our jobs properly up to this point, we have that way now.

Push the "deploy and launch" Big Green Button. We've bloody well earned it by now.

---

Any Suggestions or Comments?

Clean Code. "I do not think that word means what you think it means."

From my copy of Clean Code: A Handbook of Agile Software Craftsmanship (the paper copy of which (barely) predates my work in Rails. The highlighted paragraph is what I'd highlighted in the paper copy:

Active Record

Active Records are special forms of DTOs. They are data structures with public (or bean- accessed) variables; but they typically have navigational methods like save and find. Typically these Active Records are direct translations from database tables, or other data sources.

Unfortunately we often find that developers try to treat these data structures as though they were objects by putting business rule methods in them. This is awkward because it creates a hybrid between a data structure and an object.

The solution, of course, is to treat the Active Record as a data structure and to create separate objects that contain the business rules and that hide their internal data (which are probably just instances of the Active Record).

I've been increasingly uncomfortable with the way Rails encourages people to conflate the domain model objects with persistence in Rails' core component, ActiveRecord. Running across this bit again makes me think of Inigo Montoya from The Princess Bride: You keep using that word. I do not think it means what you think it means. Judging from my own projects and those I've participated in and observed on Github, the fact that ActiveRecord does not properly implement or encourage the Active Record pattern is a primary root cause of many bugs and much confusion and heartburn.

Happy New Year.

"You use the tools you have, not the tools you wish you had."

(with apologies to Donald Rumsfeld, and noting that this is frustrating precisely because we're so asymptotically close!

I forget who first said this, but "Experience is as much a catalogue of what doesn't work as well as you'd hoped as what does" might have had my career in mind. Or at least the 2013 segment of it.

A few short months ago, I came across Trello, by Joel Spolsky's Fog Creek Software. As with all great Web products, it started as a brilliantly simple idea (a Web version of the classic stick-user-story-index-cards-to-it whiteboard that every Agile project centres around) that's slowly sinking under a tsunami of well-intentioned customer feature requests.

So excuse me if I seem to be a)piling on in b)an arguably hypocritical manner when I say that, like most things, it's both a blessing and a curse. You use it for a while, you start thinking in terms of cards and such… and then you realise that it's really not going to help you as much as you expected. Partly because of features, alas, but more because of philosophy, or communication of same.

Some features that I would gladly pay to be able to use:

1. There doesn't appear to be a "show me a list of all cards on this board, sorted by card number (creation date), last modification, and so on" feature. Apparently, the workflow Trello expects you to adopt is to have your to-be-done and in-work cards in various lists, and your "done" cards (and lists) should be archived. If, instead, you prefer to have a "Done" list, you'll quickly find that it grows into a mess that you're going to be searching through using the (very handy in itself) filtering feature, but that requires you to remember keywords exactly and, more importantly, prevents the "aha!" moments that often come when you can look at a list of not-very-obviously-related information and draw new insights from what you're seeing.

2. Trello also have (generally deservedly) made a Big Deal out of their support for "Markdown Everywhere", but their parser doesn't support the feature that future-proofs Markdown itself; the ability to embed HTML inline seamlessly. This is especially noticeable for things like tables, which have no Markdown (or Trello) direct implementation.

   This lack means, for instance, that "rolling your own" card that contains a list of all cards in the board looks pretty cluttered and hard-to-read:

   

3. Checklists are almost optimally useful and easy to use. However, they suffer from one major misfeature: converting a checklist item to a separate card "removes the item and creates a card with the title of the item beneath the current card". It would be very useful to have Trello, instead of deleting the checklist item after the card has been created, replace the checklist item's text with a link to the new card. Many (most?) of our cards start out as blue-sky, high-level description of a functional feature. For example, "Any User, or any Guest, may view a list of all Articles with public status." That card had a six-item checklist on it, several of which turned out to be "worth" breaking out to separate cards. For each of those checklist-items-turned-cards, I went back and re-added an item to the checklist with a one-sentence summary of the card and a link (e.g., "Update the `Ability` model; see Card #47."), followed by editing the new "item" card so that its description started out like "This is a checklist item from Card #44."

   Why? In our process, each card (most granular tracked activity) should take a reasonably consistent amount of time, generally under a man-day of effort. When a single card instead takes 4-5 days (as has happened), it's easy for customer reps and investors to get the idea that we've slacked off because they understand our estimating process/goals. Being able to drill down (and up) from any given card, without the level of effort presently required on my part, helps avoid misunderstandings and time lost responding to panicked emails.

4. Trello allows you to "subscribe" to cards, lists and boards. This adds items to a list viewable when you're logged into trello. It would be much more useful if these "feeds" were available via RSS/Atom as well.

tl;dr: Trello is a great tool; so great, that many users (including me) are either pulling it either in directions it wasn't designed to go, or are demanding features that don't exist in the way we'd like them. I'll keep using it and recommending it for now, even though the time involved in running a project with it is scaling up prodigiously. That time, however, is less than the time it would take to work without Trello or something quite similar to it.

When Development DOES Go Horribly Wrong™

I just finished watching Ian Cooper's talk at NDC Oslo, TDD, where did it all go wrong. If you're doing any flavour of TDD (BDD, etc.), this will likely be one of the more important hours you spend this year. Ian helped me wrap my head around the details of what some of those voices screaming in the back of my mind as I (and likely you) write code, were trying to make me understand.

Bear in mind that I've been doing test-driven development for a decade, and some form of behaviour-driven, outside-in development for perhaps half that. I was once known as a "crazy blue-sky hacker", who composed symphonic-length pieces of software at a single sitting. (I'm explicitly disclaiming anything about the quality of that software; while it may have seemed sufficiently fit for purpose at the time, there almost certainly were bugs aplenty waiting the hapless maintainer.)

One of Ian's main points in the talk, if not the main point, is that test-driven development (and its successors such as BDD) should test behaviours of the system, not how every little internal method behaves. Lots of great insights here, from how to be more effective with fewer tests, to "write unit tests that focus on behaviours and thus can be used for acceptance". More bluntly, "the reason to test is a new behaviour, not a method or a class." I'm as guilty of that as anybody.

That last has been something I've been trying to work toward for well over a year. Unfortunately, one of the poisonously bad habits I've picked up working in Rails is the idea that there's a one-for-one mapping of specs/tests to implementation classes and methods (model, controller, helper, etc.). These are the trees that keep me from seeing the forest of what I'm ultimately trying to accomplish, and more often than not, have me polishing the nodules on each root of each tree.

One of the things that comes out of this understanding for Rails or, to be fair, for any framework in any language that focuses more on implementation classes than expressed behaviours, is that I for some time now have avoided use of the generators, particularly scaffolds, for core application components (models, controllers, views, etc). My preferred workflow has evolved to something like what I list below. Note that this is my existing workflow, and does not yet incorporate the new understanding of proper TDD.

1. I'll usually do a "first whack at" a model spec and bare-bones model, just to have something "specific" to work with in the later steps. I am beginning to see the way I have been doing this as a bad habit, particularly in "traditional" Rails. A Rails model should be concerned with persistence and validation, and an absolutely minimal amount beyond that. What I really want to start out with is my initial idea of a "business" domain object, which is Not The Same Thing;
2. I'll write what in the Rails world is called a "feature spec" and most of the rest of the omniverse knows as some form of "integration spec", modelling how a specific feature works. ("As a Member, I want to create a new BlogPost, to begin a discussion about a topic.") The initial "high-level" spec may or may not evolve into a series of specs which each demonstrate an individual sub-feature, but will always be expressed from the outside in, mocking what does not yet exist;
3. Next is a controller spec and the controller that it specifies, as a "traditional" developer-oriented spec (or set of specs). When done, I should be able to integrate this controller into the feature spec written previously (still using mock models) and demonstrate that the controller works;
4. Now, I re-look at the model(s) involved, turning those into "live" specs and "real" code that works correctly with what I've written thus far; my feature specs and controller specs still pass;
5. After the models and controllers are done and working with the feature specs, I'll do the views. I should have a very firm idea what's needed for these by now, as they've been used throughout (in mock or throwaway form) by the feature specs; this is where I write the code that emits the HTML, CSS and Script code for the UI.

That was my old workflow. I'm going to explore how to adapt and adjust this to more effectively and efficiently get the job done in light of Ian Cooper's talk and a (very near-future) reread of Kent Beck's Test-Driven Development: By Example. I say "effectively and efficiently" as my current project's spec-to-live-code ratio (in LOC) is approximately 13:4. I think most of us would agree that that's a very strong code smell.

"Easy Credit" IS "Easy Debt". It sucks for finance. It sucks just as much for your code.

One of the great things about the Rails ecosystem is that there are lots of different tools that can save you time and pain during development. Some of them work by embracing and extending the Rails standard bits, like replacing (nearly) bog-standard ApplicationController subclasses with subclasses of InheritedResources::Base. Some of them instead swap out pieces of the original Rails stack, like replacing ActionView with Erector or something similar. I wouldn't be at all surprised if my current app that mixes the two practices (in different areas) isn't at all unusual.

But whichever route or routes you choose, you need to constantly remain aware of that choice, especially when considering new tools you'd like to incorporate into your app. And you may also find that the tools you're using, while superb for their intended purpose, complicate other choices you make in the course of developing your project. A tool which was the greatest painkiller since morphine when you first picked it up, now seems to be "getting in the way" as you get farther along in development.

As practitioners of agile development, this shouldn't scare or even surprise us (though our non-technical stakeholders may initially be less sanguine). Good agile architecture says that everything in your app other than the raw business rules is an implementation detail that should be testable and fully replaceable in isolation.

Novice to journeyman Rails developers may be forgiven at this point for saying "but that's the way Rails is; it all works together, even though you can swap pieces out as you like". A majority of Rails developers have at least experimented with swapping out Test::Unit for RSpec, or erb for Haml or Slim. More adventurous still are the folks who swap out the ORM, replacing ActiveRecord with, say, DataMapper or the aforementioned Erector for ActionView. Why are they trying such things? Because they're asking (quoting Wynne again), "what does a modular Rails architecture look like?" Or, "we know we've hit the wall with our current (connected) architecture; everything we touch introduces new tech debt if not outright breakage, but we don't know how to do anything else and stay true to The Rails Way."

That "wall" you've hit has nothing to do with the number of lines of code you've written nor the number of unique visitors your production app serves per day nor the revenue your app is bringing in the door. You've realised that your ability to keep bringing in that revenue, let alone increasing it, is dependent on you somehow reconciling your understanding of "agile development" and "software engineering-style practices" with "the Rails Way". One of these things is not like the others. If you haven't yet, watch both of those videos. Particularly in Matt Wynne's talk from about 5:30 where he talks about the crossover point between a "connected" architecture being cheaper/faster/better giving way to a "modular" one… that rings a lot of bells for me. Likewise when Uncle Bob rails against Rails apps screaming "I'm a Rails app, never mind what I do". I've done several projects, never mind how many, where I've gotten to that point (which Wynne describes as "hang on; I don't want to work on that application anymore. Something's seriously wrong with it") and my reaction to the realisation that "something is seriously wrong" has been to stop, look at everything I'm doing and evaluating how well I'm following convention, "best practice" and so on, come to the conclusion that I'm doing everything exactly the way I've been taught it should be done, and deciding that that proves the fault is in my understanding of the problem, so I start over, from scratch, trying to figure out where I fell off the cliff. With one application, that involved at least five restarts over the course of a year and a half… and I must have always "fallen off the cliff" at the same spot, because I never saw it coming before I realised I was at the bottom of that cliff once again. "Uncle Bob" Martin's keynote (from Ruby Midwest 2011) does a great job of explaining why an experienced OO software developer like me has always felt uncomfortable with Rails' interpretation of the "Model-View-Controller" architecture. By writing your app centred on what the app is for rather than how the app presents itself on the Web, you make your life a lot easier and more enjoyable. Looking back at my commit history over the last year and a half, I see that well over 2/3 of the times I was "stuck" for more than a couple of hours had more to do with "how do I adapt what I'm trying to do to how Rails wants me to do things" than "how do I implement (this feature)". That should, and has been, setting off all kinds of alarm bells; I'd just drank enough of the Flavor Aid to think that pain level was "normal" for Rails. It shouldn't be.

Going forward, my strategy for Rails development is going to work something like this:

1. Model the domain objects (Martin's "interactors", "entities", "boundaries" and "entity gateways", after Jacobsen "control objects") as POROs (Plain Old Ruby Objects);
2. Wrap those up in Gems whose use can be demonstrated by specs, supplemented by a simple (Web or JSON or command-line; doesn't matter) example or set of examples.
3. Plug those into a Web app using the current framework of choice (Rails, Padrino, JoeBobsAwesomeFramework, or whatever), whose sole purpose is to implement the "client" side of the boundary layer and present what comes back over the fence.

Now we're back to something more closely approximating classical agile development that focuses on delivering business value, not playing mind games with a tool that insists on making most of the important decisions for you, and makes them in ways that make it harder for your app to be scalable, performant, or fun to work on. Or even for your app to be finished.

Thoughts?

---

P.S. Oh, and as far as InheritedResources::Base, that truly useful tool for Rails app development I mentioned before? We'll continue to use it as a default, but if we choose to use tools that "get in the way" (like wisper) by wanting all your methods explicitly defined, we'll switch back to ApplicationController, on the theory that making the puzzle one piece smaller without affecting how the other pieces fit together is a Good Thing™. But that's another decision we want to defer as long as possible; "good architecture" is all about deferring as many decisions as possible as long as possible, while still getting business value done.

Playing Mind Games With git, And Seemingly Winning

I had some bad experiences with git a few years back, and until a year or so ago, you couldn't get me to touch it with a ten-parsec pole. Let's just say that things got lost… changes, files, jobs, companies, little things like that. So, building out a Rails shop over the last year, with git being The Standard™(thank you, Github), you'd think I'd be super-extra-hyper-careful about it, right?

Well, the training wheels have to come off sometime, and if they take rather important bits of your skull along for the ride, consider it a Learning Experience. You were warned, weren't you? You did read that .2-point type on the side of the tin with the big label LIFE! that said "Death will be an inevitable finale to Life. Don't bother the lawyers", didn't you? Oh, well. Details.

Rule #16,472 in Dickey's Secrets of Software Survival states: "Never write important code while seriously ill with flu or whatnot. The results would be worse than if you had written the code while drunk. In fact, you may want to give serious consideration to getting drunk before attempting to code while sick." Because what violating 16,472 can get you is shenanigans like committing great laundry lists of files, several times, only to find yourself staring at a red bar and wondering how you got there. (Hot the Red Bar in Cable Street in London, but the red bar that is your test tool saying "your code and/or your specs/tests are busted, bub". That red bar.) This has actually happened to me not once, but twice in the last couple of months.)

At which point, you mutter dark threats under your breath at the imbecile who put you in this position (yourself) and start retracing your steps. Whereupon you find that your last genuine, not-because-things-were-wildly-cached proven-working version was five commits earlier. Five commits that have been pushed to the Master Repo on Github earlier. You then note that each of these touched a dozen files or so (because you don't like seeing "19,281 commits" in the project window when you know you're just getting started), and a couple hours of spelunking (as opposed to caving) leaves you none the wiser. What to do?

The first thing to remember is that there is nothing structurally or necessarily semantically significant about the master branch in git. As far as I can tell, it's merely the name assigned to the first branch created in a new repo, which is traditionally used as the gold-standard branch for the repo. (Create new branch, do your thing, merge back into master. Lather; rinse; repeat. The usual workflow.) On most levels, there's nothing that makes deleting the master branch any different than any other.

Of course, the devil is in the details. You've got to have at least one other surviving branch in the repo, obviously, or everything would go poof! when you killed master. And remote repos, like Github, have the additional detail that specifies HEAD as the default branch on that remote. (Thanks to Jefromi for this explanation on StackOverflow, to another guy's question.) There always has to be a default branch; it's set to master when the repo is created and very rarely touched. But it's nice to know that you can touch, or maul, it when the need arises.

Here's what I did:

1. I went back to that last proven good commit, five commits back, and created a new repair branch from that commit;

2. I scavenged the more mundane parts of what had been done in the commits on master after the branch point, and made (bloody well) sure specs fully covered the code and passed;

3. I methodically added in the code pieces that weren't so mundane, found the bug that had led me astray earlier, and fixed it. This left me with a repair branch that was everything master would have been had it been working.

Now for the seatbelts-and-helmets-decidedly-off part.

4. I verified that I was on the repair branch locally;

5. I deleted the master branch in my local repo

6. I ran git branch master to create a new master branch. (Note that we haven't touched the remote repo yet);

7. I checked out the (new) master branch (which at this point is an exact duplicate of repair, remember);

8. I (temporarily) pushed the repair branch to origin;

9. I used the command git remote set-head origin repair to remove the only link that really mattered to the existing master branch;

10. I deleted the master branch on the remote ("origin") repo as I would any other remote branch;

11. I force-pushed the new master to the remote, using the command git push --force origin master. I needed the --force option to override git's complaint that the remote branch was ahead of the one I was pushing;

12. I ran git remote set-head origin master to restore git's default remote branch to what it had recently been; and

13. I deleted the repair branch from origin

Transmogrification complete, and apparently successful. Even my GUI git client, SourceTree, only paused a second or so before displaying the newly-revised order of things in the repo. It's useful to know that you can trade the safety scissors for a machete when you really do feel the need.

However… so can anyone else with access to the repo. And I can easily see scenarios in a corporate setting where that might be a Very Bad Thing indeed. In a repo used by a sizeable team, with thousands of commits, it wouldn't be all that difficult for a disgruntled or knowingly-soon-to-be-disgruntled team member to write a program that would take the repo back far enough into the mish-mashed past to deter detection, create a clandestine patch to a feature branch that (in the original history) was merged into master some time later, and then walk the commits back onto the repo, including branches, before pushing the (tainted) repo back up to origin. I'd think that your auditing and SCM controls would have to be pretty tight to catch something like that. I'd also think that other DVCSes such as Mercurial or Bazaar would have a much harder time successfully doing what I did, and therefore, would be less likely to such exploitation. That this hasn't been done on a scale wide enough to be well-publicised, I think, speaks quite loudly about the ethics, job satisfaction, and/or laziness of most development-team members.

What do you think?

Tap-Dancing on the Shoulders of Giants, Cont'd

(after Bernard of Chartres)

Some days you wake up feeling old. Some things that happen during the day make you feel much, much older.

In times technologically ancient but historically recent, people wrote computer programs by writing code for virtually every instruction to be executed by that program. Later, as capabilities grew (and we learned better), we used underlying systems and libraries to reduce the amount of new code we had to write for increasingly complex programs. Later still, networks and the Internet augmented the resources we could use in our creations. As anyone who's written desktop applications for a recent major operating system can attest, the complexity of these applications can meet and often exceed what one person can keep in his or her head. Consequently, a high priority became "doing more with less", while still delivering an efficient, usable, correct-enough system.

Recent development, especially Web development, has taken that priority and raised it to an imperative. To be a good developer, you don't just have to be able to code well, you must be able to efficiently find, evaluate, learn, use, and collaborate on further development of these components. Write code once, that's fine. Write code twice, write it again in a more easily reusable form, and rewrite the first two to use the new one. What used to take a team months and thousands of lines of code can now be accomplished by a single developer in days or weeks, with dozens to scores of lines of more readily understandable code — that uses ever-more-complete and -complex underlying libraries.

I was reminded of this yet again by a recent blog post by David Guttman (with accompanying Heroku demo and source code). There's a moderate amount of framework-type boilerplate around it, but the source file I linked does all the heavy lifting.

It's less than fifty source lines of CoffeeScript. Fifty. Lines.

Not since the heyday of APL has a fifty-line program done so much. However, fifty lines of CoffeeScript is far more readily understandable by mere mortals than fifty characters of APL. (Browse this to get an idea of what I'm talking about.)

When I was a wee lad in uni, I wrote code, in ALGOL 68 on a Burroughs B6700, to drive a Tektronix graphics display to produce graphics similar to Guttman's. My program, when complete, nearly filled two 1600-character-per-inch 9-track tapes. (Yes, those erratically whirring reels you see in old newsreels and oddball posts on YouTube.) Each tape stored about 40 megabytes. (Less space than the music file I'm listening to at the moment; about 1/8 of a CD altogether.)

That project took the better part of two school quarters, about four months.

You can still spend months researching if you have to learn some Brave (totally) New Domain and a Strange New Language at the same time, but to actually write a fifty-line program? Days? Hours?

And that brings me to the point of the title; too many of us, not just younger developers, now take these new tools for granted. The problem in doing so, in not understanding how things were done before, blinds us to important details about the now-modern tools we use today. By not having a clear understanding of the problems solved by our tools, by not understanding the tradeoffs and alternatives involved in developing the tools we use every day, it's practically impossible for us to make the most effective use of those tools. And that can have some serious, project-endangering consequences, obvious in hindsight to those who have hit them at least once or twice..

We all stand on the shoulders of giants. Don't get too cocky with your tap-dancing, lest you follow in the shuddering footfalls of one Wile E. Coyote.

It's Not Just "JavaScript" Anymore

Late to the party again, as usual, but…

I've been reading up on ECMAScript 5, the standardised language historically and popularly known as "JavaScript". Even though I'm doing most of my work in CoffeeScript now, ECMAScript is relevant because that's what CoffeeScript compiles down to. (At least, to a 'safe', 'good bits' subset thereof). So here I've gone and mentioned three different names for "very similar" languages in the same paragraph. When I mean one in particular from here out, I'll name it; otherwise Script refers to all three to the best of my knowledge. Anyway...

One of the things I find myself doing, and reading in just about everybody else's code in most languages, is a prudent sanity-check validation before doing something "dangerous" that profoundly changes the state of the system based on the state of various objects within it. A decidedly non-trivial part of that is usually simple validation of individual property values; is the someFutureDate value actually greater than the value for Now? Should we check that every time we're about to make an important decision based on the value of someFutureDate? What a pain…especially knowing that the one time you forget to do that check often turns out to be the root cause of a critical bug?

ECMAScript 5 has completely rethought how properties on objects are implemented. They can be made read-only, non-enumerable, non-modifiable once set. Perhaps more interestingly, they can now be defined either using a simple value as before, or by defining getters and setters, as is done in several other languages. What this buys you, at the cost of a bit of relocated complexity, is the ability to validate assigned values without the assigning code doing anything other than assigning a value to the property. The domain logic relevant to a specific property of an object can now be coupled more tightly to the property and the details less visible or relevant to outside code, in effect creating a conceptual "mini-class" around a property and its getter/setter logic that is transparent to outside code.

This and several other features of ECMAScript 5 now make it easier to write nice, fine-grained, SOLID code in ECMAScript 5 than was previously possible in any Script dialect. Huge win all around.

Which brings up questions. How could support for property descriptors and the like be added to CoffeeScript? Should they be added? Would support for these underlying ECMAScript features require changes to the CoffeeScript compiler itself, or could it be achieved less intrusively?

Stubs Aren't Mocks; BDD Isn't TDD; Which Side(s) Are You On?

I just finished re-reading Martin Fowler's Mocks Aren't Stubs from 2007. I wasn't as experienced then in the various forms of agile development as I am now, so couldn't quite appreciate his perspective until somebody (and I'm sorry I can't find whom) brought up the paper again in a tweet a month or two ago. (Yes, that's how far behind I am; how do you do when you're working 15- to 18-hour days, 6 or 7 days a week for 6 months?)

In particular, the distinctions he draws between "classical" and "mockist" test-driven development (TDD), and then between mockist TDD and behaviour-driven development (BDD) are particularly useful given the successes and challenges of the last dozen or so projects I've been involved with. I wouldn't quite say that many teams are doing it wrong. They/we have been, however, operating on intuition, local folklore and nebulously-understood principles gained through trial-and-error experience. Having a systematic, non-evangelistic, nuts-and-bolts differentiation and exploration of various techniques and processes is (and should be) a basic building block in any practitioner's understanding of his craft.

Put (perhaps too simply), the major distinction between classic and mockist TDD is that one focuses on state while the other focuses on specific, per-entity function; projects that mix the two too freely often come to grief. I believe that projects, especially midsize, greenfield development projects by small or inexperienced teams should pick one approach (classic or mockist TDD, or BDD) and stick with it throughout a single major-release cycle. You may credibly say "we made the wrong choice for this product" after getting an initial, complete version out the door, and you should be able to switch the next full release cycle to a different approach. But if you don't know why you're doing what you're doing, and what the coarse- and fine-grained alternatives are to your current approach, you can't benefit from having made a conscious, rational decision and your project thus can't benefit from that choice.

Anything that gives your team better understanding of what you're doing, why and how will enhance the likelihood of successfully delivering your project and delighting, or at least satisfying, your customers. Even on a hobby project where your customer is…you yourself. Because, after all, your time is worth something to you, isn't it?

Rules can be broken, but there are Consequences. Beware.

If you're in an agile shop (of whatever persuasion), you're quite familiar with a basic, easily justifiable rule:

No line of production code shall be created, modified or deleted in the absence of a failing test case, for which the change is required to make the test case pass.

Various people add extra conditionals to the dictum (my personal favourite is "…to make the test case pass with the minimal, most domain-consistent code changes currently practicable") but, if your shop is (striving towards being) agile, it's very hard to imagine that you're not honouring that basic dictum. Usually in the breach at first, but everybody starts in perfect ignorance, yes?

I (very) recently was working on a bit of code that, for nearly its entire existence over several major iterations, had 100% test coverage (technically, C0 or line coverage) and no known defects in implemented code. It then underwent a short (less than one man-week) burst of "rush" coding aimed at demonstrating a new feature, without the supporting tests having been done beforehand. It then was to be used as the basis for implementing a related set of new features, that would affect and be affected by the state of several software components.

That induced some serious second-guessing. Do we continue mad-hatter hacking, trusting experience and heroic effort to somehow save the day? Do we go back and backfill test coverage, to prove that we understand exactly what we're dealing with and that it works as intended before jumping off into the new features (with or without proper specs/tests up front)? Or do we try to take a middle route, marking the missing test coverage as tech debt that will have to be paid off sometime in The Glorious Future To Come™? The most masochistic of cultists (or perhaps the most serenely confident of infinite schedule and other resources) would pick the first; the "agile" cargo-cultist with an irate manager breathing fire down his neck the second; but the third is the only pragmatic hope for a way forward… as long as development has and trusts in assurances that the debt will be paid in full immediately after the current project-delivery arc (at which time increased revenue should be coming in to pay for the developer time).

The moral of the story is well-known, and has been summed up as "Murphy " (of Murphy's Law fame) "always gets his cut", "payback's a b*tch" and other, more "colourful" metaphors. I prefer the dictum that started this post, perhaps alternately summed up as

Don't point firearms at bits of anatomy that you (or their owner) would mind losing. And, especially, never, ever do it more than once".

Because while even the most dilettante of managers is likely to have heard Fred Brooks' famous "adding people to a late project only makes it later" or his rephrasing as "nine women cannot have a baby in one month", too few know of Steve McConnell's observation (from 1995!) that aggressive schedules are at least equally delay-inducing. If your technical people say that, with the scope currently defined, that something will take N man-weeks, pushing for N/4 is an absolutely fantastic way to turn delivery in N*4 into a major challenge.

Remember, "close" only counts in horseshoes and hand grenades and, even then, only if it's close enough to have the intended effect. Software, like the computers it runs on, is a binary affair; either it works or it doesn't.

'Least Knowledge' May Be More than You Think

I was chatting on IRC the other day and got into a discussion that stuck in the back of my mind until now. This guy was a fresh convert to the ideas of SOLID and to the Law of Demeter (or Principle of Least Knowledge). Now, these are principles I hold at least as strongly as the average experienced developer, and we eventually agreed to disagree over the propriety of code such as a line I just wrote (which in fact prompted this post).

Consider the Ruby code:

    data[:original_content] = article.content.lines.first.rstrip

This says:

1. "take whatever your article is;
2. get whatever its content attribute is or method returns;
3. (assuming that is a String or something that can be treated as one,) get its lines enumeration;
4. get the first entry in that enumeration;
5. (again, assuming it's a String or workalike,) strip any trailing whitespace from that string; and
6. stuff it into the collection data, indexed by the symbol :original_content."

It's not hard to argue that this is a bad code smell, being a long list of assumed return types and so on, except for one thing: in my view, the LoD does not cover sequencing a series of standard library calls.

Look at the code again. The bit that references my code is the fragment

  data[:original_content] = article.content

That fragment is clean; article is an object and content is an attribute of or method on that object. This code assumes that the value of article.content is a standard String. As a standard class, it's virtually guaranteed not to change in backwards-incompatible ways over the useful life of the code. What then? Standard library calls!

1. String#lines returns a standard Enumerator instance;
2. Enumerable#first (via the earlier Enumerator) returns a standard String instance again; and finally
3. String#rstrip cleans off any trailing whitespace.

Nowhere are used any internal APIs that could change. Breaking that one line apart into 5 would produce repetitive, non-semantic, anti-Ruby-best-practices code of the style pandemic on large Java projects. We're in Ruby (and Rails) for a reason: part of that reason is that the language idioms encourage an almost literary expressiveness that makes more explicit the reality that a program is a conversation between the humans developing and maintaining it, that just happens to be executable by a computer. Literate, semantic programming is like behaviour-driven development; once you've wrapped your head around it and see how much better a programmer it makes you, you really don't want to go back to BDUF jousting in Java or PHP.

On further review...

Now, if you argue that the LoD is an aid in reducing coupling between classes, which reduces per-statement complexity, I can see that. But, in the example above, we're back to the fact that there is only one dot before you start jamming together standard library calls. Had I instead written:

content = article.content lines = content.lines first_line = lines.first data[:original_content] = first_line.rstrip

then I'd argue that I've introduced three unnecessary local variables when only one (the second) could foreseeably change. If I later modified the return value from article.content to something other than a Stringlike object, then probably the assignment to lines would have to change. But only that one line — the same level of change that would be required in my current code.

Comments?

If At First You Don't Succeed, Try, Try Again

In a very real sense, that's a big part of what we do in test-driven or behaviour-driven development, isn't it?

It doesn't only apply to classes and projects. Teams work that way, too. You can have someone who, on paper, seems just great to help you out with the cat-herding problem that's a large part of any significant project. The interview is fantastic; he's 110% buzzword-compliant with what (you think) you need The New Hand to be able to pick up and run with. There might be a few minor quirks, niggles; a voice in the back of your mind standing up and clearing its throat obnoxiously, and when that doesn't get your attention, starts screaming loudly enough that surrounding folk reach for ear protection. When that doesn't get your attention — because, after all, you're talking to such a fine candidate — the voice might well be forgiven for giving up in disgust and seeking the farthest corner of your brain to sulk in. And give a good kick to, every now and again.

And then Something happens, usually a string of increasingly serious, increasingly rapidly-occurring Somethings, and you realise that this really isn't working out so well after all. The candidate-turned-colleague may well be a very nice person, but you just can't communicate effectively. Or the "incremental" test that she writes requires changes to a dozen different classes that, in turn, break 15 existing tests. If that doesn't make you go 'hmmm...', then maybe your other team members are concerned about his professionalism: consistently showing up late to meetings; giving a series of increasingly outlandish reasons for not coming into the office and working from home despite having previously promised not to, and so on. Seemingly before you know it, the project is at a virtual standstill, team dysfunction has reached soap-operatic heights, and at least one vital member of the team starts running around and shouting "off with her head!" All of which does absolutely wonderful things to your velocity and project schedule. Or maybe not so wonderful.

And the funny thing is that, at about that time, you remember that voice in the back of your mind that was trying to point out that something was rotten in this particular virtual state of Denmark from the beginning; you were just too optimistic, and likely too desperately stubborn, to recognise it at the time. If you're going to go about hiring someone, have a good rapport with your instincts. Know when to walk away; know when to run.

Long story short, we are looking for an awesome new team member. As you might have gathered from the last few blog posts, we're a Ruby on Rails shop, so you'd imagine that heavy Rails experience would be tops on our list, right? Not so fast, friend. I/we subscribe to the philosophy that source code (and the accompanying artefacts) are part of the rich communication that must continuously thrive between team members, past, present and future. These artifacts are unusual in that they are in fact executable (directly or indirectly) by a computer, but they're primarily a means of communication. As such, ability to communicate effectively in English trumps wizard-level Ruby skills. If we can't communicate effectively, it doesn't matter how good you are, does it?

What we're looking for at the moment can be summarised thusly:

1) Demonstrated professionalism. This does not mean "a long string of successful professional engagements", though obviously that doesn't hurt, especially if they individually lasted more than the sensible minimum. Do you generally keep your word? (Shtuff happens to everybody; nobody's perfect, but overall trends do matter.) Are colleagues secure in the knowledge that you routinely exceed expectations, or are they continuously making contingency plans?

2) Fluent, literate standard English at a level capable of understanding and conveying nuance and detail effectively and efficiently. This would be first on the list, were it not for the series of unfortunate events that led to this post being written at all. It is an absolutely necessary skill for each member of a truly successful team, and a skill that is becoming depressingly rare.

3) Customer-facing development experience, ideally in a high-availability, high-reliability environment. We're building a service that, if we're successful, people are going to "just assume" Works whenever they want it to, like flicking on the lights when you walk into a room at night. Those of us who've been around for a while remember the growing pains that many popular services have gone through; we have no plans whatever to implement a "fail whale".

4) Ruby exposure, with Rails 3 a strong preference. You can fill in along the edges as we go along, but if we can't have an informed discussion of best practices and how to apply them to our project, you're (and we're) hurting. If you're unclear on the concepts of RSpec or REST or Haml or _why, you're going to have to work really, really hard to convince us that we're not all wasting time we can't afford to. And, finally,

5) Deep OO experience in at least two different languages. If you can explain and contrast how you've used different systems' approaches to design by contract, encapsulation, metaprogramming, and so on, and have good ideas on how we can apply that to making our code even more spectacular, rock on. "Why isn't this #1 or #2 on your list," you might ask. If you can convince me why it should be, and why your skills are so obviously better than anybody else I'm likely to have walk in the virtual door... then you've already taken care of the others, anyway.

We're a startup, so we're not made of money. But since we're a startup, money isn't all we have to offer the right person. Know who that might be? Drop me a line at jeff.dickey at theprolog dot NOSPAM com. Those who leave LinkedIn-style "please hire me" comments to this post will be flamed, ridiculed, and disqualified.

Cover Yourself: Toolchains Are Agile, Too

As people who know me professionally and/or read my blog know well, I have been a (raucously) loud evangelist for test-first development (TDD, BDD, Scrum, whatever your flavour) for years now. If I write even an exploratory bit of code and don't have tests in place first, I get very uncomfortable. As complexity increases, without tests (preferably automated, repeatable tests), I argue that I simply can't know what's really going on, because I can't prove it.

A major corollary to this is test coverage reporting. If I can't see what's been tested and what hasn't, then in a very real sense nothing has been, since I can't document/prove what has been and what hasn't. And the better (more productive) teams I've worked in have established, and regularly hit, coverage testing better than 95%, with 100% being a common (and commonly attained) goal. (Edit: Note that this is for C0 and C1 test coverage; tools that cover C3 and C4 are rare to nonexistent in most languages, such as Ruby.)

As you may also know, I've been getting (back) into Ruby development, using Rails 3 on Ruby 1.9. Ruby's long-time de facto standard coverage tool for many years was rcov, which generally worked well. However, Rob Sanheim has stated that "RCov does not, and will not support C based Ruby 1.9.x implementations due to significant changes between 1.8 and 1.9.". He recommends either SimpleCov or CoverMe. Ripping out RCov and replacing it with CoverMeSimpleCov on a test project took all of five minutes and left me with attractive, functional, (so far apparently) quite accurate reports.

One of the basic principles of agile development is that the team must actively embrace constructive change as their project evolves. It's often easy for harried, hurried people to forget that that applies to their tools as much as it does to what they produce using those tools.

Just a thought as my evening winds down.

DRY May Not Be Wet, But It Sure Is Cool

As any developer who values his time, sanity, or amicable relations with teammates who have to maintain his code knows, one of the cardinal rules of modern programming is "Don't Repeat Yourself", or DRY. This is obviously something that should make reading code easier. Obviously, therefore, it should be applied except where the lengths you go to to not repeat yourself make your code harder for someone else to read. The idioms of some languages help this more than those of others.

I was recently browsing through a (rather poor) Ruby programming book when this hit me between the eyes. Consider this ERB example from the book.

Pretty ugly, yes? Part of that ugliness is intrinsic to ERB; which is why people are leaving it in droves.

Compare this exact translation into Haml:

= form_for @product do |f|
  - if @product.errors.any?
    .error_explanation
      %h2= "#{pluralize(@product.errors.count, "error")} prohibited this product from being saved:"

      %ul
        - @product.errors.full_messages.each do |msg|
          %li= msg

  .field
    = f.label :title
    %br/
    = f.text_field :title

  .field
    = f.label :description
    %br/
    = f.text_area :description

  .field
    = f.label :image_url
    %br/
    = f.text_field :image_url

  .field
    = f.label :price
    %br/
    = f.text_field :price

  .actions
    = f.submit

A bit easier to understand, now that we've gotten most of the clutter out of the way, yes? Now, the repeating field definitions stick out like a sore thumb; they're almost the same, but not exactly. We've got three text_fields and one text_area, so a simple loop that just plugs in values won't quite cut it, will it? How about this:

= form_for @product do |f|
  - if @product.errors.any?
    .error_explanation
      %h2= "#{pluralize(@product.errors.count, "error")} prohibited this product from being saved:"

      %ul
        - @product.errors.full_messages.each do |msg|
          %li= msg

  - [:title, :description, :image_url, :price] do |name|
    .field
      = eval("f.label :#{name}")
      %br/
      = eval("f.text_field :#{name}") unless name == :description
      = eval("f.text_area  :#{name}") if     name == :description

  .actions
    = f.submit

If you're coming to Ruby from a PHP background, you've been conditioned not to use eval; there's all sorts of nastiness that could lurk there, especially when code is sloppy. (And PHP code is notorious for its sloppiness.) But in Ruby and Haml, this lets us solve a problem more eloquently than the two or three other possibilities that come to mind. Further, the idiom of the repeated-but-not-repeated text_field/text_area fragment

      = eval("f.text_field :#{name}") unless name == :description
      = eval("f.text_area  :#{name}") if     name == :description

should make it clear to even the most hurried reader that we're dealing with a special case. This is one of those situations where a helper method would be nuclear overkill. (If we were scattering dozens of these all-but-one-fields-is-the-same forms through an app, I'd revisit that stance.)

---

OK, but why blog about this in the first place? Well, the company I'm working with now is looking for a senior Ruby/Rails dev. Most of the CVs we're getting are from mid-level guys (if you're a Rails doyenne (i.e., female), please email!) who have done some Ruby and one or two other languages, generally PHP and Java. What I've learned, on both sides of the table, is that if you've got some experience in several languages that aren't all closely-related to each other, it's a lot easier for you to ramp up on any language you need. Any language will eventually go out of favour. Being able to code idiomatically in anything you need to, won't.

A Blessing is ALWAYS A Curse (and Vice Versa)

I've recently started as Senior Architect at Savant Degrees, a Singapore-based Web consultancy. Quite often, I feel like my job is as much "Senior Curmudgeon" as anything. According to Wiktionary.org, a curmudgeon is "[a]n ill-tempered (and frequently old) person full of stubborn ideas or opinions." I'm undoubtedly the oldest person among any of my colleagues that I've met, and I was told fairly explicitly that I was being hired on the basis of the variety and depth of my experience — i.e., for my ideas and opinions. And, like most people, I think I'm only stubborn when I'm right.

Naturally enough, I'm doing a lot of learning, too. This is my first experience with Groovy, a nicely dynamic language which runs atop the Java VM. Like any language worth its weight, a rich ecosystem is growing around it. Like the ecosystems around most languages, there is at least one ultra-massive tool that takes new folk some time to wrap their minds around. With PHP, for instance, it's the Zend Framework, which arguably makes the C++ Standard Library look minimalistic. With Java, it's pretty much anything having to do with J2EE1 (to which Groovy itself is a reaction).

The biggest tool I've yet stubbed my toe on in the Groovy world is, unquestionably, Gradle2. In the grand Zend tradition, the PDF version of the user guide is nearly 300 pages long and, also like Zend, is considerably out-of-date. It's considerably easier to argue that Gradle justifies the bulk, however, since Gradle is a "let's build anything" build manager/dependency tracker/dessert topping/floor wax. The basics can be understood in half a day, even if you've only nodding familiarity with Groovy3. To truly master it, however, requires significant extra time — as much as you feel you derive benefit from.

And that (finally) brings me to the point of this post. The group I'm working with now has some very talented and moderately experienced developers in it; they're open to new ideas and new ways of doing things, more than is the local cultural norm. In our craft, that's not just a Good Thing™; it's necessary for continuing success4. But that enthusiasm is tempered by a keen consciousness of impending deadlines, particularly the "curse of success;" a month, later this year, when we will be spread extremely thin amongst several important projects. The ways we've done things have been successful, but we see the down-sides of success as well.

So, new ways of doing things, and the hope of new, increasing success. New things like continuous integration using automated builds and unit tests, among other such recently-popular tools and techniques. (They're popular because they generally work better than Ye Olde Ways.) I'm arguing for the team to use tools that will, at the click of a button or the tick of the clock, automatically check out, build, test and package the code and associated resources for our project. The team is new to this, having used "traditional" "waterfall" processes in the past. Knowing one is to be hanged, as Mark Twain said, tends to focus the mind wonderfully; a rational person (or team) will avoid being put in that position. My job, so to speak, is to make the tools that help us cut the hangman's rope, and teach their use. Presenting already-busy people with several hundred pages of documentation is probably not the most effective way to do that.

So, at the moment, the most pressing blessing is that we have such wonderful tools available, with more on the way. Most categories of tools have several at-least-usable alternatives available; it's very easy to go into information overload. When that happens, often you just choose to use what some exemplar, say a tool vendor, use themselves. It becomes tragically ironic when a development group pursuing a more agile process has their agility and effectiveness limited by their "agile" tools. (I don't mean to be commenting on any specific company or team here; I've seen this effect on numerous projects over the years.)

Choosing deliberately increases the odds that you will choose wisely, and those odds will increase along with your experience. Experience is perhaps the ultimate professional blessing-and-curse-combined; "you never learn anything from doing it right the first time." I prefer the Nietzsche:

That which does not kill you, makes you stronger.

We're not dead yet — so we must be strong, yes? Another "stubborn idea and opinion."

Comments?

Footnotes:

1. Now known as JEE, because "Java 2" (1.2) is so last decade; we're on 1.6 now. (Return)

2. No, I haven't looked at Grails yet. Being a close of Ruby on Rails, I expect that to be the proverbial "kitchen sink with subdivisions attached." Fortunately (?), we're a Tomcat shop. (Return)

3. But, of course, experience counts; Gradle rewards your groovy Groovy skills with more efficiently expressive power in your project automation. (Return)

4. There's another curmudgeonly opinion for you. (Return)

Windows? Check. Skype? Check. Quality? Oops…

Several cultures share variations of a proverb which reminds us that you cannot build a house on sand and have it endure the test of time — or even the next tide.

Skype learned that the hard way a couple of weeks ago. Like all peer-to-peer, or P2P, networks, Skype relies on (a varying subset of) its users' systems to handle routine network-housekeeping chores, including routing. Fine; that's the way many applications have done things for more than a decade. Skype have been living unusually close to the edge in two respects: they have an unusually large, widely-distributed number of simultaneous users relative to other P2P networks, and it implicitly trusts the reliability of its Windows-based application and the underlying Windows operating system when performing at any needed scale. This is what sent the waves crashing in upon the sand-borne house.

Microsoft Windows is, inarguably, a wildly successful software system; along with Microsoft's Office set of applications, it is the basis of Microsoft's billions of dollars a year in revenue and profit. But just because something is successful or because it's made by a major corporation doesn't necessarily mean it's well-designed or -built. (See, for example, the Ford Pinto.)

By relying on the (always-questionable) stability and reliability of each of the Windows PCs acting as a "supernode" in its network architecture, Skype was setting itself on a path to eventual inevitable, foreseeable large-scale failure.

Many of the well-known problems with keeping Windows stable and running properly have to do with the ridiculous ease of developing, distributing and using malware; in some countries (such as Singapore), estimated infection rates of Internet-connected Windows PCs routinely hit or exceed 90%. Besides the obvious threat to any "secure" information on those systems (credit card info, login IDs, and so on), those infected Windows systems have a relatively large share of their processing power diverted for the benefit of someone other than the legitimate user or of applications other than those that s/he has authorised.

Another cause of instability and other problems, as with the Skype outage, is that it is very difficult to develop, deploy and maintain even moderately complex applications under Windows with any real means of asserting that the software is reliable and robust. The only way to gain any such real assurance is by testing as many hardware and software combinations with your software as you can, and hoping that you've found all the "most likely" problems. (This is why Microsoft and others employ thousands of software testers, in addition to a massive automated-testing infrastructure.) And, though Microsoft have made some relatively amazing improvements in this area (compare, say, Windows 98 to Windows 7), even they can't catch everything — and their development process and system architecture essentially require them to. Every Windows usee in the last 25 years has seen the "Blue Screen of Death;" that is merely an acknowledgement by Windows that it's completely lost its mind, usually after innumerable random acts of lesser senility.

Skype were bitten by both ends of this issue: a new release of their client application for Windows had defects that were not caught during development or internal testing, and by deploying that defective application on an operating system that provides little-to-no protection against maliciously or inadvertently defective applications, a large "time bomb" started ticking away, unnoticed by all until its spectacular detonation on or around 22 December 2010.

For a Skype client to be a "supernode," it cannot be behind a "firewall" that uses network address translation to isolate its clients from the big, bad Net. (There are ways around that, too, of course; but they're generally very black-hat techniques that don't scale well on a commercial basis.) A client does not have to be running on Windows to be a supernode, but given the demographics of Skype's user base, and the larger PC user base in general, the majority are.

And so, when Skype's defective Windows client ran into mortal difficulties on Windows' defective platform, the ordinary user PCs that were acting as supernodes started dropping like flies. Other Skype clients, as with any proper P2P network, simply rerouted to other supernodes. However, since many (if not most) of the Windows clients were behind NAT firewalls and there were (and are) a relatively small percentage of non-Windows clients available to serve as supernodes, the number of "leaf node" clients chasing the decreasing number of supernodes put the remaining systems under an unsustainable load, and so those Skype clients (with their supernode capabilities) dropped off the network.

Had there been a more heterogenous mix of Skype clients so that those available to perform supernode duties were not almost exclusively Windows PCs, the failure might well have been mitigated if not eliminated altogether. Or, were it practical to develop a (reasonably) guaranteed-stable application on Windows, the defect that initiated the failure cascade might well never have occurred. But by "putting all their eggs in one basket" which was known to have large, gaping holes in it, Skype were guaranteeing that their metaphorical sidewalk would eventually be covered hip-deep in broken eggs.

Monocultures in technology, as in farming, are generally Bad Things; a threat which can attack any member of a monoculture can attack all such members. (Think of the Windows malware-infection rates as an example of this.) Single points of commonality (or of control) are always eventual single points of failure.

The Skype failure could well have been a reminder of these long-known basic truths. The commercial interests involved, however, ensure that such a lesson will go unheeded. It will therefore be repeated; very possibly not with Skype, but with some other aspect of our grossly defective information-technology ecosystem. Think of it as "Unsafe at Any Speed" meeting Groundhog Day until something is done to prevent yet another China Syndrome.

Keywords (aka "tags") ARE NOT Structure

And I understand that's the whole point. But sometimes, you're dealing with a concept or with data that is most naturally and properly organised in a structured hierarchy. While keywords1 are a very convenient tool for filtering ad hoc queries2, they're a poor stand-in for truly structured information.

One (possibly trivial) case in point: managing your browser bookmarks. What I want to be able to do is to organise bookmarks3 and be able to view them in the hierarchy and order that they will be in in the browser "Bookmarks" menu. That way, regardless of which file format4 is being imported from or exported to, while you're using the bookmark manager, you have an accurate view of how the ("folder"-based) organisation of the bookmarks will appear in your browser.

So why not, as several managers do, use keywords to imply structure? Let's take a look at that idea, with a scenario. Let's say you have 4,000-5,000 bookmarks in a fairly deep and wide hierarchy of about 800 folders, some nested five and six levels from the root. While that may sound like a 1996-era Yahoo!, that's roughly what I deal with, and I know quite a few long-time Web users who have significantly more.

OK, so you're going to do your internal organisation using keywords. You're a Web developer, so you might have a few bookmarks in a folder with the path Development/Web Development/PHP/Frameworks/Agavi. Each bookmark's reference in the bookmark-manager software would have the keywords Development, Web Development, PHP, and so on. It clutters up the keyword list, and it necessarily complicates the display and editing of those bookmarks within the software. It also means that more data is stored for each and every bookmark. Disk drives are cheaper than they have ever before been, but they're not free.

And, what's worse, if you accidentally (or otherwise) delete one of the keywords from one of the bookmarks, when you export your list again, that bookmark will be in a brand-new folder somewhere else in the hierarchy. Finding and fixing that, if you have a reasonably complex bookmark list, can be difficult. If you don't find and fix it, of course, then it won't be there when you go looking in what should be the right place.

And finally, you're twisting the whole structure of the way people (and browsers) view bookmarks; you're distorting the semantics of the system. This may seem a fine point in today's "get-it-done-yesterday!" way of doing everything. It's not. By not having a clean, self-consistent, efficient conceptual organisation for your software, you make it more difficult to learn, more difficult to use, more complex to develop, more expensive to maintain, and more likely to have more severe problems over the lifetime of the software5.

One final mental exercise. Think of that CD collection I alluded to at the beginning of this post, that's now stored on your computer using whatever music player you like. If I delete the Wakeup Music keyword from the track info for "Come Fly With Me," and then I play the album, nothing has changed. That's metadata. But if I change the bit-rate of the recording, or move the track into the folder for the album "My Way," that's different, isn't it? That's a change to the data, or to the organisation of the data. One is a matter of choice; the other is a matter of history.

By conflating data and metadata (literally, "data about data,") we do a disservice to both.

And now, if either of you happen to know of a (preferably not browser-based) bookmark manager that does things The Right Way™ — or reasonably close to it — I'm all ears.

Thanks for reading, and replying.

Footnotes

1. Though they're popularly called "tags" by all the Kool Kids™ these days, and "Labels" here on Blogger, I'm going to refer to the data items as "keywords" because that's what the library- and information-sciences worlds call them, as well as any programmer who started his Craft before, oh, about 1995. (Return)

2. "List all my Frank Sinatra album CD titles that I've tagged as Wakeup Music." (Return)

3. Each "bookmark" should store URL and page title at an absolute minimum; keywords are good to have; additional metadata optional. (Return)

4. XBEL, OPML, LSMFTML, or whatever the new "standard" is next month. (This would be a good feature to think about some sort of plugin architecture for.) (Return)

5. Look at practically any Windows program for examples of this, or, better, any three Windows programs together. It's not that they don't work, but after a few hours of use you can think of several ways that each could be more consistent with itself and with others. (Return)

IE9 and Standards? Nothing to see here, folks; move along...

This is a re-post of a comment I left to the post "HTML5 Support in Internet Explorer 9 on Louis Lazaris' Impressive Webs blog (which I highly recommend for people working in or interested in Web development). I've slightly edited a few places for clarity. The comments that had been left by people were almost universally complaining about Microsoft in general and IE's continuing history of imperfect-at-best compliance with either-you-do-or-you-might-as-well-not-try standards. Having worked in and with Microsoft on several occasions, and being active in a number of open-source projects besides, I have a slightly different view, as I state below. Thanks for reading.

---

Actually, I'm amazed that it's progressed as much as it has.

Let me be clear; I'm not in any way praising IE; the industry as a whole needs to bury it — decisively, irretrievably and imminently. We as development professionals need to have the professional self-respect to tell current and potential clients, "We develop to current Web standards that are supported across various browsers and platforms. To the degree that any particular release of Microsoft Internet Exploder supports the markup, styling and behaviour of any particular site, that's well and good. However, without monumental additional time and budget resources, no attempt will be made to provide IE support when those resources could instead be used to improve the experience for users of modern browsers."

I firmly believe that IE hasn't progressed as far as, say, Chromium, can be laid firmly at the feet of Microsoft's existing code base. Microsoft's developers and managers are fully experienced with the reality that complete green-field rewrites of existing projects almost never succeed. They've got a code base where the list of major Windows modules and subsystems that do not have dependency relationships with IE could be read aloud comfortably in a single breath. That was done initially by choice; now, it doesn't matter how well-meaning the intentions or competent the team are, they have to live with the codebase they have. It's all legacy code. The developers at Microsoft are (with rare exception) not morons, but living in the sewer while they're trying to make something that can stand next to the shiny new browsers next door has to be a psychologically toxic exercise. Their baby is blowing up the Web left and right; they know it, and they know they can't do a damned thing about it without staging a coup d'état, replacing dozens of levels of management and senior executives and fundamentally changing the culture of the organisation. Don't hold your breath waiting for that to happen.

That isn't sour grapes or a diss against the IE developers; it's simple reality. Microsoft do some amazing things — just not so much for (or on) Windows. Unfortunately for all of us, Windows and Office for Windows are the herds of cash cows for Microsoft, and anything that could be seen as even potentially disrupting that would get shot down faster than you can say S-E-C; the investors would never stand for it. And, with the system and the rules the way they are, they'd be perfectly right. Innovation isn't as important to the bean-counters (or the regulators) as "maintaining and enhancing shareholder value," and MSFT have had enough problems with that lately. (Just compare their share values to, say, AAPL over the last ten years.) Doing anything "risky" is simply politically (and legally, really) impossible.

So, no matter how many times the IE team jump up and down and say "look at all our neat new features," without mentioning the standard features left unimplemented because they pulled off numerous miracles just making what they have work, the response has to be "nothing to see here, folks; move along."

And move along we shall.