Bootstrap and alternates are GREAT! Except when they lead to...

...divitis like this (in a Slim template):

Yes, that's eleven levels of indentation for the link within the .list-group-item list item. Can *you* glance at that and grok it in reasonable fullness without spending ten minutes rebuilding the DOM in your head? I can't, and I wrote the sucker yesterday.

What's the problem? Nesting styles is the problem. Using line 23 as an example, we have, in reverse order:

• an anchor link;
  • within a list-item tag;
    • within a loop (that arguably doesn't count, but does kick the indentation in a level);
      • within an unordered list;
        • within a .row div;
          • within a .col-md-10 div;
            • within another .row div;
              • within a .col-md-12 div
                • within an outermost, containing .row div.

Got all that? At least we're using something like Slim and not, $DEITY forfend I'd need to use something that looked more like HTML soup, like ERB.

Why bother? Because, as far as I can tell, that's what it takes to work with nested columns in Bootstrap. Everything's within an outermost row, in which you define columns; within any given column, you can use another row to break the containing column into sub-columns. Remember when, in ancientest days of the Web (circa 2001 at latest), we started telling people not to use tables for layout because it led to inscrutably crufty hairballs? They're baaaaaack!

There's got to be a better way. Isn't there? Isn't there?!?!?

In other news...

By this time next week, after over ten years of blogging on Blogger.com, I expect to have the process of moving my blogs (and my Tumblr, and many of my G+ posts well underway, if not yet completed. Why? I'd like to

1. Reduce my Google footprint or, more precisely, the footprint GOOG leaves on me;
2. Be able to point people to one place where they can see all my non-code online writing;
3. Use a markup language that's less actively hostile to writing than HTML is; with Markdown and Jekyll on Github Pages, I get that;
4. Have more control over that "one place" than any of the Google-based "places" presently give me; and
5. Be able to embed Gists or other bits of code in my writing and have them look better than the embedded Gist at the top of this post.

It's not for everybody. But if you've been using Blogger in HTML-editing (rather than WYSIWYG) mode, as I have since the beginning, you can certainly learn (and benefit from and enjoy) Markdown and Jekyll.

Rails, Circular Logic and Incremental Big Bang Architecture

(Initially published on my Tumblr, which I'm finding increasingly usable. Thoughts on Tumblr vs Blogger.com?)

---

Rails is a fantastic solution to a well-understood class of problem: the CRUD-oriented (or REST-oriented) query/report/update Web application, circa 2004–2006. Its philosophy of curation over configuration, despite having been likened to the report-producing languages of old (such as IBM RPG II, means that you really only have two things to worry about: your domain business logic and page transitions (which is another way of saying “your domain business logic, as it applies to presentation”).

And yet, the farther your app gets from that sweet spot,, either by decoupling the UI and using a fancy JavaScript framework; by needing to support user inputs other than typing into form fields and clicking buttons; or even the distance imposed by being nearly ten years on from the “sweet spot” and so many tools and other bits of code have changed around it — the harder it gets to do what you want to do in The Rails Way. Lots of people much smarter than me, from Robert C “Uncle Bob” Martin to Jim Weirich to Sandi Metz to Matt Wynne and a host of others, have spent many person-months thinking about, grappling with and writing about how to mitigate Rails, the knots it ties you into and the practices that much of the rest of our craft has built consensus on as being essential for good practice, that you can’t use in Rails without superhuman effort. Because the apps we’re building now, even the relatively simple ones, aren’t largely analogous to Basecamp (the application from which Rails was originally extracted).

So here’s the problem I’m facing, and the problem you’ve more than likely faced if you’ve done a non-trivial app in Rails.

If you’ve done any medium-to-large-scale development in other systems, especially in a decent OO language, one of your high priorities is going to be separating your domain logic from your “infrastructure” logic, including layering your presentational logic related to your data models separately from the data models themselves. Small is beautiful, and Ruby has some great tools to let you do that, including in particular the whole universe of Gems. Rails, however, makes it bloody hard to build what the rest of the craft would view as modular apps that follow SOLID principles. The retort to that is generally along the lines of “Rails is omakase” — which is fine as long as your app is living within that circa 2004–2006 sweet spot I mentioned earlier. The clock in the upper-right corner of my display here informs me that today is Saturday 7 June 2014 as I write this. Oops.

So we have great ideas like Hexagonal Rails , Architecture: The Lost Years and so on, that showed ways, with varying degrees of universality, to “get a handle on” the application/system you were developing.

But, can you answer what your application does? Not “what is its data model”, “what are the different types of users and permissions”, but a pithy sentence or two that gets across the idea of what makes your application different than every other CRUDtastic app out there, Rails or otherwise. Because if you can’t, none of this Rails-survival lore will help you.

And, equally importantly, as Jim Weirich points out in his Cincy.rb talk [Decoupling from Rails](http://youtu.be/tg5RFeSfBM4), is the question of “has this ever been successfully done before? If so, how? If not, what’s been the stumbling block that we have to stand back and work around using something else?” Because, in all fairness to Uncle Bob, if there aren’t examples of his built-on-Rails architecture in the wild for people to poke at, projector-ware can be inspirational but not terribly practical. For instance, as Jim points out at about 10 minutes into his presentation, Uncle Bob’s architecture shows where Rails controllers and views go, but ActiveRecord “models” don’t seem to be there at all*. Separating domain logic from persistence just happens to be one of the hardest nuts to crack in a Rails app to begin with. Hmmm.

Is the solution more decoupling? Jim spends the rest of that talk isolating and extracting what several teams publish as “interactors”, or support for what I call in my own apps “domain service objects” or DSOs. These help, a lot; having “skinny” controllers that don’t really know anything at all about your app beyond how to hand data off to classes that do lets you refactor your business rules efficiently by putting their control logic all in one place separate from Rails. Similarly, implementing façades or “decorators” with presentational logic around your models, as with Draper and others, helps you write clean, business-logic-free view templates.

But you’ve still got ActiveRecord to deal with in your models, and it leads you down the garden path and over the cliff with things like, oh, relations. When you have, say, a Blog model that has_many :posts and a Post model that belongs_to :blog, what you’ve got is a circular dependency; you can’t make changes to your Posts with certainty that those changes are isolated from your Blog, or vice versa, because each depends on the other. There have been numerous serious attempts to effect such a decoupling; perhaps the most widely known is Ruby Object Manager, whose Version 2 has been in the works for over two years now. It’s led by one of the greats in the Ruby community, Piotr Solnica, but it’s a Hard Problem. Especially when coexisting with Rails. People even resort to uncomfortable hacks like this to Get Things Done. There has to be a better way, and monorail (traditional Rails all-in-one non-modular app) is not that way.

And there are other, seemingly orthogonal issues to deal with. Authentication? Authorisation? Several solutions that are widely used exist for these in the “traditional” Rails “monorail” application style; how will adapting your architecture affect these, and how will you implement them?

Anybody?

Why is "doing things right" a "thing"? Whither professionalism?

Over on the Reddit r/programming subreddit, /u/jfalvarez posted a link to Tom Stuart's great presentation at the Scottish Ruby Conference 2014, entitled "The DHH Problem".

If you're involved with Ruby or Web development in any way, it's well worth three and a half minutes of your time.

On the Reddit post where Mr Alvarez posted the link to the video, another Redditor, dventimi, asked

Why is this a thing?

I disagree with the later poster who snarked "People like gossip." It's much, much more important than that.

What follows is the content of my reply to "why is this a thing?" I'd love to have a discussion with you about it, either here or on the original post.

---

Because people are finally waking up to the fact that the coding style and philosophy espoused by DHH are disastrous in non-toy applications maintained over any significant period of time.

That this should have been obvious to all, but wasn't discussed as such, once Twitter, after having conflated Rails with Ruby as a whole, left Ruby because they couldn't make Rails do what they needed to no matter how many times DHH came onsite and insulted their intelligence, speaks eloquently to how broken things are.

Most master software developers I've met and worked with have been mediocre-at-best promoters, including self-promoters. DHH has the opposite problem: his initial extraction of Rails from Basecamp's codebase "worked", for apps very much like Basecamp at the time. His fixed, what some have called "trolling", attitudes and the personality they describe, prevent him from seeing things from any other perspective or from growing his skills beyond a certain point. The echo chamber of acolytes and zealots his sublime self-promotional talents have allowed him to accrete around himself "ensure" that he never has to.

That's a problem for the rest of us, who have deadlines, work to finish within those deadlines, and paycheques we'd like to be able to cash that we won't get unless we meet those deadlines with that work. Especially for those of us who wish to improve our craft and, in whatever way we can, nudge ourselves and others towards helping the craft of software development more closely approximate a true engineering discipline.

Paying excessive heed to the opinions-masquerading-as-principles brayed by DHH, and twisting ourselves to fit inside The Rails Way, actively impede all of the above tasks. Tom Stuart was absolutely right to thank DHH for producing Rails and promoting it sufficiently that the rest of us get to write Ruby for a living…just before tearing DHH a new one, in the polite, discreet way only a well-educated Brit can, for getting so much of Rails and how software should be written so disastrously wrong.

Thank you, Tom Stuart. An entire industry may well be in your debt.

Rails is Rails is NOT Rails. Oh, my aching head.

For a couple of decades, I was into Linux. One of the ways you could tell good distributions and (especially) good documentation for those distributions is that they were (usually) careful to make notes like

This is a new feature as of kernel 2.6, first introduced in QuuxLinux in Version 6.0.

or

That API changed in kernel 2.4 and again in kernel 3.0, first introduced in QuuxLinux in Versions 2.0 and 11.0 respectively.

Now I'm a Ruby and (pro tem) Rails developer, and all that is old is new again, and you don't always know it.

I just spent a day and a half of highly-energetic tail-chasing, working at getting my code and specs for (what is so far) a "toy" demo project to understand Rails routing when namespaces are introduced. Namespaces, which are (as far as I can tell) functionally equivalent to Ruby modules, let you provide an additional level of separation between different parts of your app. They're also practically essential for writing Ruby Gems of any complexity, which is a basic requirement for "Vendor Everything" — an important tool in structuring your apps for maintainability and reusability. So "supporting namespaces" was an essential, if not universally obvious, part of the exploration that I'd embarked upon.

As I write in the project README, I want to "namespace everything". I also write "no code in the absence of a failing spec", so obviously my specs need to work properly with namespaces. Believing in six impossible things before breakfast is trivial in comparison, if you've never done it before.

Well, how better to learn than with a demo project? And how better for my current and future employers to judge my progress and reasoning than by making such (non-proprietary) explorations in the full glare of public scrutiny and ridicule, à la GitHub?

After accomplishing this first tiny step of The Mission, I'm even more convinced than ever that part of what makes Rails development "interesting" is that there's such a mix of information about Rails 2.3, Rails 3.x and Rails 4 out in the wild, treated as though they are slight variations on a common theme, rather than three different products unified mainly by a shared product name.

For those who care, the initial plea for help was on Reddit's /r/ruby sub-Reddit. Initially I'd written up this Gist to document the files I was using and the output I was getting. After reading /u/Nitrodist's response, I made some changes, as reflected in this Gist. Finally, after further prodding from Nitrodist and others, and intensive BFI, I determined the two changes needed to my original code. Documented in this final Gist, and pulling together information from several sources referencing different versions of Rails, this *early step* in the exploration has been completed — with several times the expected effort.

Things have to get better, don't they? Only if you're not aware that Captain Edward A Murphy was a raving optimist.

Let's Stop Pretending that Source Control is "Optional but Recommended"

Essentially all of us who make our living from the craft of software development, early in our careers, have had the experience of losing source code that we couldn't get back. Oh, we could type what we remembered into a file of the same name, but it has important differences that we will discover as we work with it. (There is no such thing as eidetic, or photographic, memory).

This has been recognised as a sufficiently universal phenomenon that most developers and managers I have known in the lsat 10-15 years (at least) use use of a versioning system as a filter against dangerous dilettantes; if someone claims to have developed "the next VisiCalc for Linux" without using one, the file containing that person's "CV" may be safely deleted (and do remember to Empty the Trash).

So why on $DEITY's green Earth do we still see well-meaning tutorials on blogs the Internet over which include a step titled "Setup source control (Optional but recommended)"? The particular tutorial that drew my ire this morning even went out of its way to note that you will need to use a "text editor of your choice" with the author noting that "I use Emacs". (Subject and direct object inverted in the original sentence, but that's another rant.)

Don't

DO

that!!

The person reading your tutorial is probably going to be someone with strong continuing economic motivation to improve his software skills; he's reading your tutorial on the off chance that you'll actually teach him something useful. Seeing those three words ("optional but recommended") tells him, almost literally, that you don't really take him seriously. Your half-dozen other readers are going to be (would-be) apprentices in the craft, just learning their way around; they can and, based on experience, will take those three words as permission to just blaze on ahead and, when something goes pear-shaped, to start all over. Maybe you think "nobody I know would take it that way", or even "how could anybody who can read this take it that way", but The Voice of Experience™ is here to tell you that this Internet thingie is a global phenomenon. People from all walks of life, with every language, technical and educational level variation imaginable, can wake up one fine morning and say "I'm going to learn something new, using the Internet", and set out to do just that. That's a feature, not a bug.

I can hear you sputtering "but I didn't want to have to cover how to set up a VCS or use it in the tutorial's workflow; I just wanted to teach people how to use batman.js". Fair enough, and a noble goal; I don't mean to dissuade you (it's actually a pretty good tutorial, by the way). Might I suggest adding to the bullet list under "Guide Assumptions" a new bullet with text approximating

• A version control system of your choice (I use Foo)

with an appropriate value of Foo. We don't care whether the Genteel Reader uses git, Bazaar, sccs, or Burt Wonderstone's Incredible Magical VCS; we can merely assume that if he and we are worth each other's time, he's using something; we only need that little extra bit of reinforcement. (Though some choices on that list might be cause for grave concern.)

One of the ways in which the craft of software development needs to change if it's ever going to become a professional engineering discipline is that we're going to have to hammer out and apply a shared set of professional ethics; the closest we've got now is various grab-bag lists of "best practices". I'd argue that "optional but recommended" source control is on the wrong side of one of the lines we need to draw.

Trust, Safety, Sandi Metz and A Proverb Walked Into A Bar...

Rereading POODR again. I love the bit in Chapter 6 where she talks about using a post_initialize method to decouple subclasses from having to remember to call super in an initialize method override. That's the sort of thing I remember making work in UCSD Pascal back when it wanted to be the Next Hot Language™, and in a few languages since.

But one question I never found an acceptable answer to was, how do I prevent myself from forgetting to call super (or its equivalent) in a hierarchy with multiple levels? Sure, the base class implements a do-nothing post_initialize-type method, so that every single subclass anywhere in the hierarchy can "kick it upstairs" and things will Just Work.

But at some point, you have to trust yourself, your team, your successors who grapple with your code long after you are gone. Comprehensive tests will catch the fallout from a forgotten upstairs kick and give you a lovely red bar — or will show you that the parent's chained behaviour you thought you needed was either frivolous or incorrectly verified. (Or both. Shtuff happens; nobody ever went broke underestimating the degree to which developers can fool ourselves.) Whatever twisty little maze of keystrokes leads to you that point, you've discovered a refactoring opportunity. Enjoy your Homer Simpson Moment, make the appropriate changes, verify that the tests still pass, and move on.

Доверяй, но проверяй.

Going Forward; Avoiding the Cliff While Looking for Cliff's Notes

Over on the Ruby sub-Reddit, I got into a discussion (mostly with Jelly_Jim, the OP, and with realnti) about the conflation in far too many minds of Ruby, the programming language, with Ruby on Rails, the famously-"opinionated" Web application framework. While pontificating on Jelly_Jim's original question, I described my frustration-bordering-on-antipathy with the traditional (known as "The Rails Way") application structure; listed a few presentations, books and blog posts that have influenced me greatly, and described my own take on the Kübler-Ross model for Ruby-based (and at least initially, largely Rails-based) Web development.

That discussion, over the course of roughly a day, helped me figure out how our app is going to survive the expected Slashdot effect, transitioning from a pretty traditional, megalithic Rails app to an app that uses Rails components in a sharply different, more recognisably "object-oriented", service-based architecture.

Leaving aside my more prosaic experiential difficulties with Rails (I really loathe ActiveRecord as commonly used), and taking into account the "Uncle" Bob Martin keynote, the various blog posts I referenced in the original Reddit post, and a couple of books I mentioned in a later reply, I think I've possibly hit upon a way to get where I want us to go.

That's the easy part, of course; it only took me never-mind-how-many months. The "interesting" part is going to be getting there — understaffed, on a shoestring budget for the moment even by startup standards (another problem I'm working), and working a schedule for two and a half years that would land a healthy twenty-year-old in hospital by now. Comments/suggestions are, as always, greatly appreciated.

---

Basic Architectural Principles

• Loosely-coupled, granular architecture FTW. That means, inter alia,
  • Nearly the entire app to be packaged as Gems used by an otherwise minimal application;
  • Rails mountable engines to package assets (particularly CoffeeScript/JavaScript) and major components;
  • Plan and design for updating and provisioning components which communicate across natural architectural seams.
• Hexagonal, "ports-and-adapters", or "clean" conceptual architecture; all dependencies point inward. Your domain entities, at the centre of the system, will change less than details like the delivery mechanism (Web UI, phone app, etc) or the database (which the domain shouldn't even be able to prove exists as such).
• By adopting a heavily job-story-oriented view of the workflows; with the use of online tools like Trello, WebSequenceDiagrams.com; and with supporting code tools like ActiveInteraction and so on, we should be a lot more productive and a lot more Agile than we have been during our forty (calendar time: two) years lost in the wilderness.
• And oh, yeah, just to keep things interesting: we've got to keep regularly demonstrating forward user-visible progress, for all the obvious reasons and a couple I really can't go into here.

---

How We Get There, v. 28.0

First, since the major focus of late has been on some unique (and not-so-unique) CoffeeScript code, start there. Separate the existing Script code out into four engine-based Gems, following/adapting Derek Prior's example. These levels are, from conceptual bottom to top:

1. A "gateway" CoffeeScript class we've written as a poor man's RequireJS, that gives us just enough pseudo-dynamic referencing of data/code without changing anything the Rails asset pipeline assumes;
2. Our directly DOM-aware Script code, as one or possibly two Gems (one is a major crown jewel; we might decide to split off utility/support code into a separate Gem, so as not to touch one when maintaining the other);
3. The code proxying our internally-used message queue (decoupling "FTW", remember?) and the Ajax code to communicate with the Rails back end; and
4. The code implementing various domain/service objects sitting atop the Script stack; one Gem per service.

Next, adapt the relatively-straightforward (as yet) Ruby/Rails code to honour the architectural principles listed earlier. This is where tools like ActiveInteraction pull their weight (and solve several existing pain points in the process).

Just to make sure we really do understand where we're going with this, take our first user story-specific code (including interactor) and package it up as a Gem that should then Just Work as in the previous step. Start thinking about whether we really want traditionally-packaged Gems or unbuilt dependencies that remain part of the single app repository.

Proceed to reimplement and repackage the (again, relatively few) job stories already existing as was done in the preceding step.

Start knocking down the remaining Trello cards as we complete our app.

We've been saying since the Very Beginning that anything that we were building that wasn't inextricably tied to proprietary, commercially-valuable code (or that gives too much of a hint into any unique details it may have) "should be" openly released, but hadn't yet figured out a feasible, cleanly reusable way to do that. If we've done our jobs properly up to this point, we have that way now.

Push the "deploy and launch" Big Green Button. We've bloody well earned it by now.

---

Any Suggestions or Comments?

Clean Code. "I do not think that word means what you think it means."

From my copy of Clean Code: A Handbook of Agile Software Craftsmanship (the paper copy of which (barely) predates my work in Rails. The highlighted paragraph is what I'd highlighted in the paper copy:

Active Record

Active Records are special forms of DTOs. They are data structures with public (or bean- accessed) variables; but they typically have navigational methods like save and find. Typically these Active Records are direct translations from database tables, or other data sources.

Unfortunately we often find that developers try to treat these data structures as though they were objects by putting business rule methods in them. This is awkward because it creates a hybrid between a data structure and an object.

The solution, of course, is to treat the Active Record as a data structure and to create separate objects that contain the business rules and that hide their internal data (which are probably just instances of the Active Record).

I've been increasingly uncomfortable with the way Rails encourages people to conflate the domain model objects with persistence in Rails' core component, ActiveRecord. Running across this bit again makes me think of Inigo Montoya from The Princess Bride: You keep using that word. I do not think it means what you think it means. Judging from my own projects and those I've participated in and observed on Github, the fact that ActiveRecord does not properly implement or encourage the Active Record pattern is a primary root cause of many bugs and much confusion and heartburn.

Happy New Year.

Trust, Tools, and Tripping Over Shoelaces While Dancing

On the shoulders of giants, that is.

While "a poor craftsman blames his tools", sometimes those tools lead you down the garden path to such a degree that you don't realise that you're doing a Wile E Coyote impersonation until about the third bounce off the canyon floor. For your amusement and edification, if not abject stupefaction, I present a subset of a passing spec file. This is CoffeeScript, using teaspoon and mocha for specs. It's deceptively straightforward, IMAO:

#= require 'gateway'

describe 'Gateway class', ->

  beforeEach ->
    @subject = new window.meldd.Gateway()
    @name = 'foo'
    @value = $('body')

  # other specs elided...

  describe 'has a "useGroup" method that', ->

    it 'takes one parameter', ->
      expect(@subject.useGroup).to.have.length 1

    describe 'when called using', ->

      it 'a nonexistent group name, returns an empty object hash', ->
        group = @subject.useGroup 'not found'
        expect(group).to.be.an 'object'
        expect(group).to.be.empty()

      describe 'a group with one key, returns an object', ->

        beforeEach ->
          @subject.register 'foo', 'bar', 'group1'
          @group = @subject.useGroup 'group1'

        it 'hash', ->
          expect(@group).to.be.an 'object'

        it 'with the correct single item name as the only key', ->
          expect(@subject.groups['group1'][0]).to == 'bar'
          expect(Object.keys(@group)).to == ['foo']

        it 'with the correct item value as the only value', ->
          expect(@group['foo']).to == 'bar'

      describe 'two groups with one identical key, returns', ->

        beforeEach ->
          @subject.register 'this', @, 'group1', 'group2'
          @group1 = @subject.useGroup 'group1'
          @group2 = @subject.useGroup 'group2'

        it 'the identical object in both groups', ->
          # equality to avoid "Converting circular structure to JSON" error
          expect(@group1['this']).to == @
          expect(@group2['this']).to == @
          # identity; they're the exact same object instance
          expect(@group1['this']).to.be @group2['this']

Pretty straightforward, and both of you Scripters reading this can probably code up a proper Gateway.useGroup method without knowing much about the rest of the class. I'd bet the rent that your method would not closely resemble this:

  useGroup: (param) ->
    {}

And yet...

What. The. Fox?!?

Buy Shares in the Garden Path, friends; it's a wonder!

Not giving actual securities advice, of course; merely commenting on the metaphorical "garden path" one is "led down" by seemingly trustworthy sources. The next thing you know, it's three minutes since you started falling; you've no idea where "bottom" is and even less idea how you got there. (For the record, at three minutes, were it not for atmospheric drag slowing you down, you'd have been falling for ~158,760 meters and have an instantaneous speed of ~1764 m/s. Happy landings!)

All right, what am I blathering on about? It's really simple. (Until you fall off that cliff.) See here…

The problem is actually documented rather clearly in the Position section of the W3C "DOM Level 2 Traversal Range" spec, and had I read that instead of relying on the documentation of the jQuery++ class that wraps it, I'd have saved several hours and $DEITY-knows-how-many litres of stomach acid. According to the spec,

The offset within the node is called the offset of the boundary-point and its position. If the container is [any of 5 different types of] node, the offset is between its child nodes. If the container is a CharacterData, Comment or ProcessingInstruction node, the offset is between the 16-bit units of the UTF-16 encoded string contained by it.

(Emphasis mine.)

The jQuery++ documentation gives an example that uses a single element containing a single text node. No mention is made of the (sensible once you figure it out) distinction between offsets in a text node vs. offsets in an element. And attempting to adapt their example to your markup, which is unlikely to be so trivially simple, is bound to lead to confusion unless you know the answer to The Riddle of the Magical Redefining Offset™.

Now you do, and you can be about your work a great deal more quickly than I.

So how do I do what I set out to do, which is to build up an HTML fragment matching selected text on a page? With great and grandiose ceremony, alas.

1. Get the active Range for the block-level element containing my content. That gives me the starting and ending offsets (child nodes of that outer block) corresponding to the child nodes that the selection overlaps.
2. If that outermost range spans multiple child nodes
   1. walk down the first selected node and its descendants, until we find the actual text node containing the start of the selection;
   2. Add that text and the trailing child nodes, if any, of the element node containing that text node to a buffer;
   3. Iterate for that element node's parent element node's trailing child nodes, and on again until we've walked back up to the outermost content area;
   4. Add the entire markup of each succeeding top-level child node up to but not including the block identified by that initial ending offset;
   5. Descend into the top-level ending node and its descendants, adding markup of each node until we reach the actual node containing the endpoint of the selection;
   6. Add the selected text fragment from the end point text node to the buffer. Done.
3. If that outermost range has a single child node (the starting and ending offsets are the same)
   1. walk down the selected node and its descendants, until we find the actual text node containing the start of the selection;
   2. Add that text and the trailing child nodes, if any, of the element node containing that text node to a buffer;
   3. Iterate for that element node's parent element node's trailing child nodes, and on again until we find a node containing the endpoint node (or that node itself);
   4. Descend into the top-level ending node and its descendants, adding markup of each node until we reach the actual node containing the endpoint of the selection;
   5. Add the selected text fragment from the end point text node to the buffer. Done.

Oh, my aching head.

If anybody has any better ideas, I'd love to hear them. Oh, did I mention that this is in CoffeeScript/JavaScript in the browser, so we don't have any fancy tools like Nokogiri, which I've previously described as "the Swiss Army Ginsu Chainsaw for parsing markup". With tooling like that, this little exercise would be over and done with in an hour. Building a no doubt buggy improper subset of its functionality, in Script, has taken days. Plural, and never mind how plural. If I were a drinking man, there'd be a crate of top-shelf Scotch in my immediate future.

No, You Can't Always Get What You Want...

…but as Mick Jagger and Keith Richards remind us, if you try "sometimes you get what you need". Any software developer lives with that on a daily basis; the implications of that are one of the major things that separate our craft from the profession of engineering. I had several of those "implications" blow up in my face this week as I was working on (what else?) a Rails app.

Or, more properly, an app that uses Rails. Because what the majority of my time has been spent with for the last few weeks has not been Rails, or even Ruby, but trying to get the front-end features that have to be implemented in JavaScript (or CoffeeScript or another language that compiles to JavaScript) (generically here, Script) because they're dependent on user-agent and DOM events. The last time I went down this road, I wound up with an app that had five lines of CoffeeScript for every two of Ruby; this one is well along that path. That's relevant; I suspect that there are far more non-trivial apps that use Rails in production than there are Rails apps; for anything beyond a simple pseudo-blog or relatively straightforward database front end, you sometimes wonder what the early Rails core team was thinking when they established some of these "conventions" that trump "configuration". Granted, there are a lot of people using that "sweet spot" for Rails app development, and more power to them, but I'd bet heavily that there are at least as many who started out trying to do an app that uses Rails and fairly quickly find themselves working around or against things as much as they work with them.

A prime example of this is front-end work. Rails has support for *Script and the asset pipeline as ballyhooed features. Once you start writing more than a few dozen lines of Script code in three or four classes or modules, you quickly come to the conclusion that the people who designed and held court over how Rails works really didn't have much more Script experience than using snippets of jQuery or Prototype.js to "jazz up" a form. Heck, the "unobtrusive JavaScript" feature is essentially useless outside forms. What I'm working with at the moment is an app that intelligently reacts to a user selecting content with the mouse, based on the position of the content, the permissions and authorisation of the user, and other factors. Having a few Script classes (and associated specs) to deal with that introduces issues that may not be readily apparent in the beginning, but will bite your team over time.

The JavaScript community have developed several module loaders to work around the lack of such a feature in the core language (for now, at least); there seems to be something of a transition underway from CommonJS to RequireJS, with several alternatives providing variations on the theme. This is important for several obvious reasons, including the ability that programmers have enjoyed in various languages since at least the 1960s to develop, test and maintain cooperating code modules (objects, classes, etc) relatively independently. By having each such module load, and know about, only the other modules it uses, those modules (should) remain unaffected by other modules in the system, even indirect dependencies that continue to work as their dependents expect. Programming 101 stuff; nothing new for anyone reading this.

But, and it's a very large but, the Web prizes a site's ability to perform as few network accesses as practical to get its resources loaded and start doing useful work. To this end, Rails since version 3.1 has included the asset pipeline, which has the effect (broadly) of concatenating and compressing Script files, CSS stylesheets, and other resources so that they can all be loaded in a single request. This is done using an add-in Gem (component) called Sprockets. This works quite well for its intended purpose and truly is one of the nicest things about working in Rails — until you start writing modular Script code.

If you know that the asset pipeline, used by default, means that everything is loaded by the time any of your Script front-end code starts running, but you want to break your code into modules, the "obvious" choice is to create a lot of "global" objects for your classes and shared data and such. If you're an experienced developer, this feels wrong, almost dirty. You might then try only having a single global that functions as a top-level "namespace" for your application, with artefacts spreading out in a sensibly-organised tree below that. This path leads to names like @com.hudsucker.proxy.activation.foo.bar.baz.joebob.and.his.third.cousin. Maintaining such, or even keeping it straight in your head (and your teammates') quickly becomes an "interesting challenge". You quickly realise that there must be a better way, simply because there have been other developers before you who have remained (apparently) sane.

But to use loaders such as RequireJS from Rails, you need components like requirejs-rails, which is wildly, justifiably "popular". Sane Script dependency management should be in Rails, you start thinking. And I'd agree with you. However…

The Might-As-Well-Be-Official Rails "Solution" to the problem is: don't do it that way. Write your front-end code in a traditional Script framework like Dojo or Node or Ember or JoeBobBriggsAmazingFramework or any of no doubt numerous other excellent choices. Use Rails "merely" as the back-end API server for persistence and auth and so on. And yes, with planning and preparation and an adequately-staffed and -trained team starting well before your deadline, that's a splendid choice. You may even realise that Rails is overkill for your needs and move to something like Sinatra or Padrino or, again, numerous others.

But…what if you don't have that time or that team, but you do have a mass of rigidly-organised, fragile Script code that you just promised would be part of a working app in two weeks' time? Assuming that you agree that resignation and suicide are suboptimal strategies, you could try what I'm in the middle of retrofitting our code to:

1. Create a single, gateway object that you use to get all your other objects. Let's call it a Gateway object. give it a global location at @ourapp_gateway. Give it two methods:
   1. a register method, that associates a string name with an arbitrary object/class/other collection of bits;
   2. a use method, that takes a string parameter and returns the item associated with that name.
2. Remove the Sprockets #= require modulename lines from each of your existing Script files, and replace them with immediate calls to @ourapp_gateway.use corresponding to each of the objects you need from Your Outside App, as well as calls to @ourapp_gateway.register to register the object(s) declared in that source file with the Gateway;
3. Add Sprockets #= require directives to your application.js, so that Sprockets actually loads the suckers;
4. Assign the return values from @ourapp_gateway.use to variables as would be done in Node or whatever.

Why bother, I hear you thinking. Because, even without a proper module loader, this still delivers at least one major benefit: it abstracts away the relationship between your artefacts, the DOM, and the asset pipeline. It gives you reasonable flexibility in moving things around within the DOM and with respect to source files by having the Gateway instance as an intermediary. And, if you're thoughtful, diligent and very, very lucky, it gives you a leg up when you do get around to using a proper module loader, by letting you slip that in behind your Gateway abstraction.

And if you think I'm getting somewhat jaded/disillusioned with my continuing Rails experience and integrating a Script-using front end that's not a complete app in itself…congratulations; you nailed it.

Anybody have any better ideas that they've actually made work?

Tap-Dancing Among the Land Mines

…is best done very carefully, and with the knowledge that not everything works as you'd like it do. That's a good metaphor for my week this week.

Working on our new platform and its (eventual) variety of products and supporting tools, mostly in Ruby using Rails, some of the absolutely key elements are primarily front-end code, written as CoffeeScript running in the browser. Typically enough, we make heavy use of jQuery, various plugins, and other libraries. When understood and used correctly and they work as expected, they're great for reducing the effort needed to accomplish new and useful things.

The moderately-sized application I'm presently working on uses over 125 third-party components, many of which are dependencies of components we directly/intentionally use. Of those, some three dozen components have yet to reach a "1.0" release level. That's a lot of trust we put into people we've never met and teams completely outside our organisation. The fact that this is typical to the point of being mundane among Ruby shops is one of the phenomena most commonly cited by new Ruby devs and managers as "mind-blowing".

When everything works properly, it's awesome in a very different way than anything I've done in my nearly 35-year career. When anything fails to work properly, it's equally spectacular; just in a very different way.

Consider today. I've been working on a Script component (which I expect to publish in due course as a jQuery plugin itself). Being a proper behaviour-driven development project, I've been writing specs using Mocha and running them with Teaspoon. Teaspoon lets you run your spec suite either in a browser or from the command line. The browser-based interface is easy and convenient (almost fun) to work with during development; with a mouse click, you can focus on just the spec you're working with at the moment, or step back and run a group of related specs or your entire suite. After finally finding and killing the last of never-mind-how-many typos and silly errors (CoffeeScript, alas, has not yet implemented proper DWIM support), I had the component working. It was clean. It was more than fast enough for my purposes. And, most importantly, it worked properly, against every condition I threw at it.

That is, when I was running Teaspoon from the browser. The command-line invocation, needed for continuous integration among other things, failed in a very odd fashion. See, for example, this bit of screen output:

[jeffdickey@Jeffs-Prolog-iMac select_demo (wip)]$ teaspoon -q | tapout
Started w/ Seed: 0
..................F

1. FAIL markup from selectedMarkup()

    Unknown
    
    TypeError: 'undefined' is not a function (evaluating 'currentNode.contains(last)') in http://127.0.0.1:58247/assets/selection_wrapper.js?body=1 (line 111)
    
    window.meldd.SelectionWrapper for rich single-paragraph content, returns the correct markup from selectedMarkup().#:0


Finished in 0.092s (206.522 test/s, 0.004842s avg.)

19 tests: 18 pass, 1 fail, 10 exit, 0 todo, 0 omit

Now, bear in mind that when Teaspoon was running inside the browser, no such error message was displayed; all specs passed. The code being complained about looked something like:

  getMarkupUntil: (currentNode, contentString, last, offset) ->
    while currentNode?
      if currentNode.contains(last) # <-- THIS IS THE LINE BEING COMPLAINED ABOUT
        if currentNode == last
          # do something...
        else  # currentNode contains last but is not last
          # do something else...
      else # currentNode does not contain last but we're still looking
          # do something different...
    # return our results

Now, currentNode.contains is making use of a bog-standard HTML DOM function that's been in every major browser for years. Even Internet Exploder 5, from the Dark Ages of 1999, managed to support it! What could possibly go wrong?!?

Tap dancing on the shoulders of not-always-giants, it turns out.

Teaspoon, by default, uses PhantomJS as a "headless Webkit browser workalike with JavaScript support". If you're going to automate testing of your Web-based application (or any of several other related activities), this is the tool you want to use. Except, o course, when it doesn't work correctly. I changed the getMarkupUntil method I showed earlier to test my hunch:

  getMarkupUntil: (currentNode, contentString, last, offset) ->
    while currentNode?
      return null unless currentNode.contains?
      if currentNode.contains(last) # <-- THIS IS THE LINE BEING COMPLAINED ABOUT
        if currentNode == last
          # do something...
        else  # currentNode contains last but is not last
          # do something else...
      else # currentNode does not contain last but we're still looking
          # do something different...
    # return our results

The new line returns a null value from the function if and only if currentNode has no property (data or method) named contains. I reasoned that this would (fairly obviously) cause the calling code to fail, and the difference in the failure message (from what was shown previously) would confirm to me that this was indeed the problem. Again, running the specs from inside the browser presented no problems. Running from the command line, however, yielded

[jeffdickey@Jeffs-Prolog-iMac select_demo (wip)]$ teaspoon -q | tapout
Started w/ Seed: 0
..................F

1. FAIL markup from selectedMarkup()

    Unknown
    
    TypeError: Type error in http://127.0.0.1:58271/assets/selection_wrapper.js?body=1 (line 174)
    
    window.meldd.SelectionWrapper for rich single-paragraph content, returns the correct markup from selectedMarkup().#:0


Finished in 0.090s (211.111 test/s, 0.004737s avg.)

19 tests: 18 pass, 1 fail, 10 exit, 0 todo, 0 omit

As expected, the null value caused other code to die.

I then browsed the Teaspoon documentation, which pointed out that although PhantomJS was used by default, Selenium could be used instead, by installing the selenium-webdriver Gem and updating the project configuration accordingly. After re-bundling the app, I could run my spec suite from the command line. Hurrah!

Well, golf-clap applause at least. We'd had previous problems with Selenium testing the Script code as run from within a Web app. We'll burn that bridge again when it falls on us. Again.

"You use the tools you have, not the tools you wish you had."

(with apologies to Donald Rumsfeld, and noting that this is frustrating precisely because we're so asymptotically close!

I forget who first said this, but "Experience is as much a catalogue of what doesn't work as well as you'd hoped as what does" might have had my career in mind. Or at least the 2013 segment of it.

A few short months ago, I came across Trello, by Joel Spolsky's Fog Creek Software. As with all great Web products, it started as a brilliantly simple idea (a Web version of the classic stick-user-story-index-cards-to-it whiteboard that every Agile project centres around) that's slowly sinking under a tsunami of well-intentioned customer feature requests.

So excuse me if I seem to be a)piling on in b)an arguably hypocritical manner when I say that, like most things, it's both a blessing and a curse. You use it for a while, you start thinking in terms of cards and such… and then you realise that it's really not going to help you as much as you expected. Partly because of features, alas, but more because of philosophy, or communication of same.

Some features that I would gladly pay to be able to use:

1. There doesn't appear to be a "show me a list of all cards on this board, sorted by card number (creation date), last modification, and so on" feature. Apparently, the workflow Trello expects you to adopt is to have your to-be-done and in-work cards in various lists, and your "done" cards (and lists) should be archived. If, instead, you prefer to have a "Done" list, you'll quickly find that it grows into a mess that you're going to be searching through using the (very handy in itself) filtering feature, but that requires you to remember keywords exactly and, more importantly, prevents the "aha!" moments that often come when you can look at a list of not-very-obviously-related information and draw new insights from what you're seeing.

2. Trello also have (generally deservedly) made a Big Deal out of their support for "Markdown Everywhere", but their parser doesn't support the feature that future-proofs Markdown itself; the ability to embed HTML inline seamlessly. This is especially noticeable for things like tables, which have no Markdown (or Trello) direct implementation.

   This lack means, for instance, that "rolling your own" card that contains a list of all cards in the board looks pretty cluttered and hard-to-read:

   

3. Checklists are almost optimally useful and easy to use. However, they suffer from one major misfeature: converting a checklist item to a separate card "removes the item and creates a card with the title of the item beneath the current card". It would be very useful to have Trello, instead of deleting the checklist item after the card has been created, replace the checklist item's text with a link to the new card. Many (most?) of our cards start out as blue-sky, high-level description of a functional feature. For example, "Any User, or any Guest, may view a list of all Articles with public status." That card had a six-item checklist on it, several of which turned out to be "worth" breaking out to separate cards. For each of those checklist-items-turned-cards, I went back and re-added an item to the checklist with a one-sentence summary of the card and a link (e.g., "Update the `Ability` model; see Card #47."), followed by editing the new "item" card so that its description started out like "This is a checklist item from Card #44."

   Why? In our process, each card (most granular tracked activity) should take a reasonably consistent amount of time, generally under a man-day of effort. When a single card instead takes 4-5 days (as has happened), it's easy for customer reps and investors to get the idea that we've slacked off because they understand our estimating process/goals. Being able to drill down (and up) from any given card, without the level of effort presently required on my part, helps avoid misunderstandings and time lost responding to panicked emails.

4. Trello allows you to "subscribe" to cards, lists and boards. This adds items to a list viewable when you're logged into trello. It would be much more useful if these "feeds" were available via RSS/Atom as well.

tl;dr: Trello is a great tool; so great, that many users (including me) are either pulling it either in directions it wasn't designed to go, or are demanding features that don't exist in the way we'd like them. I'll keep using it and recommending it for now, even though the time involved in running a project with it is scaling up prodigiously. That time, however, is less than the time it would take to work without Trello or something quite similar to it.

Lies, Damned Lies, and Truths Backed By Statistics

(with apologies to Mark Twain and Henry Du Pré Labouchère)

UPDATED 6 October 2014; see below. Original content follows:

This afternoon, I came across a fascinating post by Prem Sichanugrist on the Thoughtbot blog, courtesy of the ginormous informational firehose known as Twitter. In it, he discusses a new feature of Bundler 1.4.0, currently in prerelease, that lets you use multiple cores to install the Gems for your project.

Finally, you exult. One of the slowest (read: most network-bound) tools in the Ruby toolbox is getting at least some help. In his post, Prem recommends "setting the size" (number of concurrent bundle subtasks to run concurrently "to the number of CPU cores on your machine".

Don't do that! At least, not with doing some more exhaustive testing first. Not because the feature doesn't work (it does, splendidly) but because matching the --jobs option to the number of cores may well give you suboptimal performance.

Benchmarks

Following is a summary of the results on my current project on my main development system, a Mid-2011 iMac with a 3.1 GHz Core i5 CPU (4 cores) and 16 GB of RAM under OS X 10.8.4. The command lines used for benchmarking were

for BIJOBS in 1 2 3 4
do
  export T1=`date`
  rm -rf Gemfile.lock vendor/cache/* vendor/ruby/*
  bundle install --jobs $BIJOBS --path vendor
  bundle package --all
  bundle install --binstubs
  echo $T1
  date
end

Time to Install Gems with Different --jobs Settings

Value of	Elapsed	Time Savings from	Time Savings in % from
$BIJOBS	Time	$BIJOBS	$BIJOBS
1	10m 41s	0m 00s	0%
2	6m 21s	4m 20s	40.56%
3	4m 44s	5m 57s	55.69%
4	6m 31s	4m 10s	39.62%

As you can see, the trend was progressing nicely for 2 and 3 cores, at ~40% and ~56% time savings relative to --jobs 1 respectively. When --jobs had the value of 4, however, the savings was less than when using only two cores. Why?

My conjecture is that Bundler suffers a (relative) drop in efficiency if it has to swap out use of one of the CPU cores between bundling Gems and everything else. I suspect that many people will find the advice

Set the value of --jobs to one less than the number of cores in your CPU(s)

to be optimal.

Conclusion

It's good to have our tools use the available hardware resources (CPU cores, RAM, etc) more efficiently. But we should always remember that the obvious "best" configuration is not always the actual best configuration.

Thanks again to Prem Sichanugrist of Thoughtbot, as well as to Kohei Suzuki who Prem credits with the original patch enabling multiple cores in Bundler. Thanks, guys!

---

UPDATE — 6 October 2014: Matthew Rothenberg has published new benchmarks and a discussion for how things have (apparently) changed significantly with the current-as-I-write-this Bundler 1.7.3 and apparently since 1.5.3. I don't have the resources to replicate his test environment (local propaganda is titled "From Third World to First"; we made it halfway) but I don't doubt his results. It will be interesting to see what insights may be drawn from repeating his benchmarks on slower hardware and a far slower Internet connection. I expect things to become I/O-bound far more quickly than in his results, but I'll be benchmarking and reporting again by mid-October.

Thanks again, Matthew!

When Development DOES Go Horribly Wrong™

I just finished watching Ian Cooper's talk at NDC Oslo, TDD, where did it all go wrong. If you're doing any flavour of TDD (BDD, etc.), this will likely be one of the more important hours you spend this year. Ian helped me wrap my head around the details of what some of those voices screaming in the back of my mind as I (and likely you) write code, were trying to make me understand.

Bear in mind that I've been doing test-driven development for a decade, and some form of behaviour-driven, outside-in development for perhaps half that. I was once known as a "crazy blue-sky hacker", who composed symphonic-length pieces of software at a single sitting. (I'm explicitly disclaiming anything about the quality of that software; while it may have seemed sufficiently fit for purpose at the time, there almost certainly were bugs aplenty waiting the hapless maintainer.)

One of Ian's main points in the talk, if not the main point, is that test-driven development (and its successors such as BDD) should test behaviours of the system, not how every little internal method behaves. Lots of great insights here, from how to be more effective with fewer tests, to "write unit tests that focus on behaviours and thus can be used for acceptance". More bluntly, "the reason to test is a new behaviour, not a method or a class." I'm as guilty of that as anybody.

That last has been something I've been trying to work toward for well over a year. Unfortunately, one of the poisonously bad habits I've picked up working in Rails is the idea that there's a one-for-one mapping of specs/tests to implementation classes and methods (model, controller, helper, etc.). These are the trees that keep me from seeing the forest of what I'm ultimately trying to accomplish, and more often than not, have me polishing the nodules on each root of each tree.

One of the things that comes out of this understanding for Rails or, to be fair, for any framework in any language that focuses more on implementation classes than expressed behaviours, is that I for some time now have avoided use of the generators, particularly scaffolds, for core application components (models, controllers, views, etc). My preferred workflow has evolved to something like what I list below. Note that this is my existing workflow, and does not yet incorporate the new understanding of proper TDD.

1. I'll usually do a "first whack at" a model spec and bare-bones model, just to have something "specific" to work with in the later steps. I am beginning to see the way I have been doing this as a bad habit, particularly in "traditional" Rails. A Rails model should be concerned with persistence and validation, and an absolutely minimal amount beyond that. What I really want to start out with is my initial idea of a "business" domain object, which is Not The Same Thing;
2. I'll write what in the Rails world is called a "feature spec" and most of the rest of the omniverse knows as some form of "integration spec", modelling how a specific feature works. ("As a Member, I want to create a new BlogPost, to begin a discussion about a topic.") The initial "high-level" spec may or may not evolve into a series of specs which each demonstrate an individual sub-feature, but will always be expressed from the outside in, mocking what does not yet exist;
3. Next is a controller spec and the controller that it specifies, as a "traditional" developer-oriented spec (or set of specs). When done, I should be able to integrate this controller into the feature spec written previously (still using mock models) and demonstrate that the controller works;
4. Now, I re-look at the model(s) involved, turning those into "live" specs and "real" code that works correctly with what I've written thus far; my feature specs and controller specs still pass;
5. After the models and controllers are done and working with the feature specs, I'll do the views. I should have a very firm idea what's needed for these by now, as they've been used throughout (in mock or throwaway form) by the feature specs; this is where I write the code that emits the HTML, CSS and Script code for the UI.

That was my old workflow. I'm going to explore how to adapt and adjust this to more effectively and efficiently get the job done in light of Ian Cooper's talk and a (very near-future) reread of Kent Beck's Test-Driven Development: By Example. I say "effectively and efficiently" as my current project's spec-to-live-code ratio (in LOC) is approximately 13:4. I think most of us would agree that that's a very strong code smell.

A Product Idea: e-Books Are More Like Paper Books Than You Think

I prefer e-books to paper books these days, particularly for technical or reference books. So do most people reading these words, I expect. Being able to carry around a shipping container full of books without carrying a shipping container as your briefcase has an undeniable appeal. But the form has not yet been perfected; especially if you're looking at reference or technical books, current e-books (whether in PDF or more proprietary e-reader format) has yet to conquer one of the true banes of paper books.

A relevant aside: This afternoon, I received a notice that the latest update to the new edition of the (excellent) Everyday Rails Testing with RSpec by Aaron Sumner was available, so I went and downloaded the update.

Then ensued an hour of activity that is familiar to any student or researcher since long before Gutenberg: opening both old and new editions (in a novel twist, opening these in my reader, Preview on OS X) and applying the highlights and notes to the new edition that I'd made to the old. What allowed me to complete the task, double-checked, in an hour rather than a week, was Previews "Highlights & Notes" view, which let me skip to the next scribble I'd made on the old copy, find the corresponding bit of the new, and adapt the highlighting or note as appropriate. Tedious enough to momentarily wish for an automated "copy my notes over" feature in Preview, which I quickly mentally filed in the "first world problems" circular file.

So why bother blogging about this? Two points, first: yes, of course we're far more efficient and effective in this use case with e-books than paper books. But, most importantly: yes, there really does need to be a better way to deal with matching changing versions of underlying documents to annotations. I've been developing professionally for the Web almost literally as long as there's been a "World Wide Web" for the general public to develop for and use, and I have yet to see any serious headway made. Does anybody know any different?

"Easy Credit" IS "Easy Debt". It sucks for finance. It sucks just as much for your code.

One of the great things about the Rails ecosystem is that there are lots of different tools that can save you time and pain during development. Some of them work by embracing and extending the Rails standard bits, like replacing (nearly) bog-standard ApplicationController subclasses with subclasses of InheritedResources::Base. Some of them instead swap out pieces of the original Rails stack, like replacing ActionView with Erector or something similar. I wouldn't be at all surprised if my current app that mixes the two practices (in different areas) isn't at all unusual.

But whichever route or routes you choose, you need to constantly remain aware of that choice, especially when considering new tools you'd like to incorporate into your app. And you may also find that the tools you're using, while superb for their intended purpose, complicate other choices you make in the course of developing your project. A tool which was the greatest painkiller since morphine when you first picked it up, now seems to be "getting in the way" as you get farther along in development.

As practitioners of agile development, this shouldn't scare or even surprise us (though our non-technical stakeholders may initially be less sanguine). Good agile architecture says that everything in your app other than the raw business rules is an implementation detail that should be testable and fully replaceable in isolation.

Novice to journeyman Rails developers may be forgiven at this point for saying "but that's the way Rails is; it all works together, even though you can swap pieces out as you like". A majority of Rails developers have at least experimented with swapping out Test::Unit for RSpec, or erb for Haml or Slim. More adventurous still are the folks who swap out the ORM, replacing ActiveRecord with, say, DataMapper or the aforementioned Erector for ActionView. Why are they trying such things? Because they're asking (quoting Wynne again), "what does a modular Rails architecture look like?" Or, "we know we've hit the wall with our current (connected) architecture; everything we touch introduces new tech debt if not outright breakage, but we don't know how to do anything else and stay true to The Rails Way."

That "wall" you've hit has nothing to do with the number of lines of code you've written nor the number of unique visitors your production app serves per day nor the revenue your app is bringing in the door. You've realised that your ability to keep bringing in that revenue, let alone increasing it, is dependent on you somehow reconciling your understanding of "agile development" and "software engineering-style practices" with "the Rails Way". One of these things is not like the others. If you haven't yet, watch both of those videos. Particularly in Matt Wynne's talk from about 5:30 where he talks about the crossover point between a "connected" architecture being cheaper/faster/better giving way to a "modular" one… that rings a lot of bells for me. Likewise when Uncle Bob rails against Rails apps screaming "I'm a Rails app, never mind what I do". I've done several projects, never mind how many, where I've gotten to that point (which Wynne describes as "hang on; I don't want to work on that application anymore. Something's seriously wrong with it") and my reaction to the realisation that "something is seriously wrong" has been to stop, look at everything I'm doing and evaluating how well I'm following convention, "best practice" and so on, come to the conclusion that I'm doing everything exactly the way I've been taught it should be done, and deciding that that proves the fault is in my understanding of the problem, so I start over, from scratch, trying to figure out where I fell off the cliff. With one application, that involved at least five restarts over the course of a year and a half… and I must have always "fallen off the cliff" at the same spot, because I never saw it coming before I realised I was at the bottom of that cliff once again. "Uncle Bob" Martin's keynote (from Ruby Midwest 2011) does a great job of explaining why an experienced OO software developer like me has always felt uncomfortable with Rails' interpretation of the "Model-View-Controller" architecture. By writing your app centred on what the app is for rather than how the app presents itself on the Web, you make your life a lot easier and more enjoyable. Looking back at my commit history over the last year and a half, I see that well over 2/3 of the times I was "stuck" for more than a couple of hours had more to do with "how do I adapt what I'm trying to do to how Rails wants me to do things" than "how do I implement (this feature)". That should, and has been, setting off all kinds of alarm bells; I'd just drank enough of the Flavor Aid to think that pain level was "normal" for Rails. It shouldn't be.

Going forward, my strategy for Rails development is going to work something like this:

1. Model the domain objects (Martin's "interactors", "entities", "boundaries" and "entity gateways", after Jacobsen "control objects") as POROs (Plain Old Ruby Objects);
2. Wrap those up in Gems whose use can be demonstrated by specs, supplemented by a simple (Web or JSON or command-line; doesn't matter) example or set of examples.
3. Plug those into a Web app using the current framework of choice (Rails, Padrino, JoeBobsAwesomeFramework, or whatever), whose sole purpose is to implement the "client" side of the boundary layer and present what comes back over the fence.

Now we're back to something more closely approximating classical agile development that focuses on delivering business value, not playing mind games with a tool that insists on making most of the important decisions for you, and makes them in ways that make it harder for your app to be scalable, performant, or fun to work on. Or even for your app to be finished.

Thoughts?

---

P.S. Oh, and as far as InheritedResources::Base, that truly useful tool for Rails app development I mentioned before? We'll continue to use it as a default, but if we choose to use tools that "get in the way" (like wisper) by wanting all your methods explicitly defined, we'll switch back to ApplicationController, on the theory that making the puzzle one piece smaller without affecting how the other pieces fit together is a Good Thing™. But that's another decision we want to defer as long as possible; "good architecture" is all about deferring as many decisions as possible as long as possible, while still getting business value done.

Tools Save You Time, Except When They Don't

Two of the greatest boons yet to hit Railskind, IMAO, have been Platformatec's Devise and inherited_resources. Surprisingly, given how closely they're related, they don't always play nice together. If you ever get a failing feature spec that's just modifying non-password data on your user-profile page, it can really confuse you until you do some Googling around. (That's how you got here, yes?)

You'll need to update your User controller so that it doesn't send empty password fields to Devise for the update action. This won't affect creating Users; you want Devise to reject empty passwords there. Here's what I have for my users_controller.rb (also available as a Gist):

---

# Actions for Users. Not necessarily a finer distinction one way or another yet.
class UsersController < InheritedResources::Base
  # FIXME respond_to :json # should spec JSON actions, too
  respond_to :html
  load_and_authorize_resource

  def update
    # Devise has the convention that, for updates where the password is NOT to
    # be modified, the password and password-confirmation fields MUST NOT be
    # submitted. inherited_resources, of course, just sees two text fields and
    # wraps them up along with the rest of the form. Oops.
    remove_password_parameters if password_fields_are_empty?
    super
  end

  private

  def password_fields_are_empty?
    params['user']['password'].empty? and
      params['user']['password_confirmation'].empty?
  end

  def remove_password_parameters
    params['user'].delete 'password'
    params['user'].delete 'password_confirmation'
  end

end

This is still a heck of a lot cleaner/shorter than what I'd have written for a controller subclassed directly from ApplicationController. But is it universally better, or better for your specific needs? Thoughts on that in the next post.

FOCUS! Oooh, shiny, shiny...

That didn't last long; having a nice set of components to build on and being satisfied that it would carry us forward indefinitely…

As Ruby developers on Rails or Padrino or bare metal, we're absolutely spoilt for choice. As I write this, Rubygems claims 1,577,138,099 downloads of 56,183 gems since July 2009. (Compare to ~495 PHP packages in the official PEAR repo on Github.) If you can figure out what you're looking for well enough for a Google search, chances are excellent that you'll find something you can at least use; likely several somethings. As with all such, this is both a blessing and a curse.

The blessing is that the stuff is out there; it is mostly at least usable, especially for the gems that are at or past Version 1.0 (yes, there's a formal numbering system for gems); and if it doesn't do precisely what you want, chances are likewise excellent that you can wrap your own code around it to do your bidding, and/or pull a copy of the official source, make changes you'd like to see, and submit a pull request to tell the original developer(s) what you'd like to see pulled into the "official" version. ANd so on.

Back a couple of years ago when I was doing PHP for a living, it wasn't that hard to keep a rough idea of what was both available and usable on PEAR in my head, so that as I developed a project, I knew what I had available and could try to structure my code to work effectively with it. It's a lot easier when you can reverse that: think of what you want and how you'd like to work with it, and go run down the list of candidates to plug that hole.

But that touches on the other edge of the sword, and it's sharp. Doing all that browsing and cataloguing and selecting and integrating takes time and effort; time and effort that your customers (external or otherwise) generally assume you're putting into making a kick-ass product. Well, you are, sort of; you're just trying to figure out which giants' shoulders to stand on. Throw in the usual social media like Google+ and Twitter and so on, alerting you to the newest and shiniest of the new-and-shiny, and it's a wonder you get any "real work" done.

Memo to bosses: This is real work, and it's almost certainly less expensive real work than having your folks reinvent several dozen (or several hundred) wheels as they develop your Awesome Product, Version next.0. Deal.

But the "is it done yet?" bosses do have a very good point, devs. Promises have been made; contracts have been let; people's jobs (including yours) may well be on the line. Focus is necessary. You just have to learn to do it in a reasonably structured way, like your RSI breaks.

When you come across the latest shiny-shiny that you just know is going to make your job so much easier, better, more fun and profitable, add it to a list. When you're about to start on some new part of your project, glance at the list and see if it makes sense to use your new feature-thingy as a test bed for the new gem. Be careful that it can peacefully coexist with the rest of your codebase that hasn't been updated yet, and do remember to use a new feature branch. Set yourself a time limit for how long you'll allow yourself to get the new gem working. When that's expired, give yourself at most one extension (hey, you're exploring the not-yet-known-to-you, after all) and if it's still not walking and chewing gum at the same time, go back to where you branched and try again, the "old" way. This is so simple, yet not doing it can have a devastating effect on your schedule (and project, and…)

But given a choice between running the risk of distraction by a myriad assortment of Things That Generally Work, and trying to make one of a smaller number of Things That Should Work But Don't, Quite… well, that's what brought so many VB6 and Java folk to PHP in the first place, and what brought so many PHP people (including yours truly) into Ruby. Given sufficient resources and patience, you can make almost anything work in almost any language; I once saw a quite-playable version of the classic Star Trek game written in RPG II. I thought, and still think, that the guy who wrote it was absolutely nuts (especially since he knew at least three other languages that would have been easier/better aligned to the job), but *it worked*. So, as the adage goes, never argue with success. However, given the chance early enough in the project for it to make a difference, do respectfully suggest that jobs are best done with the available tools most closely aligned to the job. RPG is still in production for what it does best: complicated report generation from highly-structured data. PHP remains a valid choice for small projects, or for those companies unwilling to rely on non-local talent and willing to trade initial cost for quality. Ruby has numerous tools available, for just about anything, that challenge its practitioners to be better developers than they started out to be. Part of that challenge is the sheer wealth of tools available, and their haphazard discoverability. Part of it is looking back at code you wrote a few short months, or even weeks, ago; knowing it could be so measurably better; and having the self-discipline to leave it be until you have a business reason to change it.

Problems like that make my days more enjoyable than solutions in other languages generally have. Now if I could just turn this megalithic Rails app into something more componentised, service-supporting. Oooh, shiny, shiny…

Yes, It's Nonsense.

No doubt profitable nonsense. What follows is a reply to a comment by one Andrew Webb on an InfoQ.com sales-pitch-as-technical-"journalism" puff piece. The piece was "reporting" on a study by Dr Donnie Berkholz of InfoQ.com. This kind of hucksterism is one of the most formidable barriers blocking our craft of software development from ever becoming a professional engineering discipline but, well, read on if you will.

Why here rather than on InfoQ? Simple: even though their reply-entry form states that <a> is an accepted HTML element in comments, it refused to accept any of the links I had in this post. Shoddy "journalism", neet shoddy Web development.

---

Worse than nonsense; this was written for PHBs, likely as a tool to sell consulting hours training teams in the "more expressive" languages. Anybody who remembers Java's transition from a language and VM into a marketing platform, circa 1997-1998, has seen this done before and better.

Note that I am not talking about Dr Berkholz' original study, the link to which was buried in this puff piece. But there appears to be a real problem with the data that the study was based on from Ohloh. From what I've been able to see, the data covers a period of at least 15 years (late 1990s-present). Subversion(!) was used for 53 percent of the projects in the data set; when you add in now-Palaeolithic CVS, the share rises to nearly 2/3 of the projects covered by the data set.

Let me beat that timeframe to death one more time: In the last 20 years, we've gone through at least two major revolutions in the techniques mainstream developers use. Twenty years ago, if you mentioned "OOP" in half the corporate development centres in North America or Asia, the response would have been "what's wrong?" We as a craft, we were just beginning to get our minds wrapped around object-oriented software development, moving it out of its metaphorical Bronze Age, when the larger revolution which that enabled, BDD/TDD/whatever your flavour of Agile, hit like a silver tsunami. Twenty years ago, I was writing "C/C++", Fortran and Ada code, using various odd bits of version control (anybody else remember how good SourceSafe was before Microsoft bought it?), and checking in massive commits, because centralised SCM systems like SourceSafe, CVS and RCS are a pain; seen less as a design/team-support tool than an insurance policy against Raj's hard drive being wiped by some virus or other. Network connectivity was not as omnipresent, reliable or fast as it is today, by several orders of magnitude.

Nowadays, regardless of what language you're in, you're (hopefully) using some form of agile process. You're strongly encouraged to tunnel in, specifying the smallest change that could possibly fail, and then implement just enough code to make that spec pass. You're not going to check in thousands (or even hundreds) of lines of code at one go; you're going to build that feature you're working on over several commits, on a branch of your SCM tree, and then merge it back into the mainline when it's done. The last ten PHP projects I worked on, over a six-year period, averaged less than 50 lines per commit – in one of the most overly verbose languages since ALGOL 68 and COBOL.

Changes in development practices, and tools, will hopelessly skew that data set unless additional controls are applied, and I could find no description of any. That, to me, says that this is, at best, an enjoyable-in-the-moment bit of data-mining with all the relevance to day-to-day developers' lives of the Magic 8-Ball.

This article was even worse; no examination of assumptions, no discussion of changing trends in the craft and industry, just a vacuous puff piece. An insult to the intelligence of the readers and, despite the possible flaws in the original study, to Dr Berkholz' work as well. If my mind could go take a hot shower to wash the oily residue off, it would. I used to think ZDNet was the bottom of the barrel for this sort of thing. I was wrong.

---

NOTE: Earlier, I'd previously written "Twenty years ago, I was writing "C/C++", Fortran and Ada code, using Subversion", which is an anachronism, of course. Subversion came out in 2000, and I was not in a time warp in 1994. It just feels like I'd been using (used by?) it for that long.