Going Forward; Avoiding the Cliff While Looking for Cliff's Notes

Over on the Ruby sub-Reddit, I got into a discussion (mostly with Jelly_Jim, the OP, and with realnti) about the conflation in far too many minds of Ruby, the programming language, with Ruby on Rails, the famously-"opinionated" Web application framework. While pontificating on Jelly_Jim's original question, I described my frustration-bordering-on-antipathy with the traditional (known as "The Rails Way") application structure; listed a few presentations, books and blog posts that have influenced me greatly, and described my own take on the Kübler-Ross model for Ruby-based (and at least initially, largely Rails-based) Web development.

That discussion, over the course of roughly a day, helped me figure out how our app is going to survive the expected Slashdot effect, transitioning from a pretty traditional, megalithic Rails app to an app that uses Rails components in a sharply different, more recognisably "object-oriented", service-based architecture.

Leaving aside my more prosaic experiential difficulties with Rails (I really loathe ActiveRecord as commonly used), and taking into account the "Uncle" Bob Martin keynote, the various blog posts I referenced in the original Reddit post, and a couple of books I mentioned in a later reply, I think I've possibly hit upon a way to get where I want us to go.

That's the easy part, of course; it only took me never-mind-how-many months. The "interesting" part is going to be getting there — understaffed, on a shoestring budget for the moment even by startup standards (another problem I'm working), and working a schedule for two and a half years that would land a healthy twenty-year-old in hospital by now. Comments/suggestions are, as always, greatly appreciated.

---

Basic Architectural Principles

• Loosely-coupled, granular architecture FTW. That means, inter alia,
  • Nearly the entire app to be packaged as Gems used by an otherwise minimal application;
  • Rails mountable engines to package assets (particularly CoffeeScript/JavaScript) and major components;
  • Plan and design for updating and provisioning components which communicate across natural architectural seams.
• Hexagonal, "ports-and-adapters", or "clean" conceptual architecture; all dependencies point inward. Your domain entities, at the centre of the system, will change less than details like the delivery mechanism (Web UI, phone app, etc) or the database (which the domain shouldn't even be able to prove exists as such).
• By adopting a heavily job-story-oriented view of the workflows; with the use of online tools like Trello, WebSequenceDiagrams.com; and with supporting code tools like ActiveInteraction and so on, we should be a lot more productive and a lot more Agile than we have been during our forty (calendar time: two) years lost in the wilderness.
• And oh, yeah, just to keep things interesting: we've got to keep regularly demonstrating forward user-visible progress, for all the obvious reasons and a couple I really can't go into here.

---

How We Get There, v. 28.0

First, since the major focus of late has been on some unique (and not-so-unique) CoffeeScript code, start there. Separate the existing Script code out into four engine-based Gems, following/adapting Derek Prior's example. These levels are, from conceptual bottom to top:

1. A "gateway" CoffeeScript class we've written as a poor man's RequireJS, that gives us just enough pseudo-dynamic referencing of data/code without changing anything the Rails asset pipeline assumes;
2. Our directly DOM-aware Script code, as one or possibly two Gems (one is a major crown jewel; we might decide to split off utility/support code into a separate Gem, so as not to touch one when maintaining the other);
3. The code proxying our internally-used message queue (decoupling "FTW", remember?) and the Ajax code to communicate with the Rails back end; and
4. The code implementing various domain/service objects sitting atop the Script stack; one Gem per service.

Next, adapt the relatively-straightforward (as yet) Ruby/Rails code to honour the architectural principles listed earlier. This is where tools like ActiveInteraction pull their weight (and solve several existing pain points in the process).

Just to make sure we really do understand where we're going with this, take our first user story-specific code (including interactor) and package it up as a Gem that should then Just Work as in the previous step. Start thinking about whether we really want traditionally-packaged Gems or unbuilt dependencies that remain part of the single app repository.

Proceed to reimplement and repackage the (again, relatively few) job stories already existing as was done in the preceding step.

Start knocking down the remaining Trello cards as we complete our app.

We've been saying since the Very Beginning that anything that we were building that wasn't inextricably tied to proprietary, commercially-valuable code (or that gives too much of a hint into any unique details it may have) "should be" openly released, but hadn't yet figured out a feasible, cleanly reusable way to do that. If we've done our jobs properly up to this point, we have that way now.

Push the "deploy and launch" Big Green Button. We've bloody well earned it by now.

---

Any Suggestions or Comments?

Trust, Tools, and Tripping Over Shoelaces While Dancing

On the shoulders of giants, that is.

While "a poor craftsman blames his tools", sometimes those tools lead you down the garden path to such a degree that you don't realise that you're doing a Wile E Coyote impersonation until about the third bounce off the canyon floor. For your amusement and edification, if not abject stupefaction, I present a subset of a passing spec file. This is CoffeeScript, using teaspoon and mocha for specs. It's deceptively straightforward, IMAO:

#= require 'gateway'

describe 'Gateway class', ->

  beforeEach ->
    @subject = new window.meldd.Gateway()
    @name = 'foo'
    @value = $('body')

  # other specs elided...

  describe 'has a "useGroup" method that', ->

    it 'takes one parameter', ->
      expect(@subject.useGroup).to.have.length 1

    describe 'when called using', ->

      it 'a nonexistent group name, returns an empty object hash', ->
        group = @subject.useGroup 'not found'
        expect(group).to.be.an 'object'
        expect(group).to.be.empty()

      describe 'a group with one key, returns an object', ->

        beforeEach ->
          @subject.register 'foo', 'bar', 'group1'
          @group = @subject.useGroup 'group1'

        it 'hash', ->
          expect(@group).to.be.an 'object'

        it 'with the correct single item name as the only key', ->
          expect(@subject.groups['group1'][0]).to == 'bar'
          expect(Object.keys(@group)).to == ['foo']

        it 'with the correct item value as the only value', ->
          expect(@group['foo']).to == 'bar'

      describe 'two groups with one identical key, returns', ->

        beforeEach ->
          @subject.register 'this', @, 'group1', 'group2'
          @group1 = @subject.useGroup 'group1'
          @group2 = @subject.useGroup 'group2'

        it 'the identical object in both groups', ->
          # equality to avoid "Converting circular structure to JSON" error
          expect(@group1['this']).to == @
          expect(@group2['this']).to == @
          # identity; they're the exact same object instance
          expect(@group1['this']).to.be @group2['this']

Pretty straightforward, and both of you Scripters reading this can probably code up a proper Gateway.useGroup method without knowing much about the rest of the class. I'd bet the rent that your method would not closely resemble this:

  useGroup: (param) ->
    {}

And yet...

What. The. Fox?!?

No, You Can't Always Get What You Want...

…but as Mick Jagger and Keith Richards remind us, if you try "sometimes you get what you need". Any software developer lives with that on a daily basis; the implications of that are one of the major things that separate our craft from the profession of engineering. I had several of those "implications" blow up in my face this week as I was working on (what else?) a Rails app.

Or, more properly, an app that uses Rails. Because what the majority of my time has been spent with for the last few weeks has not been Rails, or even Ruby, but trying to get the front-end features that have to be implemented in JavaScript (or CoffeeScript or another language that compiles to JavaScript) (generically here, Script) because they're dependent on user-agent and DOM events. The last time I went down this road, I wound up with an app that had five lines of CoffeeScript for every two of Ruby; this one is well along that path. That's relevant; I suspect that there are far more non-trivial apps that use Rails in production than there are Rails apps; for anything beyond a simple pseudo-blog or relatively straightforward database front end, you sometimes wonder what the early Rails core team was thinking when they established some of these "conventions" that trump "configuration". Granted, there are a lot of people using that "sweet spot" for Rails app development, and more power to them, but I'd bet heavily that there are at least as many who started out trying to do an app that uses Rails and fairly quickly find themselves working around or against things as much as they work with them.

A prime example of this is front-end work. Rails has support for *Script and the asset pipeline as ballyhooed features. Once you start writing more than a few dozen lines of Script code in three or four classes or modules, you quickly come to the conclusion that the people who designed and held court over how Rails works really didn't have much more Script experience than using snippets of jQuery or Prototype.js to "jazz up" a form. Heck, the "unobtrusive JavaScript" feature is essentially useless outside forms. What I'm working with at the moment is an app that intelligently reacts to a user selecting content with the mouse, based on the position of the content, the permissions and authorisation of the user, and other factors. Having a few Script classes (and associated specs) to deal with that introduces issues that may not be readily apparent in the beginning, but will bite your team over time.

The JavaScript community have developed several module loaders to work around the lack of such a feature in the core language (for now, at least); there seems to be something of a transition underway from CommonJS to RequireJS, with several alternatives providing variations on the theme. This is important for several obvious reasons, including the ability that programmers have enjoyed in various languages since at least the 1960s to develop, test and maintain cooperating code modules (objects, classes, etc) relatively independently. By having each such module load, and know about, only the other modules it uses, those modules (should) remain unaffected by other modules in the system, even indirect dependencies that continue to work as their dependents expect. Programming 101 stuff; nothing new for anyone reading this.

But, and it's a very large but, the Web prizes a site's ability to perform as few network accesses as practical to get its resources loaded and start doing useful work. To this end, Rails since version 3.1 has included the asset pipeline, which has the effect (broadly) of concatenating and compressing Script files, CSS stylesheets, and other resources so that they can all be loaded in a single request. This is done using an add-in Gem (component) called Sprockets. This works quite well for its intended purpose and truly is one of the nicest things about working in Rails — until you start writing modular Script code.

If you know that the asset pipeline, used by default, means that everything is loaded by the time any of your Script front-end code starts running, but you want to break your code into modules, the "obvious" choice is to create a lot of "global" objects for your classes and shared data and such. If you're an experienced developer, this feels wrong, almost dirty. You might then try only having a single global that functions as a top-level "namespace" for your application, with artefacts spreading out in a sensibly-organised tree below that. This path leads to names like @com.hudsucker.proxy.activation.foo.bar.baz.joebob.and.his.third.cousin. Maintaining such, or even keeping it straight in your head (and your teammates') quickly becomes an "interesting challenge". You quickly realise that there must be a better way, simply because there have been other developers before you who have remained (apparently) sane.

But to use loaders such as RequireJS from Rails, you need components like requirejs-rails, which is wildly, justifiably "popular". Sane Script dependency management should be in Rails, you start thinking. And I'd agree with you. However…

The Might-As-Well-Be-Official Rails "Solution" to the problem is: don't do it that way. Write your front-end code in a traditional Script framework like Dojo or Node or Ember or JoeBobBriggsAmazingFramework or any of no doubt numerous other excellent choices. Use Rails "merely" as the back-end API server for persistence and auth and so on. And yes, with planning and preparation and an adequately-staffed and -trained team starting well before your deadline, that's a splendid choice. You may even realise that Rails is overkill for your needs and move to something like Sinatra or Padrino or, again, numerous others.

But…what if you don't have that time or that team, but you do have a mass of rigidly-organised, fragile Script code that you just promised would be part of a working app in two weeks' time? Assuming that you agree that resignation and suicide are suboptimal strategies, you could try what I'm in the middle of retrofitting our code to:

1. Create a single, gateway object that you use to get all your other objects. Let's call it a Gateway object. give it a global location at @ourapp_gateway. Give it two methods:
   1. a register method, that associates a string name with an arbitrary object/class/other collection of bits;
   2. a use method, that takes a string parameter and returns the item associated with that name.
2. Remove the Sprockets #= require modulename lines from each of your existing Script files, and replace them with immediate calls to @ourapp_gateway.use corresponding to each of the objects you need from Your Outside App, as well as calls to @ourapp_gateway.register to register the object(s) declared in that source file with the Gateway;
3. Add Sprockets #= require directives to your application.js, so that Sprockets actually loads the suckers;
4. Assign the return values from @ourapp_gateway.use to variables as would be done in Node or whatever.

Why bother, I hear you thinking. Because, even without a proper module loader, this still delivers at least one major benefit: it abstracts away the relationship between your artefacts, the DOM, and the asset pipeline. It gives you reasonable flexibility in moving things around within the DOM and with respect to source files by having the Gateway instance as an intermediary. And, if you're thoughtful, diligent and very, very lucky, it gives you a leg up when you do get around to using a proper module loader, by letting you slip that in behind your Gateway abstraction.

And if you think I'm getting somewhat jaded/disillusioned with my continuing Rails experience and integrating a Script-using front end that's not a complete app in itself…congratulations; you nailed it.

Anybody have any better ideas that they've actually made work?

It's Not Just "JavaScript" Anymore

Late to the party again, as usual, but…

I've been reading up on ECMAScript 5, the standardised language historically and popularly known as "JavaScript". Even though I'm doing most of my work in CoffeeScript now, ECMAScript is relevant because that's what CoffeeScript compiles down to. (At least, to a 'safe', 'good bits' subset thereof). So here I've gone and mentioned three different names for "very similar" languages in the same paragraph. When I mean one in particular from here out, I'll name it; otherwise Script refers to all three to the best of my knowledge. Anyway...

One of the things I find myself doing, and reading in just about everybody else's code in most languages, is a prudent sanity-check validation before doing something "dangerous" that profoundly changes the state of the system based on the state of various objects within it. A decidedly non-trivial part of that is usually simple validation of individual property values; is the someFutureDate value actually greater than the value for Now? Should we check that every time we're about to make an important decision based on the value of someFutureDate? What a pain…especially knowing that the one time you forget to do that check often turns out to be the root cause of a critical bug?

ECMAScript 5 has completely rethought how properties on objects are implemented. They can be made read-only, non-enumerable, non-modifiable once set. Perhaps more interestingly, they can now be defined either using a simple value as before, or by defining getters and setters, as is done in several other languages. What this buys you, at the cost of a bit of relocated complexity, is the ability to validate assigned values without the assigning code doing anything other than assigning a value to the property. The domain logic relevant to a specific property of an object can now be coupled more tightly to the property and the details less visible or relevant to outside code, in effect creating a conceptual "mini-class" around a property and its getter/setter logic that is transparent to outside code.

This and several other features of ECMAScript 5 now make it easier to write nice, fine-grained, SOLID code in ECMAScript 5 than was previously possible in any Script dialect. Huge win all around.

Which brings up questions. How could support for property descriptors and the like be added to CoffeeScript? Should they be added? Would support for these underlying ECMAScript features require changes to the CoffeeScript compiler itself, or could it be achieved less intrusively?

Even Typos Can Teach You Something: BE CAREFUL!

CoffeeScript is an interesting language, in both senses of that word. Whereas it tries to (and largely succeeds in) insulating the hapless coder from what Douglas Crockford calls "the bad parts" of JavaScript, it does not filter out all of the mind-blowing bits of JavaScript. Arrays are a good example.

Think back to your CS 101 class (or your K&R for you self-taught folks). What is an array?

In most languages, an array is a numbered sequence of elements, using (usually) sequential non-negative integers for identifying a specific element. Many statically-typed languages (such as C and Java) require that array elements each be of the same type; dynamic languages such as Python relax that to one degree or another.

But CoffeeScript and its underlying JavaScript (which I collectively call just Script) suck that up and then do some anatomically improbable things with/to it. Consider this bit of CoffeeScript interaction from a terminal (in Mac OS X 10.8, CoffeeScript version 1.2.0):

Jeffs-iMac:tmp jeffdickey$ coffee
coffee> foo = []                      # declare an empty array
[]
coffee> foo[4] = 'a'                  # assign offset 4; 0-3 have undefined values
'a'
coffee> foo[2.718281828459045] = 'e'  # Floating-point array index? Why not?
'e'
coffee> foo[-2] = 'b'                 # Negative numbers are just fine, too
'b'
coffee> foo['c'] = 27.4               # A string can be an index. It's still an array.
27.4
coffee> foo                           # What do we have now?
[ ,
  ,
  ,
  ,
  'a',
  '2.718281828459045': 'e',           # Non-sequential offsets look a lot like object fields
  '-2': 'b',
  c: 27.4 ]
coffee> foo.length
5
coffee>                               # Control-D to get out of the REPL
Jeffs-iMac:tmp jeffdickey$

So? What's the upshot?

Well, one thing many languages do to/for you is to throw an exception when your program gets too fast and loose with its indexing into an array. In Script, that doesn't happen; any scalar is a usable index, and if you pass something in that isn't a scalar (like an object), it'll have its toString() method called to generate an index value. And did I mention that this is all case-sensitive? Your typo opportunities are limited only by your imagination and by the accuracy of your fingers…

In case it wasn't obvious from the preceding incredulous rant, I really wonder why this "feature" is in the lnaguages; would it have really been that coercive to say "I'm sorry, Dave; I'm afraid I can't do that. Perhaps you'd rather use an object hash instead?'

An Immodest Proposal: Show Me the Code

I've been doing a lot of CoffeeScript work lately, along with Ruby with Rails and Sinatra. Especially with regards to my CoffeeScript, I've moved away from my recent Ruby-coloured "your code should be all the documentation you need" philosophy.

I've found all kinds of uses for Markdown, particularly including docco.coffee, written by the developer of CoffeeScript itself. Docco "is a quick-and-dirty, hundred-line-long, literate-programming-style documentation generator. It produces HTML that displays your comments alongside your code. Comments are passed through Markdown, and code is passed through Pygments syntax highlighting." It may not be the greatest thing since sliced bread, but the output does make complex yet deliberately-written code much clearer.

The downside to well-commented code, of course, is that it gets bulky. One of my larger CoffeeScript files has ~140 lines of actual code, engulfed in a source file that currently tops 330 lines. Ouch.

I was working on it just now, and an idea popped into my head: the "proposal" of the title.

Comments should be selectively elided or folded, in a fashion similar to the code folding feature offered by most modern editors.

I meander around between four different editors on my Mac (TextMate, Sublime Text 2, Komodo IDE and MacVim), and none of them appear to support such a feature out-of-the-box.

Does anybody know of a plugin for any of the above that does this?