Showing posts with label architecture. Show all posts
Showing posts with label architecture. Show all posts

Saturday, 7 June 2014

Rails, Circular Logic and Incremental Big Bang Architecture

(Initially published on my Tumblr, which I'm finding increasingly usable. Thoughts on Tumblr vs Blogger.com?)


Rails is a fantastic solution to a well-understood class of problem: the CRUD-oriented (or REST-oriented) query/report/update Web application, circa 2004–2006. Its philosophy of curation over configuration, despite having been likened to the report-producing languages of old (such as IBM RPG II, means that you really only have two things to worry about: your domain business logic and page transitions (which is another way of saying “your domain business logic, as it applies to presentation”).

And yet, the farther your app gets from that sweet spot,, either by decoupling the UI and using a fancy JavaScript framework; by needing to support user inputs other than typing into form fields and clicking buttons; or even the distance imposed by being nearly ten years on from the “sweet spot” and so many tools and other bits of code have changed around it — the harder it gets to do what you want to do in The Rails Way. Lots of people much smarter than me, from Robert C “Uncle Bob” Martin to Jim Weirich to Sandi Metz to Matt Wynne and a host of others, have spent many person-months thinking about, grappling with and writing about how to mitigate Rails, the knots it ties you into and the practices that much of the rest of our craft has built consensus on as being essential for good practice, that you can’t use in Rails without superhuman effort. Because the apps we’re building now, even the relatively simple ones, aren’t largely analogous to Basecamp (the application from which Rails was originally extracted).

So here’s the problem I’m facing, and the problem you’ve more than likely faced if you’ve done a non-trivial app in Rails.

If you’ve done any medium-to-large-scale development in other systems, especially in a decent OO language, one of your high priorities is going to be separating your domain logic from your “infrastructure” logic, including layering your presentational logic related to your data models separately from the data models themselves. Small is beautiful, and Ruby has some great tools to let you do that, including in particular the whole universe of Gems. Rails, however, makes it bloody hard to build what the rest of the craft would view as modular apps that follow SOLID principles. The retort to that is generally along the lines of “Rails is omakase” — which is fine as long as your app is living within that circa 2004–2006 sweet spot I mentioned earlier. The clock in the upper-right corner of my display here informs me that today is Saturday 7 June 2014 as I write this. Oops.

So we have great ideas like Hexagonal Rails , Architecture: The Lost Years and so on, that showed ways, with varying degrees of universality, to “get a handle on” the application/system you were developing.

But, can you answer what your application does? Not “what is its data model”, “what are the different types of users and permissions”, but a pithy sentence or two that gets across the idea of what makes your application different than every other CRUDtastic app out there, Rails or otherwise. Because if you can’t, none of this Rails-survival lore will help you.

And, equally importantly, as Jim Weirich points out in his Cincy.rb talk [Decoupling from Rails](http://youtu.be/tg5RFeSfBM4), is the question of “has this ever been successfully done before? If so, how? If not, what’s been the stumbling block that we have to stand back and work around using something else?” Because, in all fairness to Uncle Bob, if there aren’t examples of his built-on-Rails architecture in the wild for people to poke at, projector-ware can be inspirational but not terribly practical. For instance, as Jim points out at about 10 minutes into his presentation, Uncle Bob’s architecture shows where Rails controllers and views go, but ActiveRecord “models” don’t seem to be there at all*. Separating domain logic from persistence just happens to be one of the hardest nuts to crack in a Rails app to begin with. Hmmm.

Is the solution more decoupling? Jim spends the rest of that talk isolating and extracting what several teams publish as “interactors”, or support for what I call in my own apps “domain service objects” or DSOs. These help, a lot; having “skinny” controllers that don’t really know anything at all about your app beyond how to hand data off to classes that do lets you refactor your business rules efficiently by putting their control logic all in one place separate from Rails. Similarly, implementing façades or “decorators” with presentational logic around your models, as with Draper and others, helps you write clean, business-logic-free view templates.

But you’ve still got ActiveRecord to deal with in your models, and it leads you down the garden path and over the cliff with things like, oh, relations. When you have, say, a Blog model that has_many :posts and a Post model that belongs_to :blog, what you’ve got is a circular dependency; you can’t make changes to your Posts with certainty that those changes are isolated from your Blog, or vice versa, because each depends on the other. There have been numerous serious attempts to effect such a decoupling; perhaps the most widely known is Ruby Object Manager, whose Version 2 has been in the works for over two years now. It’s led by one of the greats in the Ruby community, Piotr Solnica, but it’s a Hard Problem. Especially when coexisting with Rails. People even resort to uncomfortable hacks like this to Get Things Done. There has to be a better way, and monorail (traditional Rails all-in-one non-modular app) is not that way.

And there are other, seemingly orthogonal issues to deal with. Authentication? Authorisation? Several solutions that are widely used exist for these in the “traditional” Rails “monorail” application style; how will adapting your architecture affect these, and how will you implement them?

Anybody?

Wednesday, 19 February 2014

Going Forward; Avoiding the Cliff While Looking for Cliff's Notes

Over on the Ruby sub-Reddit, I got into a discussion (mostly with Jelly_Jim, the OP, and with realnti) about the conflation in far too many minds of Ruby, the programming language, with Ruby on Rails, the famously-"opinionated" Web application framework. While pontificating on Jelly_Jim's original question, I described my frustration-bordering-on-antipathy with the traditional (known as "The Rails Way") application structure; listed a few presentations, books and blog posts that have influenced me greatly, and described my own take on the Kübler-Ross model for Ruby-based (and at least initially, largely Rails-based) Web development.

That discussion, over the course of roughly a day, helped me figure out how our app is going to survive the expected Slashdot effect, transitioning from a pretty traditional, megalithic Rails app to an app that uses Rails components in a sharply different, more recognisably "object-oriented", service-based architecture.

Leaving aside my more prosaic experiential difficulties with Rails (I really loathe ActiveRecord as commonly used), and taking into account the "Uncle" Bob Martin keynote, the various blog posts I referenced in the original Reddit post, and a couple of books I mentioned in a later reply, I think I've possibly hit upon a way to get where I want us to go.

That's the easy part, of course; it only took me never-mind-how-many months. The "interesting" part is going to be getting there — understaffed, on a shoestring budget for the moment even by startup standards (another problem I'm working), and working a schedule for two and a half years that would land a healthy twenty-year-old in hospital by now. Comments/suggestions are, as always, greatly appreciated.


Basic Architectural Principles

  • Loosely-coupled, granular architecture FTW. That means, inter alia,
    • Nearly the entire app to be packaged as Gems used by an otherwise minimal application;
    • Rails mountable engines to package assets (particularly CoffeeScript/JavaScript) and major components;
    • Plan and design for updating and provisioning components which communicate across natural architectural seams.
  • Hexagonal, "ports-and-adapters", or "clean" conceptual architecture; all dependencies point inward. Your domain entities, at the centre of the system, will change less than details like the delivery mechanism (Web UI, phone app, etc) or the database (which the domain shouldn't even be able to prove exists as such).
  • By adopting a heavily job-story-oriented view of the workflows; with the use of online tools like Trello, WebSequenceDiagrams.com; and with supporting code tools like ActiveInteraction and so on, we should be a lot more productive and a lot more Agile than we have been during our forty (calendar time: two) years lost in the wilderness.
  • And oh, yeah, just to keep things interesting: we've got to keep regularly demonstrating forward user-visible progress, for all the obvious reasons and a couple I really can't go into here.

How We Get There, v. 28.0

First, since the major focus of late has been on some unique (and not-so-unique) CoffeeScript code, start there. Separate the existing Script code out into four engine-based Gems, following/adapting Derek Prior's example. These levels are, from conceptual bottom to top:

  1. A "gateway" CoffeeScript class we've written as a poor man's RequireJS, that gives us just enough pseudo-dynamic referencing of data/code without changing anything the Rails asset pipeline assumes;
  2. Our directly DOM-aware Script code, as one or possibly two Gems (one is a major crown jewel; we might decide to split off utility/support code into a separate Gem, so as not to touch one when maintaining the other);
  3. The code proxying our internally-used message queue (decoupling "FTW", remember?) and the Ajax code to communicate with the Rails back end; and
  4. The code implementing various domain/service objects sitting atop the Script stack; one Gem per service.

Next, adapt the relatively-straightforward (as yet) Ruby/Rails code to honour the architectural principles listed earlier. This is where tools like ActiveInteraction pull their weight (and solve several existing pain points in the process).

Just to make sure we really do understand where we're going with this, take our first user story-specific code (including interactor) and package it up as a Gem that should then Just Work as in the previous step. Start thinking about whether we really want traditionally-packaged Gems or unbuilt dependencies that remain part of the single app repository.

Proceed to reimplement and repackage the (again, relatively few) job stories already existing as was done in the preceding step.

Start knocking down the remaining Trello cards as we complete our app.

We've been saying since the Very Beginning that anything that we were building that wasn't inextricably tied to proprietary, commercially-valuable code (or that gives too much of a hint into any unique details it may have) "should be" openly released, but hadn't yet figured out a feasible, cleanly reusable way to do that. If we've done our jobs properly up to this point, we have that way now.

Push the "deploy and launch" Big Green Button. We've bloody well earned it by now.


Any Suggestions or Comments?

Monday, 1 July 2013

"Easy Credit" IS "Easy Debt". It sucks for finance. It sucks just as much for your code.

One of the great things about the Rails ecosystem is that there are lots of different tools that can save you time and pain during development. Some of them work by embracing and extending the Rails standard bits, like replacing (nearly) bog-standard ApplicationController subclasses with subclasses of InheritedResources::Base. Some of them instead swap out pieces of the original Rails stack, like replacing ActionView with Erector or something similar. I wouldn't be at all surprised if my current app that mixes the two practices (in different areas) isn't at all unusual.

But whichever route or routes you choose, you need to constantly remain aware of that choice, especially when considering new tools you'd like to incorporate into your app. And you may also find that the tools you're using, while superb for their intended purpose, complicate other choices you make in the course of developing your project. A tool which was the greatest painkiller since morphine when you first picked it up, now seems to be "getting in the way" as you get farther along in development.

As practitioners of agile development, this shouldn't scare or even surprise us (though our non-technical stakeholders may initially be less sanguine). Good agile architecture says that everything in your app other than the raw business rules is an implementation detail that should be testable and fully replaceable in isolation.

Novice to journeyman Rails developers may be forgiven at this point for saying "but that's the way Rails is; it all works together, even though you can swap pieces out as you like". A majority of Rails developers have at least experimented with swapping out Test::Unit for RSpec, or erb for Haml or Slim. More adventurous still are the folks who swap out the ORM, replacing ActiveRecord with, say, DataMapper or the aforementioned Erector for ActionView. Why are they trying such things? Because they're asking (quoting Wynne again), "what does a modular Rails architecture look like?" Or, "we know we've hit the wall with our current (connected) architecture; everything we touch introduces new tech debt if not outright breakage, but we don't know how to do anything else and stay true to The Rails Way."

That "wall" you've hit has nothing to do with the number of lines of code you've written nor the number of unique visitors your production app serves per day nor the revenue your app is bringing in the door. You've realised that your ability to keep bringing in that revenue, let alone increasing it, is dependent on you somehow reconciling your understanding of "agile development" and "software engineering-style practices" with "the Rails Way". One of these things is not like the others. If you haven't yet, watch both of those videos. Particularly in Matt Wynne's talk from about 5:30 where he talks about the crossover point between a "connected" architecture being cheaper/faster/better giving way to a "modular" one… that rings a lot of bells for me. Likewise when Uncle Bob rails against Rails apps screaming "I'm a Rails app, never mind what I do". I've done several projects, never mind how many, where I've gotten to that point (which Wynne describes as "hang on; I don't want to work on that application anymore. Something's seriously wrong with it") and my reaction to the realisation that "something is seriously wrong" has been to stop, look at everything I'm doing and evaluating how well I'm following convention, "best practice" and so on, come to the conclusion that I'm doing everything exactly the way I've been taught it should be done, and deciding that that proves the fault is in my understanding of the problem, so I start over, from scratch, trying to figure out where I fell off the cliff. With one application, that involved at least five restarts over the course of a year and a half… and I must have always "fallen off the cliff" at the same spot, because I never saw it coming before I realised I was at the bottom of that cliff once again. "Uncle Bob" Martin's keynote (from Ruby Midwest 2011) does a great job of explaining why an experienced OO software developer like me has always felt uncomfortable with Rails' interpretation of the "Model-View-Controller" architecture. By writing your app centred on what the app is for rather than how the app presents itself on the Web, you make your life a lot easier and more enjoyable. Looking back at my commit history over the last year and a half, I see that well over 2/3 of the times I was "stuck" for more than a couple of hours had more to do with "how do I adapt what I'm trying to do to how Rails wants me to do things" than "how do I implement (this feature)". That should, and has been, setting off all kinds of alarm bells; I'd just drank enough of the Flavor Aid to think that pain level was "normal" for Rails. It shouldn't be.

Going forward, my strategy for Rails development is going to work something like this:

  1. Model the domain objects (Martin's "interactors", "entities", "boundaries" and "entity gateways", after Jacobsen "control objects") as POROs (Plain Old Ruby Objects);
  2. Wrap those up in Gems whose use can be demonstrated by specs, supplemented by a simple (Web or JSON or command-line; doesn't matter) example or set of examples.
  3. Plug those into a Web app using the current framework of choice (Rails, Padrino, JoeBobsAwesomeFramework, or whatever), whose sole purpose is to implement the "client" side of the boundary layer and present what comes back over the fence.

Now we're back to something more closely approximating classical agile development that focuses on delivering business value, not playing mind games with a tool that insists on making most of the important decisions for you, and makes them in ways that make it harder for your app to be scalable, performant, or fun to work on. Or even for your app to be finished.

Thoughts?


P.S. Oh, and as far as InheritedResources::Base, that truly useful tool for Rails app development I mentioned before? We'll continue to use it as a default, but if we choose to use tools that "get in the way" (like wisper) by wanting all your methods explicitly defined, we'll switch back to ApplicationController, on the theory that making the puzzle one piece smaller without affecting how the other pieces fit together is a Good Thing™. But that's another decision we want to defer as long as possible; "good architecture" is all about deferring as many decisions as possible as long as possible, while still getting business value done.

Sunday, 26 May 2013

FOCUS! Oooh, shiny, shiny...

That didn't last long; having a nice set of components to build on and being satisfied that it would carry us forward indefinitely…

As Ruby developers on Rails or Padrino or bare metal, we're absolutely spoilt for choice. As I write this, Rubygems claims 1,577,138,099 downloads of 56,183 gems since July 2009. (Compare to ~495 PHP packages in the official PEAR repo on Github.) If you can figure out what you're looking for well enough for a Google search, chances are excellent that you'll find something you can at least use; likely several somethings. As with all such, this is both a blessing and a curse.

The blessing is that the stuff is out there; it is mostly at least usable, especially for the gems that are at or past Version 1.0 (yes, there's a formal numbering system for gems); and if it doesn't do precisely what you want, chances are likewise excellent that you can wrap your own code around it to do your bidding, and/or pull a copy of the official source, make changes you'd like to see, and submit a pull request to tell the original developer(s) what you'd like to see pulled into the "official" version. ANd so on.

Back a couple of years ago when I was doing PHP for a living, it wasn't that hard to keep a rough idea of what was both available and usable on PEAR in my head, so that as I developed a project, I knew what I had available and could try to structure my code to work effectively with it. It's a lot easier when you can reverse that: think of what you want and how you'd like to work with it, and go run down the list of candidates to plug that hole.

But that touches on the other edge of the sword, and it's sharp. Doing all that browsing and cataloguing and selecting and integrating takes time and effort; time and effort that your customers (external or otherwise) generally assume you're putting into making a kick-ass product. Well, you are, sort of; you're just trying to figure out which giants' shoulders to stand on. Throw in the usual social media like Google+ and Twitter and so on, alerting you to the newest and shiniest of the new-and-shiny, and it's a wonder you get any "real work" done.

Memo to bosses: This is real work, and it's almost certainly less expensive real work than having your folks reinvent several dozen (or several hundred) wheels as they develop your Awesome Product, Version next.0. Deal.

But the "is it done yet?" bosses do have a very good point, devs. Promises have been made; contracts have been let; people's jobs (including yours) may well be on the line. Focus is necessary. You just have to learn to do it in a reasonably structured way, like your RSI breaks.

When you come across the latest shiny-shiny that you just know is going to make your job so much easier, better, more fun and profitable, add it to a list. When you're about to start on some new part of your project, glance at the list and see if it makes sense to use your new feature-thingy as a test bed for the new gem. Be careful that it can peacefully coexist with the rest of your codebase that hasn't been updated yet, and do remember to use a new feature branch. Set yourself a time limit for how long you'll allow yourself to get the new gem working. When that's expired, give yourself at most one extension (hey, you're exploring the not-yet-known-to-you, after all) and if it's still not walking and chewing gum at the same time, go back to where you branched and try again, the "old" way. This is so simple, yet not doing it can have a devastating effect on your schedule (and project, and…)

But given a choice between running the risk of distraction by a myriad assortment of Things That Generally Work, and trying to make one of a smaller number of Things That Should Work But Don't, Quite… well, that's what brought so many VB6 and Java folk to PHP in the first place, and what brought so many PHP people (including yours truly) into Ruby. Given sufficient resources and patience, you can make almost anything work in almost any language; I once saw a quite-playable version of the classic Star Trek game written in RPG II. I thought, and still think, that the guy who wrote it was absolutely nuts (especially since he knew at least three other languages that would have been easier/better aligned to the job), but *it worked*. So, as the adage goes, never argue with success. However, given the chance early enough in the project for it to make a difference, do respectfully suggest that jobs are best done with the available tools most closely aligned to the job. RPG is still in production for what it does best: complicated report generation from highly-structured data. PHP remains a valid choice for small projects, or for those companies unwilling to rely on non-local talent and willing to trade initial cost for quality. Ruby has numerous tools available, for just about anything, that challenge its practitioners to be better developers than they started out to be. Part of that challenge is the sheer wealth of tools available, and their haphazard discoverability. Part of it is looking back at code you wrote a few short months, or even weeks, ago; knowing it could be so measurably better; and having the self-discipline to leave it be until you have a business reason to change it.

Problems like that make my days more enjoyable than solutions in other languages generally have. Now if I could just turn this megalithic Rails app into something more componentised, service-supporting. Oooh, shiny, shiny…