Rules can be broken, but there are Consequences. Beware.

If you're in an agile shop (of whatever persuasion), you're quite familiar with a basic, easily justifiable rule:

No line of production code shall be created, modified or deleted in the absence of a failing test case, for which the change is required to make the test case pass.

Various people add extra conditionals to the dictum (my personal favourite is "…to make the test case pass with the minimal, most domain-consistent code changes currently practicable") but, if your shop is (striving towards being) agile, it's very hard to imagine that you're not honouring that basic dictum. Usually in the breach at first, but everybody starts in perfect ignorance, yes?

I (very) recently was working on a bit of code that, for nearly its entire existence over several major iterations, had 100% test coverage (technically, C0 or line coverage) and no known defects in implemented code. It then underwent a short (less than one man-week) burst of "rush" coding aimed at demonstrating a new feature, without the supporting tests having been done beforehand. It then was to be used as the basis for implementing a related set of new features, that would affect and be affected by the state of several software components.

That induced some serious second-guessing. Do we continue mad-hatter hacking, trusting experience and heroic effort to somehow save the day? Do we go back and backfill test coverage, to prove that we understand exactly what we're dealing with and that it works as intended before jumping off into the new features (with or without proper specs/tests up front)? Or do we try to take a middle route, marking the missing test coverage as tech debt that will have to be paid off sometime in The Glorious Future To Come™? The most masochistic of cultists (or perhaps the most serenely confident of infinite schedule and other resources) would pick the first; the "agile" cargo-cultist with an irate manager breathing fire down his neck the second; but the third is the only pragmatic hope for a way forward… as long as development has and trusts in assurances that the debt will be paid in full immediately after the current project-delivery arc (at which time increased revenue should be coming in to pay for the developer time).

The moral of the story is well-known, and has been summed up as "Murphy " (of Murphy's Law fame) "always gets his cut", "payback's a b*tch" and other, more "colourful" metaphors. I prefer the dictum that started this post, perhaps alternately summed up as

Don't point firearms at bits of anatomy that you (or their owner) would mind losing. And, especially, never, ever do it more than once".

Because while even the most dilettante of managers is likely to have heard Fred Brooks' famous "adding people to a late project only makes it later" or his rephrasing as "nine women cannot have a baby in one month", too few know of Steve McConnell's observation (from 1995!) that aggressive schedules are at least equally delay-inducing. If your technical people say that, with the scope currently defined, that something will take N man-weeks, pushing for N/4 is an absolutely fantastic way to turn delivery in N*4 into a major challenge.

Remember, "close" only counts in horseshoes and hand grenades and, even then, only if it's close enough to have the intended effect. Software, like the computers it runs on, is a binary affair; either it works or it doesn't.

It's Not Amtrak: Enjoy Riding the Rails

Historical note: Most of this post was written in the first week of October, 2011.

---

I've spent much of the last three weeks wrapping my head around Rails and Ruby after several years of mostly PHP with a couple of Groovy/Java side trips. It's been interesting, and a lot of fun so far.

I'm new to Ruby on Rails, but not to Ruby itself — though I took a long breather when Rails first became trendy. It was not at all clear to me at the time whether the (then-)popular trend among both the media and technology-involved people who Should Know Better to treat Ruby as essentially a synonym for "Ruby on Rails" was going to kill the language for other uses. It was quite clear to me that a huge chunk of early Rails culture was built around a cult-of-personality dedicated to Rails' primary creator, David Heinemeier Hansson. I try very hard to avoid cults; they almost invariably either spectacularly self-destruct (see Jonestown, Guyana) or get co-opted to serve unrelated or even opposing purposes (see State religions).

In about the last year or two, Rails has visibly grown up, both in capability and in outlook. There are other people who've stepped up and delivered great, widely-used code (does anybody know what happened to _why?), and Rails is now used in lots of systems both interesting and serious. I'd been thinking that maybe I ought to start getting myself up to scratch, particularly as I've been increasingly disenchanted with both the technical and sociopolitical aspects of PHP. The feelings of excitement, progress and everybody-pitch-in community that sustained PHP through its phenomenal growth in the last decade are, at best, flickering.

The final kick in the shorts was an interview cycle for new work with a great group that does, among other things, quite a bit of Rails work. Thus motivated, I grabbed copies of Agile Web Development with Rails, Fourth Edition and The RSpec Book and started going through them.

And promptly went out in search of better books. As of April, 2012, the ones I refer to most often are:

• Russ Olsen, Eloquent Ruby. Addison-Wesley Professional, 2011. Nearly 400 excellent pages that will help your Ruby code be more effective, expressive, pleasant to work with and easier to maintain; pick any five.
• Obie Fernandez, The Rails 3 Way; 2nd Edition. Addison-Wesley Professional, 2011. An encyclopedic yet guided introduction and continually-usable reference to Rails 3. Does not cover everything in 3.1/3.2, but will leave you comfortably able to adapt to the new changes.

One of the things I appreciate about Rails is the size and openness of the communities that have grown up around it. Everywhere I look, I find new code and tools to do useful things. I'm still on the upward slope of the "each Neat New Thing points to n other Neat New Things which point to…" curve, and I expect to be for quite some time. There are useful, useable if not always perfectly ideal tools to do things, and those tend to get adopted by (large chunks of) the community as de facto working standards. The whole "gem" ecosystem is chalk-and-Friday different (and dramatically better) than its PHP analogue; it's almost as though the "gems" folks looked at nearly everything PEAR gets wrong and made the conscious, deliberate decision to do the opposite. From the perspective of a newbie who's had painful experience with other systems, it Just (Seems Like It) Works.

Now, I'm sure that the rose-coloured spectacles will eventually pop out of their frames and get ground into dust. I'll find things I'm doing or using that just seem so bone-headed that I'll think, "I'm obviously well outside the ideal target audience for this". I seriously doubt that I'd think "This part of the system is so badly broken that it can't possibly be fixed"; again, this is a relatively new stack of tools with large, creative communities built around them. There will be alternatives to the particular whatever-it-is that's causing me pain, and there is a virtual certainty that I will be able to pull the offending piece out of my stack, put the new thing in, and continue working forward more effectively. I'm also convinced that, in the unlikely event that I'm such a black-swan corner case that I get motivated to write my own replacement, that it'll get visibility and feedback from others (if only to say "Hey, you just duplicated quteobscurename; go have a look at www.github.com/JoePalooka/quteobscurename — but hey, nicely done"). That would be more useful — and far more encouraging — than the usual PEAR mishmash of components that don't even all really work with the current language, written by people who believe that terse, sloppy, untested code is self-documenting.

---

Addition: April, 2012

And so it has come to pass; over the course of developing what will very soon be a commercial Web-based service, at least three initially-chosen components have been replaced by other, better and/or more suitable components. In all cases, the time involved in flailing around and deciding "this really has to go now" was several times greater than the hour or two it took to actually replace the old code with the new. That's been one of the big wins promised by object-oriented software development for roughly half a century. (Depending on who you ask, the first object-oriented language was either ALGOL X, in 1965, or Simula 67, in 1967. C++? Well over a decade later, and that's still older than most of its coders today.)

'Least Knowledge' May Be More than You Think

I was chatting on IRC the other day and got into a discussion that stuck in the back of my mind until now. This guy was a fresh convert to the ideas of SOLID and to the Law of Demeter (or Principle of Least Knowledge). Now, these are principles I hold at least as strongly as the average experienced developer, and we eventually agreed to disagree over the propriety of code such as a line I just wrote (which in fact prompted this post).

Consider the Ruby code:

    data[:original_content] = article.content.lines.first.rstrip

This says:

1. "take whatever your article is;
2. get whatever its content attribute is or method returns;
3. (assuming that is a String or something that can be treated as one,) get its lines enumeration;
4. get the first entry in that enumeration;
5. (again, assuming it's a String or workalike,) strip any trailing whitespace from that string; and
6. stuff it into the collection data, indexed by the symbol :original_content."

It's not hard to argue that this is a bad code smell, being a long list of assumed return types and so on, except for one thing: in my view, the LoD does not cover sequencing a series of standard library calls.

Look at the code again. The bit that references my code is the fragment

  data[:original_content] = article.content

That fragment is clean; article is an object and content is an attribute of or method on that object. This code assumes that the value of article.content is a standard String. As a standard class, it's virtually guaranteed not to change in backwards-incompatible ways over the useful life of the code. What then? Standard library calls!

1. String#lines returns a standard Enumerator instance;
2. Enumerable#first (via the earlier Enumerator) returns a standard String instance again; and finally
3. String#rstrip cleans off any trailing whitespace.

Nowhere are used any internal APIs that could change. Breaking that one line apart into 5 would produce repetitive, non-semantic, anti-Ruby-best-practices code of the style pandemic on large Java projects. We're in Ruby (and Rails) for a reason: part of that reason is that the language idioms encourage an almost literary expressiveness that makes more explicit the reality that a program is a conversation between the humans developing and maintaining it, that just happens to be executable by a computer. Literate, semantic programming is like behaviour-driven development; once you've wrapped your head around it and see how much better a programmer it makes you, you really don't want to go back to BDUF jousting in Java or PHP.

On further review...

Now, if you argue that the LoD is an aid in reducing coupling between classes, which reduces per-statement complexity, I can see that. But, in the example above, we're back to the fact that there is only one dot before you start jamming together standard library calls. Had I instead written:

content = article.content lines = content.lines first_line = lines.first data[:original_content] = first_line.rstrip

then I'd argue that I've introduced three unnecessary local variables when only one (the second) could foreseeably change. If I later modified the return value from article.content to something other than a Stringlike object, then probably the assignment to lines would have to change. But only that one line — the same level of change that would be required in my current code.

Comments?

If At First You Don't Succeed, Try, Try Again

In a very real sense, that's a big part of what we do in test-driven or behaviour-driven development, isn't it?

It doesn't only apply to classes and projects. Teams work that way, too. You can have someone who, on paper, seems just great to help you out with the cat-herding problem that's a large part of any significant project. The interview is fantastic; he's 110% buzzword-compliant with what (you think) you need The New Hand to be able to pick up and run with. There might be a few minor quirks, niggles; a voice in the back of your mind standing up and clearing its throat obnoxiously, and when that doesn't get your attention, starts screaming loudly enough that surrounding folk reach for ear protection. When that doesn't get your attention — because, after all, you're talking to such a fine candidate — the voice might well be forgiven for giving up in disgust and seeking the farthest corner of your brain to sulk in. And give a good kick to, every now and again.

And then Something happens, usually a string of increasingly serious, increasingly rapidly-occurring Somethings, and you realise that this really isn't working out so well after all. The candidate-turned-colleague may well be a very nice person, but you just can't communicate effectively. Or the "incremental" test that she writes requires changes to a dozen different classes that, in turn, break 15 existing tests. If that doesn't make you go 'hmmm...', then maybe your other team members are concerned about his professionalism: consistently showing up late to meetings; giving a series of increasingly outlandish reasons for not coming into the office and working from home despite having previously promised not to, and so on. Seemingly before you know it, the project is at a virtual standstill, team dysfunction has reached soap-operatic heights, and at least one vital member of the team starts running around and shouting "off with her head!" All of which does absolutely wonderful things to your velocity and project schedule. Or maybe not so wonderful.

And the funny thing is that, at about that time, you remember that voice in the back of your mind that was trying to point out that something was rotten in this particular virtual state of Denmark from the beginning; you were just too optimistic, and likely too desperately stubborn, to recognise it at the time. If you're going to go about hiring someone, have a good rapport with your instincts. Know when to walk away; know when to run.

Long story short, we are looking for an awesome new team member. As you might have gathered from the last few blog posts, we're a Ruby on Rails shop, so you'd imagine that heavy Rails experience would be tops on our list, right? Not so fast, friend. I/we subscribe to the philosophy that source code (and the accompanying artefacts) are part of the rich communication that must continuously thrive between team members, past, present and future. These artifacts are unusual in that they are in fact executable (directly or indirectly) by a computer, but they're primarily a means of communication. As such, ability to communicate effectively in English trumps wizard-level Ruby skills. If we can't communicate effectively, it doesn't matter how good you are, does it?

What we're looking for at the moment can be summarised thusly:

1) Demonstrated professionalism. This does not mean "a long string of successful professional engagements", though obviously that doesn't hurt, especially if they individually lasted more than the sensible minimum. Do you generally keep your word? (Shtuff happens to everybody; nobody's perfect, but overall trends do matter.) Are colleagues secure in the knowledge that you routinely exceed expectations, or are they continuously making contingency plans?

2) Fluent, literate standard English at a level capable of understanding and conveying nuance and detail effectively and efficiently. This would be first on the list, were it not for the series of unfortunate events that led to this post being written at all. It is an absolutely necessary skill for each member of a truly successful team, and a skill that is becoming depressingly rare.

3) Customer-facing development experience, ideally in a high-availability, high-reliability environment. We're building a service that, if we're successful, people are going to "just assume" Works whenever they want it to, like flicking on the lights when you walk into a room at night. Those of us who've been around for a while remember the growing pains that many popular services have gone through; we have no plans whatever to implement a "fail whale".

4) Ruby exposure, with Rails 3 a strong preference. You can fill in along the edges as we go along, but if we can't have an informed discussion of best practices and how to apply them to our project, you're (and we're) hurting. If you're unclear on the concepts of RSpec or REST or Haml or _why, you're going to have to work really, really hard to convince us that we're not all wasting time we can't afford to. And, finally,

5) Deep OO experience in at least two different languages. If you can explain and contrast how you've used different systems' approaches to design by contract, encapsulation, metaprogramming, and so on, and have good ideas on how we can apply that to making our code even more spectacular, rock on. "Why isn't this #1 or #2 on your list," you might ask. If you can convince me why it should be, and why your skills are so obviously better than anybody else I'm likely to have walk in the virtual door... then you've already taken care of the others, anyway.

We're a startup, so we're not made of money. But since we're a startup, money isn't all we have to offer the right person. Know who that might be? Drop me a line at jeff.dickey at theprolog dot NOSPAM com. Those who leave LinkedIn-style "please hire me" comments to this post will be flamed, ridiculed, and disqualified.

Hallowe'en II: Boxing Day

Here's a scary/stupid trick to pull if you, like me, are a finalist for Biggest Email Pack Rat On The Internet.

Open Apple Mail's Preferences, select the "General" tab, and change Dock unread count from Inbox Only to All Mailboxes. I double-dare you.

Or, it might just be too depressing. I went from a mere(!) 355 unread messages to…

56,232

Let's see, at an average of just under 2 minutes each (clocked over a recent week), cleaning that out would take me some 78 days, 2 hours and 24 minutes, during which some 31,000 new emails (not including auto-filtered spam at ~85% of total incoming email) would arrive. Assuming I could stay awake that long. Caffeine is The Elixir of Life™, but… "filling a baby's eyedropper from a raging waterfall" does not even begin to do the image justice. Maybe the old Ragú ad slogan, "It's In There".

There has to be a better way.

Cover Yourself: Toolchains Are Agile, Too

As people who know me professionally and/or read my blog know well, I have been a (raucously) loud evangelist for test-first development (TDD, BDD, Scrum, whatever your flavour) for years now. If I write even an exploratory bit of code and don't have tests in place first, I get very uncomfortable. As complexity increases, without tests (preferably automated, repeatable tests), I argue that I simply can't know what's really going on, because I can't prove it.

A major corollary to this is test coverage reporting. If I can't see what's been tested and what hasn't, then in a very real sense nothing has been, since I can't document/prove what has been and what hasn't. And the better (more productive) teams I've worked in have established, and regularly hit, coverage testing better than 95%, with 100% being a common (and commonly attained) goal. (Edit: Note that this is for C0 and C1 test coverage; tools that cover C3 and C4 are rare to nonexistent in most languages, such as Ruby.)

As you may also know, I've been getting (back) into Ruby development, using Rails 3 on Ruby 1.9. Ruby's long-time de facto standard coverage tool for many years was rcov, which generally worked well. However, Rob Sanheim has stated that "RCov does not, and will not support C based Ruby 1.9.x implementations due to significant changes between 1.8 and 1.9.". He recommends either SimpleCov or CoverMe. Ripping out RCov and replacing it with CoverMeSimpleCov on a test project took all of five minutes and left me with attractive, functional, (so far apparently) quite accurate reports.

One of the basic principles of agile development is that the team must actively embrace constructive change as their project evolves. It's often easy for harried, hurried people to forget that that applies to their tools as much as it does to what they produce using those tools.

Just a thought as my evening winds down.

Well, that was fast.

Like many of you, I expect, I take a good look around at tools like editors when my needs change dramatically. A new system, or a language or app type I haven't worked with in a while, and I'll go out and see what the community is using (or at least buzzing about), narrow that down to a list of 2 or 3 to try, and start trying them out. Usually, I'll try one for a few days and then switch to the next one for a few days, until I've got lists of things I like and don't like, and make a decision. The last time I took a serious look at this was a couple of years ago, when I shelled out for Komodo IDE (which I still enthusiastically recommend to PHP developers in particular, by the way).

Recently, I've started working again intensely with Ruby, on a Mac instead of Linux as years earlier, and some quick research looking around mailing lists, group archives, and such, strongly suggested that TextMate was The Gold Standard.

But I'd never really done a head-to-head comparison, and I knew that BBEdit and RubyMine had passionate evangelists in the community.

So I downloaded the 30-day eval of RubyMine and fired it up.

I tried to open a Git project that I've worked with extensively in TextMate and from the Git command line. One immediate issue: copy-and-paste between the Mac pasteboard and the edit fields in the RubyMine dialog did not work. I then opened the project directory using RubyMine's "open a directory" feature; it found Git all right, but gave half a dozen red flags and refused to work further.

"What could cause such a reputed package to crap out so completely," I asked myself as I started browsing around inside the app directory. The answer became quickly obvious.

"Oh, it's a Java app." No native UI at all — though to be honest, you have to look closely at the UI widgets to tell.

It took less time to install the app, have it flame out, and uninstall it than it did to write this post.

The sooner we either a) rid the desktop world of Java or b) get the Java community to adopt usable desktop interoperability, the better. I'm years past the point where I care which option is selected, but one had better be.

YAHA! (Yet Another 'Hey, Apple…') Hey, SingTel, you, too.

Hey, Apple…

I like what works, and what helps me work better/faster/more enjoyably; the more of these boxes that get ticked, the better. After all, that's why I'm sitting in front of two iMacs with a MacBook Pro and iPad close to hand.

However…

The new iMacs are a wonder. A 27" display, 2560 x 1440 resolution; absolutely gorgeous. I can have two full-page views plus a Terminal open on the same screen. Two steps forward for usability.

Now for one step back. The mouse cursor appears to be the same size as on my 15" MacBook Pro, even though I'm now looking at nearly three times as many pixels and, more importantly, screen area. How about taking a page from somebody else's playbook (for a change) and have some key sequence that would do a "radar-style" visual locator for the mouse cursor?

The Middle, or The Muddle, or Both

Seriously, I'm in love all over again with this new system. And if I weren't in one of the last Soviet-class customer-last economies, I'd be able to make even better use of the new tool.

With new hardware and software come new opportunities for learning. Anyone who's ever learned or discovered or done something new, or even new to him- or herself, has experimented; has pushed boundaries. Sometimes, they push back the first few times. "You'll know the pioneers because they're the ones with all the arrows in their backs" might be an Americanism, but its meaning is perfectly clear to anyone who's ever challenged Donald Rumsfeld's "unknown unknowns"; it's the things you don't know that you don't know that offer the greatest opportunities for learning — if you survive them.

Apple have always had a symbiotic relationship with leading-edge technical norms. The latest example, in OS X Lion, is that for the very first time since Macs have shipped back in 1984, no operating-system media (floppies, CDs, DVDs) come with the system. If you want to nuke from orbit and start over fresh, you use something called Lion Recovery. It's a sweet, eminently sensible idea: most Mac customers have access to fast enough Internet connections that downloading most of what you need to reinstall is less of a hassle than rooting around trying to find some discs that you just know you put in a drawer. Somewhere. Maybe it was in your home office. Maybe in the away office. Maybe they're in one of those bulging boxes marked "Computer Stuff" up in the attic. It could easily take you longer to find physical media than to download four gig or so of bits on a reasonably-modern, properly-managed and -provisioned connection.

Hey, SingTel…

And then there's Singapore.

Advertising here is (in)famous for always having the language "Terms and Conditions Apply", which in practice seems to mean "We don't have to do a single thing we said we would once you give us the money if we can think of a reason not to, or if we do deign to provide the promised product or service, we expressly reserve the right to make the experience as unpleasant as our most innovative Government Scholars™ can conjure up".

Case in point: said Lion Recovery. The way it appears to work is that when your Mac phones home to Cupertino, it then fires up a for-purpose Web server that listens on the usual ports for authenticated incoming connections (from Apple). Cupertino or its CDN upload the data to your Mac, which uses it to reinitialise your installed system. This only works, of course, if your Mac is connected to a network that allows you to do such things, without being blocked by either policy or incompetence. Ah, those terms and conditions.

The Singaporean Interwebs are full of woe documenting Apple customers' fate as victims of SingTel policy and/or incompetence, with Lion Recovery being the latest poster child for the meme. SingTel and StarHub, two of the three major telecoms companies here (and the two popularly thought of as more closely tied to the PAP Government) have intermittent-or-worse problems, while the third (M1) is apparently less hassle. People speak of tethering their Macs to their M1 phones and spending over five hours (in at least one case) successfully performing Lion Recovery. This would be greatly preferable to not being able to recover at all. Unfortunately, the M1 signal in the HDB tenement-with-pretension that I'm living in is too weak and unreliable to support that.

So I'm back to waiting, impatiently, for SingTel to get their cranial appendage out of whatever orifice it's presently stuck in, and work with Apple on fixing the problem. Apparently Apple have run against the proverbial brick wall on multiple occasions to try to help solve a problem over which they have zero control.

Hey, SingTel…

ANFSD: Fascism Bites Everyone In The Pocket (Among Other Places)

Fascism should more properly be called corporatism, for it represents the fusion of State and corporate power.

— B. Mussolini

If you've lived in the USSR, former Soviet republics, or much of south Asia (including Singapore), you're quite familiar with the concept of "exclusive distributors", which the Free World seems to have thrown on the ash-heap of history at roughly the same time as leaded petrol. For those who've never had the displeasure, it works just like it sounds: one appointed business entity is the only way for subjects of a particular country to lawfully access the products or services of a particular (foreign) company. In places like Singapore which follow a State-capitalism model, that exclusive agent customarily has strong ties to either the Government or to entities acting as agents of the Government (sovereign-wealth funds, public officials who also run nominally-private-sector companies, and so on). This rarely, if ever, is a boon for the consumer.

Case in point: today, I wanted to buy a good fan, a Vornado Compact 530, comparable to the Vornado fans I owned when in the States.

Naturally, I can't buy a fan from the Vornado Website, though it gives me plenty of information about the thing. A little search engine-fu tells me that Home-Fix are the exclusive distributor for Vornado in Singapore.

Naturally, I can't order anything, or even get any product information, from Home-Fix's site. I could, however, get the phone number(!) for their nearest store, and called to enquire about availability and pricing.

The conversation went something like this:

Me: Can you tell me if you have Vornado fans?

Clerk: Yes, we do. Which one are you looking for?

Me: The "Compact 630".

Clerk: Yes, we have that one, in white and black.

Me: How much is it?

Clerk: S$170.

Me: $170?!? So your rate on the US dollar is about three Singapore dollars, then? The list price on the Vornado Website is US $49.99!

Clerk: (as to a very small, very slow child) Oh, that's the online price. It's bound to be cheaper.

Me: Well, I've done some checking on various US, European and Australian stores; the walk-in retail price is right about the same.

Clerk: Well, our price is S$170.

Me: Well, thank you for your time.

Not to be terribly unfair to either Home-Fix or the clerk; that's the way the system operates here, and any company that didn't jack their prices up to whatever the marks will pay isn't doing things The Singapore Way. It's not as though it's actually people's own money; they're just holding it until those who pull the levers of State decide they want it back2.

So, ragging at Home-Fix or any of the many, many other businesses whose prices have no apparent correlation to corresponding prices elsewhere won't accomplish anything. If, as was let slip during the Singapore Presidential "election" this year, the Government's sovereign-wealth funds and people connected to High Places really do control 2/3 or more of the domestic Singapore economy, then complaining about any individual companies is rather like the man who's been attacked by a chainsaw-wielding madman who then worries about all the blood on his shirt. Fix the real problems, and the small ones will right themselves.

Incidentally, this also illustrates why I support the Occupy movement. If Americans want to see what another decade or two of economic and political polarisation between the top 400 families and the rest of the country will look like, Singapore is a good first-order approximation. And some sixty percent of Singaporeans apparently either couldn't be bothered to think differently, or were too afraid to.

Sigh. I still need to find a reliable, efficient fan.

Footnotes:

1. According to the current XE.com conversion, 170 Singapore dollars (S$170) is approximately US$130.19,, or some 260 percent of the list price. In a free society, that would be called "gouging"; here in Singapore it's called "buying a foreign product". Remember, no foreign companies really operate here. When you walk into a McDonald's or Starbucks or Citibank in Singapore, you're walking into a local, almost invariably Government-linked, franchise or local representative. The quality and experience difference can be, charitably, stark. (Return)

2. Known in the US as the "thug defense", after a Los Angeles mugger who used that line of "reasoning" as the basis for an attempted pro se defence in court. He was, of course, unsuccessful. (Return)

Know when to walk away; know when to run

This started out as a reply to a comment on the LinkedPHPers group on LinkedIn; once I started writing, of course, it quickly grew beyond what was appropriate as a conversationally-inline comment. So I brought it over here. It's something I've been thinking about for a couple of weeks now, so let me get this anvil off my chest.

---

To R Matthew Songer1: I'd advise adding Ruby on Rails to that list of alternatives. I've been writing PHP for fun and profit since PHP 4 was still a future hope wrapped in hyperbole. Now that we've finally got a (mostly-)decent language in 5.3 for some serious software development, I'm burning out. Part of that burnout is due to what I see in the PHP community, part to geography, and part to other factors that approach "fit for use".

I've recently taken a post as Chief Engineer at a little startup nobody's heard of yet. Our prototype was done in PHP; it got the idea across well enough for our initial investors and customers to bang on our door. So the CEO and I thought, great, we'll find another senior2 PHP guy, write a real app (the prototype doesn't even have testable seams), and we're off.

If I were looking to hire a battalion of deputy junior assistant coders whose main qualification as PHP devs was being able to spell it whilst taking endless trips down the waterfall, I could do that. I was aiming higher: I wanted someone who knew his way around current best practices; who realises that being experienced is no excuse to stop learning aggressively; who understands how to build web apps that can evolve and scale; who looks for the best feasible way to do something instead of just the first one that pops into mind. I especially needed to find someone who was as BDD/TDD-infected as I am, and recognised the value of building tools and automation early (but incrementally) so that we'd be a lean, agile development machine when the rubber really needed to hit the road, a (very) few short weeks from now.

In the States, or in much of Europe, or basically anyplace outside Singapore, I'd probably have a decent chance of finding such a person. Here, not so much. I was especially concerned by the way PHP usage seems to be devolving in these parts. There are lots of folks who think that it's a good idea to reinvent Every. Single. Wheel. themselves, without really knowing all that much about what has gone before. Frameworks are a good example; if you aren't intensely aware of how good software, sites and Web apps get built; if you don't even bother with encapsulation or MVC or SOLID or on and on, how can you expect anybody who does know his Craft from a hole in the ground to take you or your "tool" seriously? It might wow the management rubes who proudly admit they don't know squat — but in the world as it is, or at least as it's going to exist in the very near future, those will (continue to) be a vanishing breed even here. Even in a Second World Potemkin village of a city-state where what you are and who you know is far more important in most situations than what you've done and what you know, you're still running a risk that somebody is going to come along who actually gets up in the morning and applies herself or himself to writing better stuff than they wrote yesterday. And, eventually, they're going to wipe the floor with you — either you as an individual or you as a society that remains stubbornly top-down-at-any-cost.3

In contrast, my experience talking with Ruby and Rails people here, even people with absolute minimal experience in Rails, is chalk-and-Friday different. For one illustrative example: one book that more than half the yes-I've-done-a-bit-of-Rails folk I've talked to here is Russ Olsen's Eloquent Ruby (ISBN 978-0-321-58410-6). Your average PHP dev — or C++ dev or Java dev — is fighting the dragons (schedule, complexity, communication, etc.) far too much to get out of fire-drill-coding mode and into writing. If you believe, as I do, that all truly competent software is written as a basis for and means of conversation between humans, and incidentally to be executed by a computer, then you know what I'm driving at here. Knuth's dictum that programming is a creative, literary act is too easily lost when you're just throwing yourself against the wall every day trying to see which of you will break first. (It's very rarely the wall.)

If you write software that can be read and understood, and intelligently expanded on or borrowed from, and your whole view of the omniverse changes. Your stress goes down, your written works' quality goes up, and you enjoy what you do a lot more. Automating away the repeated, detailed work and having a sensible process so you don't give yourself enough rope to shoot yourself in the foot regularly (with apologies to Alan Holub) is the most reliable way yet found to get your project schedule under your control instead of vice versa.

All this does however have one pre-supposition, which I have received numerous complaints on here locally: you and your team must be fully literate and fluent in a shared human language4. The vast majority of software to date seems to be associated with four such: Business Standard English, Russian, Japanese and Chinese. If your team shares a different language at a (near-)native level, with one or more of you having similar skills in the earlier four, you should be able to make do for yourself rather nicely. Having others build on your work, if desired, is going to be a bit more problematic. Poor communication has been at least a strong contributing cause, if not the root cause, of every failed project I have seen in my career. If you can't speak the customer's language effectively enough, with nuance, subtlety and clarity as needed, then your chances of project success are somewhat worse than my chances of winning the next six consecutive Toto jackpots — and I don't plan on buying any tickets.

Footnotes:

1. "I am sitting here tossing the options around for a business application, browser based, and trying to decide...PHP, Java, Python, .NET?" (Return)

2. Why insist on a senior guy when they're so rare here (at least the PHP variety)? Because I figure that with a senior dev coming on a month from now, we'd spend roughly half the time between now and our first drop-deadline writing magnificent code, and the other half building infrastructure and process to make building, testing and deploying the magnificent code we write straightforward enough not to distract the team from the creative acts of development. There's not enough time to wipe anybody's backside. (Return)

3. The phrase "at any cost" always reminds me of a company I contracted to back in the late '80s. My boss there had a (large) sign above his desk, "We will pay any price to cut costs." The company eventually paid the ultimate price — bankruptcy. (Return)

4. I'm describing what used to be called "college- or university-level language skills", but as any university teach will readily tell you, the skills exhibited by students have dropped precipitously and measurably in the last three decades or so. (Return)

ANFSD: GnuPG Semi-Pro Tip

After you use GNU Privacy Guard, or really any public-key encryption system, for a while, you'll probably have it set up for more than one of your email addresses. It's tempting to have the same pass-phrase for all your IDs.

Don't.

For instance, I use long-ish pass-phrases that are similar enough to remember easily but different enough that a dictionary attack is highly unlikely to be successful. That also protects me from doing something silly/confusing/potentially dangerous like thinking I'm sending from one email address when my email package actually defaults to another. Since the pass-phrases differ, you can't sign a message sent with Account B with the phrase from Account A (that you thought you were using but you were in too much of a hurry to pay attention to the 'From' line).

DRY May Not Be Wet, But It Sure Is Cool

As any developer who values his time, sanity, or amicable relations with teammates who have to maintain his code knows, one of the cardinal rules of modern programming is "Don't Repeat Yourself", or DRY. This is obviously something that should make reading code easier. Obviously, therefore, it should be applied except where the lengths you go to to not repeat yourself make your code harder for someone else to read. The idioms of some languages help this more than those of others.

I was recently browsing through a (rather poor) Ruby programming book when this hit me between the eyes. Consider this ERB example from the book.

Pretty ugly, yes? Part of that ugliness is intrinsic to ERB; which is why people are leaving it in droves.

Compare this exact translation into Haml:

= form_for @product do |f|
  - if @product.errors.any?
    .error_explanation
      %h2= "#{pluralize(@product.errors.count, "error")} prohibited this product from being saved:"

      %ul
        - @product.errors.full_messages.each do |msg|
          %li= msg

  .field
    = f.label :title
    %br/
    = f.text_field :title

  .field
    = f.label :description
    %br/
    = f.text_area :description

  .field
    = f.label :image_url
    %br/
    = f.text_field :image_url

  .field
    = f.label :price
    %br/
    = f.text_field :price

  .actions
    = f.submit

A bit easier to understand, now that we've gotten most of the clutter out of the way, yes? Now, the repeating field definitions stick out like a sore thumb; they're almost the same, but not exactly. We've got three text_fields and one text_area, so a simple loop that just plugs in values won't quite cut it, will it? How about this:

= form_for @product do |f|
  - if @product.errors.any?
    .error_explanation
      %h2= "#{pluralize(@product.errors.count, "error")} prohibited this product from being saved:"

      %ul
        - @product.errors.full_messages.each do |msg|
          %li= msg

  - [:title, :description, :image_url, :price] do |name|
    .field
      = eval("f.label :#{name}")
      %br/
      = eval("f.text_field :#{name}") unless name == :description
      = eval("f.text_area  :#{name}") if     name == :description

  .actions
    = f.submit

If you're coming to Ruby from a PHP background, you've been conditioned not to use eval; there's all sorts of nastiness that could lurk there, especially when code is sloppy. (And PHP code is notorious for its sloppiness.) But in Ruby and Haml, this lets us solve a problem more eloquently than the two or three other possibilities that come to mind. Further, the idiom of the repeated-but-not-repeated text_field/text_area fragment

      = eval("f.text_field :#{name}") unless name == :description
      = eval("f.text_area  :#{name}") if     name == :description

should make it clear to even the most hurried reader that we're dealing with a special case. This is one of those situations where a helper method would be nuclear overkill. (If we were scattering dozens of these all-but-one-fields-is-the-same forms through an app, I'd revisit that stance.)

---

OK, but why blog about this in the first place? Well, the company I'm working with now is looking for a senior Ruby/Rails dev. Most of the CVs we're getting are from mid-level guys (if you're a Rails doyenne (i.e., female), please email!) who have done some Ruby and one or two other languages, generally PHP and Java. What I've learned, on both sides of the table, is that if you've got some experience in several languages that aren't all closely-related to each other, it's a lot easier for you to ramp up on any language you need. Any language will eventually go out of favour. Being able to code idiomatically in anything you need to, won't.

Eloquence Is "Obsolete". We're Hurting. That's Redundant.

Code is meant to be read, understood, maintained and reused by humans, and incidentally to be executed by a computer. Doing the second correctly is far, far less difficult than doing the first well. Innovation is utterly meaningless without effective communication, and that is at least as true within a team as between it and a larger organisation, or between a company and its (current and potential) customers.

The degree to which a class framework, or any other tool, helps make communication more effective with less effort and error should be the main determinant of its success. It isn't, for at least two reasons. One, of course, is marketing; we as a society have been conditioned not to contest the assertion that the better-marketed product is in fact superior. In so doing, we abdicate a large degree of our affirmative participation in the evolution of, and the control over society at the small (team/company), mid-level (industry) and wider levels. We, as individuals or organisations, devolve from customers (participants in a conversation, known as a 'market', in which we have choices) into consumers (gullets whose purpose is to gulp endless products and crap cash).

More worrying is that effective, literate communication has gone completely out of fashion. Whether or not you blame that on the systematic laying waste of the educational system over the last thirty years, it's increasingly nonsensical to argue with the effect. People are less able to build understanding and consensus because they do not have the language skills to communicate effectively, and have been conditioned not to view that as a critical problem urgently requiring remediation. Oh, you'll hear politicians bloviating about how "the workforce needs to improve" or "education most be 'reformed' for the new era", but that's too often a device used to mollify public opinion, make it appear as though the politicians are Doing Something effective, and especially to preempt any truly effective public discussion leading to consensus that might effect real socioeconomic improvement rather than the "Hope and Change"™ genuine imitation snake oil that's been peddled for far too long.

Back on the subject of developers and tools, I would thus argue that what tools you use are a secondary concern; if you don't understand code that's been written, by others or (especially) by you, then a) that code can't be trusted to do anything in particular because b) someone didn't do their job.

Your job, as a developer, is to communicate your intent and understanding of the solution to a specifically-defined problem in such a way that the solution, and the problem, can be readily undestood, used, and built upon by any competent, literate individual or team following you. (Again, explicitly including yourself; how many times have you picked up code from a year or a decade before, that you have some degree of pride in, only to be horrified at how opaque, convoluted or incomprehensible it is?) Some computer languages make that effective communication easier and more reliable than others; some choose to limit their broad generality to focus on addressing a narrower range of applications more effectively and expressively.

That has, of course, now progressed to the logical extreme of domain-specific languages. General-purpose languages such as C, Ruby or Python try to be everything but the kitchen sink, and too often succeed; this makes accomplishing any specific task effectively and eloquently incrementally more difficult. DSLs are definitions of how a particular type of problem (or even an individual problem, singular) can be conceptualised and implemented; a program written in a proper DSL should concisely, eloquently and provably solve the problem for which it was written. This has sparked something of a continuing revolution in the craft and industry of software development; general-purpose languages arose, in part, because writing languages that are both precise enough to be understood by computers and expressive enough to be used by humans is hard; it still is. DSLs take advantage of the greatly-evolved capabilities of tools which exist to create other tools, compared with their predecessors of just a few years ago.

But the language you use to develop a program is completely irrelevant if you can't communicate with other people about your program and the ecosystem surrounding and supporting it. If half the industry reads and writes on a fifth-grade level, then we're literally unable to improve as we should.

To paraphrase the television-show title, it doesn't matter if we're smarter than a fifth-grader if that's the level at which we communicate. Improving that requires urgent, sustained and concerted attention — not only to make us better software developers, but to make the larger world in which we live a better place. Let's start by at least being able to communicate and discuss what "better" means. That in itself would be an epochal improvement, saving entire societies from becoming obsolete.

Once more into the breach, dear colleagues, once more…

…or How I Learned to Stop Worrying and Love Serving Aboard Kobayashi Maru; a history lesson.

Once again, I've had an interesting couple of months. Between modern Singapore's regular effect on my health, some insane work and the opportunity to get even more insane work if my two best references weren't indefinitely unavailable (but hey, Thailand is usually lovely this time of year…or anytime, actually).

Ahem.

I am rediscovering a love for developing in Ruby after losing touch with it some ten years ago. In the Ruby 1.5 days, things like class variables and the hook system were either new and shiny, or had finally been thrashed into something both usable and beautiful. If programming was what you did to earn a living, then programming in Ruby was something that you thanked $DEITY for every day because, after all, how many people in this world get to use beautiful tools that make you measurably better at your craft while blowing your mind on a regular basis, and get paid for the privilege?

The Fall

Then Ruby on Rails came along, with the clueless-business-media hype and cult of personality built around G-d Himself, or at least the three-initial version of same as anointed by said media, and Things Changed:

New acquaintance: So what do you do for a living?

Me: I write computer software, mostly using Ruby, or when I have to, Delphi or C++.

N.A.: Oh, you're a Ruby on Rails programmer.

So I left Ruby behind. Delphi lasted for a while, until its competitor's lock on the default system combined with some spectacularly bad corporate timing to relegate it to an "oh yeah, I heard of that once" niche. And then came this gawky new kid called PHP.

PHP wasn't exactly "new" when I first got into it; version 3.x had been out for some time, and the in-beta version 4 had class-based object-oriented programming. It was more primitive than an 1898 Duryea automobile to a Porsche driver, but you could see that the basic principles were at least in sight. And so, as the dot-com bubble was starting to really inflate, I jumped into PHP with a vengeance. It wasn't a product of one of the existing tech-industry corporate titans and it wasn't tied to a single operating system or Web server (though it did seem to work best with the then-early Linux OS and Apache server).

The great thing about PHP is that just about anybody can poke at it for a while, and at the end have something that (seems to) work. The pathologically execrable thing about PHP is that the barrier to entry is microscopically low, lower even than for Visual Basic 6 back in the day. And so, the inevitable result was (and is) that you have literally hundreds of thousands, if not millions, who get jobs by saying "Yeah, I know PHP; I've done (this little site) and (that steaming mess with a lot of Flashy bling on it)." In reality, there are thousands, at most, of good PHP developers out there, with a few more thousands actively working at improving their craft.

Purgatory

So PHP 4 had a crappy object model that anybody could poke at or ignore at will. PHP 5, from mid-2004, started to get its head on straight with respect to both OOP and what it really took to do PHP well, but lots of damage had already been done. Innumerable client projects had either failed or become incredible maintenance/performance nightmares due to the shoddy code that PHP, and the community that grew up around it, encouraged mediocre/inexperienced/inattentive developers to write.

And then, in mid-2009, PHP 5.3 came into the world, and it was a glorious golden statue with legs of brown, wet mud. Many of us who wanted to see PHP evolve into a "properly" object-oriented language, along the path that it had been following, found much to rejoice in. Support for closures. Better internationalisation support. Far better garbage collection. A rearrangement and winnowing of the extension and application repository system that was the closest thing PHP has to Python's eggs or Ruby's gems.

PHP 5.3 also introduced what had to have been the most-requested new feature for years: namespaces. Namespaces are one solution to the problem of allowing the development team to organise collections and hierarchies of classes to both make them easier to work with conceptually, and of mitigating possible naming conflicts (class Foo in namespace Bar is "obviously" distinct from class Foo in namespace Barney.)

However, this is also where the legs of the "golden statue" were transformed into wet mud: the way in which the PHP namespace features work is so semantically and visually jarring, with so many inconsistencies visible to both the experienced pre-5.3 PHP developer and the experienced developer of object-oriented software in other languages, that it quickly became a laughingstock and a millstone. A necessary millstone, but one of which many writers of both code and prose waxed eloquent in their righteous, intricately-justified derision.

A New Hope

And this eventually served as a wake-up call to a number of those who view PHP as just another tool in their toolkit, as opposed to either a cash cow to be milked or a semi-religious icon to be polished and cared for in the precise fashion that the High Priests of Zend decree, or, those who simply never learned enough to care. Seven to ten years is a long time to spend in any one language for an experienced developer, and quite a number of highly-visible PHP community stalwarts have been publicly participating in and contributing to other communities: various JVM languages like Groovy, Scala and Clojure; Objective-C; C#; and Ruby.

A few short months ago, I had urgent need to re-immerse myself in Ruby, learn modern Rails, and make myself ready in all respects to participate in Ruby on Rails projects at a senior or leading level. This gave rise to a series of fortunate events, to paraphrase Lemony Snicket. I discovered that Ruby 1.9 is now a very advanced, mature language with solid experience-based best practices that are sensible and self-consistent. I discovered that Rails 3.1 is an incredibly productive way to get a Web site or application up and running, and that the ways in which it encourages you to code and think do not have the same propensity to inflict mortal wounds as, say, the overly-trusting PHP journeyman developer. Rails itself has grown to be a much larger community that no longer revolves around a single individual, or even a small group of individuals, as that it did seven years ago or as too many other languages do now. And, importantly, many of the things that make Rails great would not be practical, or even possible, were it built on and in any other language than Ruby

Above all, Ruby gives hope to those few, mindful of Knuth's saying that "programming is a literary act", understand that computer programs are written to be read and modified by humans, with computer execution almost a secondary concern over the life of a project. If you care about thinking creatively, if you enjoy having your mind regularly blown in ways that challenge you to actively and continuously improve your mastery of the craft of software development, you are going to love Ruby, and Rails.

An Evangelist Repurposed

And so, to any who are pondering a new Web project, who see how overwhelming the mindshare of PHP is and how obviously negligent (even to non-technical eyes) is far too much PHP code and, by extension, those who made such code available, I would ask you to strongly consider Ruby, and Rails, even if your team has little or no experience in them.

After all, quality and intelligence should, in any world worth living in, be major competitive advantages — or, rather, by rights, their lack must needs be mortal.

Drafting a Classic

By the way, count me among those who find the new Blogger in Draft interface to be a huge improvement over the old interface. The composing interface is clean and functional, with few distractions to typing away. It has the bonus of peripheral features and actions being visible at the periphery, ready when you need them but not with a busy interface distracting you away from the cursor. Previews open in a new tab (depending on your browser), with formatting applied properly (a huge improvement over the original). All in all, I really don't expect to have any problems of the scale of, say, the formatting catastrophe which I blogged about earlier.

The post settings such as keywords (called 'labels' here; many people know them as 'tags') are nicely done, with clean, usable interfaces that show exactly what's available and what you've done before you hit that big orange Publish button.

Nicely done, folks!

Stock components: Nothing can go wrong…go wrong…go wrong…«boom!»

(Yes, Westworld had a view of modern software that's not so far off the mark, considering it was made in 1973.)

I use Macs for development, using both the native Unix OS and one or more VMWare Fusion VMs of Linux, BSD Unix or Windows, depending on the need. Even when I'm going to be deploying on a Linux or Unix server, I prefer to do my basic work using OS X native tools1.

Usually, all that rather famously Just Works. Sometimes, however, it provides lessons in the raving optimism that is Murphy's Law.

The remainder of this post is a summary of Wiki notes I made in chasing down one such occurrence.

---

This affects only the iMac, not the MacBook Pro2.

The Problem

PHP 5.3.4 as bundled with OS X does not work from Apache 2.2.14, also as bundled with OS X. This appears to have broken with the 10.6.7 update, as noted by many people, and was not fixed by updating to 10.6.8.

Pink Herring

(Not quite a red herring, mind you; just very close to one.)

One of the first things I thought of was to check what modifications had been made to php.ini over the course of time. Every time a new update to the PHP engine was installed as part of Apple's Software Update, the previous contents of the file were saved under a (nominally) intuitive name.

As of Saturday 3 September 2011, the iMac's /etc/ directory contained the following variants of php.ini:

-rw-r--r--  1 peerless  wheel  68920 Sep  3 12:37 php.ini
-rw-r--r--  1 peerless  wheel  45816 Oct 12  2009 php.ini-5.2-previous
-rw-r--r--  1 peerless  wheel  68920 Aug 13 19:11 php.ini.applesaved
-r--r--r--  1 peerless  wheel  69060 Dec 16  2010 php.ini.default
-r--r--r--  1 peerless  wheel  45104 Aug 25  2009 php.ini.default-5.2-previous

At this point, of course, I'm starting to say "uh-oh; a five-way diff?" Silly me; there's an easier way to deal with differences3. We'll get to that at another time; regardless of its usefulness in the current situation, it would be good information to have.

Inspiration

Then I thought, "hang on, have I checked the Apache logs?" Err... not yet. So, I opened a new browser tab and attempted to open http://localhost/~peerless/index1.php, which visibly failed; no output was rendered by the browser.

First off, check /var/log/apache2/access_log. Therein lies the first clue:

127.0.0.1 - - [03/Sep/2011:15:44:41 +0800] "GET /~peerless/index1.php HTTP/1.1" 200 20 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.126 Safari/535.1"

The 200 code means that the file was successfully sent, but it's a lot longer than 20 bytes. Clearly, something fell over and couldn't get up.

Looking at /var/log/apache2/error_log yields the crucial bit.

[Sat Sep 03 15:44:41 2011] [error] [client 127.0.0.1] PHP Fatal error:  Unknown: apc_fcntl_unlock failed: in Unknown on line 0

A quick bit of google-fu established that the APC PECL extension to PHP has a known bug in it (which is fixed in a release that should ship Any Day Now). I have three options available for action:

1. Wait for the patch to be rolled up into a new release version, and upgrade my installed copy of APC;
2. Pull the source for APC, make the change myself, and hand-build the extension; or
3. Get rid of APC entirely, since this is strictly a dev machine that (for now) doesn't need to worry about caching at all.

A few seconds after implementing option 3, refreshing the index1.php page in the browser caused it to display as expected.

PHP is up again! Great rejoicing was heard through all the land! (Well, at least the room.)

Future Action

Moral of the story: At some point in the not-too-distant future, I Really Should go through the installed PEAR and PECL extensions and get rid of any I'm not actively using. That's what VMs are for: letting me try out various configurations as needed, without stepping on the main system. Oops.4.

Additionally, we still haven't answered the question: why are there five different versions of php.ini?

Footnotes:

1. Alternatively, for non-Windows OSs, I can use sshfs with MacFUSE to mount the VMs "hard disk" as part of the OS X file system. With Windows I go the other way; have VMWare share a directory on the host Mac with the Windows instance. (Return)

2. This turned out to be because APC was never installed on the MBP. But we'll get to that. (Return)

3. One easy way(for a competent dev, at least): whip up a VCS repository and add each file in turn, by date, as php.ini. Then use the VCS' diff tool to generate patches from each to the next. (Return)

4. And writing this page took several times as long as actually solving the problem. (Return)

The Yak Shavery; J. Dickey, Proprietor

Starting a new project for a new client, coming up to speed on a bit of kit almost like something I poked around with a ways back, and a ferociously Singaporean flu do not make for a productive week. I'm almost back to where I expected to be by noon on Monday. Since it's noon on Thursday…

What I'm trying to learn and leverage is Chef, an automated-configuration tool for computer systems (mostly servers). If you want to be able to reliably, repeatably set up a server (or a server farm), Chef or one of several available alternatives, will make your life much easier. Once you get your tool of choice working, that is.

Step One is, as various Chef amateur tutorials suggest, to start out with a plain-vanilla installation of your OS (Linux, in this case) of choice. Rather obviously, getting from bare metal (or bare VM disk platters, if you'd rather) to the basic working system should be push-button automatic as well. "Fine," you say, "nearly every major distro has its own automation system; surely I should be able to just pick up, say, a Red Hat Enterprise Linux clone like CentOS or Scientific Linux and use a Kickstart file, and it should Just Work. Right?"

Close; but then, "close" only counts for scoring purposes in horseshoes and (arguably) the use of hand grenades. This is neither, though it does have the ability to blow up in our faces with unpleasant consequences.

Putting together a basic Kickstart file to set up a base system (on which we can use Chef to complete installation and configuration; the whole point, remember) is itself pretty straightforward. Except for one "little" thing:

(not my actual screenshot; this one courtesy of Máirín Duffy.)

Hitting the "Reinitialize All" button will allow the install to finish as expected, but it's still a manual action in a process that's trying to eliminate such.

Some bit of Google-fu later and asking questions on the #centos channel of irc.freenode.net (thanks, Wolfie!) pointed me to the correct option to set when clearing the disk partitions, and all should have been peachy-keen and wonderful from that point on. Except it wasn't, of course.

Tests conducted with two different versions each of three RHEL clones on my VMWare Fusion 3.1.3 system all failed with a mysterious "cannot read repodata" fatal error being thrown by the installer.

Except, of course, that mounting the DVDs in question and browsing them showed the /repodata directory and contents exactly as they should be. Fast-forward through a day of flailing and fuzzing different options "just to see what happens", and you have a classic yak shave.

So, knowing that I'll have to get the RHEL versions working eventually, and having been previously warned against using Chef-on-Ubuntu as a learning exercise, I'm now spinning up AutoYAST on openSUSE, which is (experientially) what I should have started with from the beginning.

The wall, where my head has banged against it repeatedly, can be repaired. The calendar? Time is the ultimate non-renewable resource.

(Anybody who has any suggestions for Making This Work with RHEL clones, ideally Scientific 6.1 and/or CentOS 6: enlightenment would be Greatly Appreciated.)

The Formerly Magical Trackpad

My Apple Magic Trackpad died tonight, aged less than six months, after a week-long illness. Numerous battery transplants, resets and reboots were unable to revive the patient. In lieu of flowers, donations to its replacement are now being gratefully accepted.

Whiskey. Tango. Fox?

Hardware — any hardware, but especially Apple kit — isn't supposed to die after mere months of daily use. I expect better from systems, and especially from Apple. Were I anywhere else on the planet besides Second World Singapore, I would have a reasonable degree of confidence that those expectations would be fulfilled. PAPaganda notwithstanding, this is the Potemkin city where having the highest-paid politicians on the planet doesn't mean that products or services (beyond political control-related security "services") are up to the same levels as would be expected in, say, Tanzania.

So I'm back to using the mouse that I'd been using for nearly two years before buying the Trackpad. With the wheel and five buttons (that I know of), it's certainly adequate for most ordinary-user or Web-developer tasks in Snow Leopard.

But trackpads are The Future™, a reality emphasised by the new OS X 10.7 "Lion" release, and by what's rumoured for Windows 8. While mice are comfortable and familiar to essentially everyone who's used a Mac in the last 27 years, or Windows PC in the last 15 or so, there are actions and gestures that the trackpad is simply a more natural fit for, and to which a trackpad user's "muscle memory" rapidly adapts. Going back to the mouse feels both familiar and foreign, not unlike attempting to ride a bicycle for the first time in thirty years (another recent experience).

So, I'll probably buy a replacement trackpad fairly soon. That is, if I can stay online long enough to get useful work done; we've had intermittent failures in the DSL service here of late. (Or am I allowed to say "intermittent successful access"? This is Second World Singapore, a family-run company town that dislikes any attention paid to the clay feet of its mighty statuary (and I'm not dissing the Merlion here).

Sigh.

The App Store: I Get It, but...hmmmmm...

Apple's "post-PC," "new media" iDevices are selling quite nicely, thank you. Millions of people have become enthusiastic, loyal, even evangelistic customers without ever clicking a mouse on an Apple desktop or laptop computer. Given that Apple's profit from a laptop or desktop computer is several times what it is for an iDevice, it's almost imperative for them to find ways to make the Mac desktop/laptop user experience support the traits that have been most successful on iDevices. This includes, especially, the App Store.

Fans of the App Store, either on the iDevices or the Mac, point to the fact that you can have several different devices (iPad/iPhone, iMac/MacBook) using the same Apple ID, and thus can install "content" that you "purchase" on several devices; all you need to do is register them under the same ID.

But the Mac, historically, has had a rather different model. You would (and still can, fortunately; at least for now) either purchase a disk with software on it, or download a "disk image;" a file which appeared to Mac OS to be a disk. As soon as you did that, a window would open with the program icon in it and an icon for your Applications folder, and you'd drag the icon over onto the folder. The software would be copied to your hard disk, and you'd be able to then run it like any other program. Want to uninstall the program? Drag its icon to the trash.

Another benefit of that latter model is that you have actual media — either physical or virtual — which you can use to reinstall the program on your computer or, license permitting, on other computers you personally or corporately own. You're very aware during the "normal" installation process that you're working with some sort of artefact which contains the program you're installing. That artefact either is or tries reasonably successful to simulate a physical object.

We've built up certain expectations about physical objects. They exist; though they can be destroyed, they don't generally disappear without direct action by you or another person. With very few exceptions, they can be used more than once. These and other expected traits give the object some value — economic, sentimental or otherwise.

The App Store for the Mac does away with the artefact; it replaces it with an abstraction that's not "real" in any sense you can point to afterwards. To a degree that some find disturbing, it turns the product with which you previously dealt into a service. That service can become unavailable for any number of reasons, not all of which are necessarily a result of your actions or intent.

Put another way, it turns the old "software as a product" notion, and in particular the notion of free software, on their heads. Most free software licenses (try to) make clear that the user is In Charge of his or her own system, including use of the software being licensed. The copyright holder retains certain rights, but yields the remainder to the user under conditions designed to make sure that the software remains "free" in the sense that it was originally licensed. The user may examine the software, redistribute it as he got it, (under most FSF-approved licenses) can create modified or derivative works so long as they are licensed compatibly to the original, and so on. Above all else, the user has the unquestioned right to make as many backup copies of the software as desired in case the original installation becomes unavailable (hard disk crash, extreme weather event, etc.) The user has rights, and has the means to safeguard those rights by virtue of the fact that he has a physical copy of the software in its original, usable form.

The App Store says "Mr User, you don't have to worry your pretty little head with any of that. You just point-and-click your way through the store, and when you check out, the software will be installed automatically on your system." That's great — so far as it goes. But there's no option, at least none that I have yet found, to retain a copy of the disk-image file(s) for the software you just licensed. If your drive goes south for the winter and doesn't come back, you "just" replace the drive, reinstall Mac OS X and install all your apps from the App Store.

That's fine for small programs, especially if you have a high-speed Internet connection using a properly managed ISP. But, unlike the App Store for iDevices, several programs licensed through the App Store for the Mac are in the hundreds-of-megabytes-to-gigabytes range. Try downloading Xcode 4 on a 128K ISDN line. And iLife. And iWork. And... you see the problem. Wouldn't it be much better if you at least had the option to manage all that locally, yourself, the way it's traditionally been for the Mac? And that doesn't even take into account Apple's well-documented "control-freakery," where software available one day — or even already installed on your system — may not be available the next. Even if we give Apple the benefit of all doubt, ascribing to them motives as pure as the driven snow, there's still a real problem here.

The iTunes Music Store has now moved away from "protected" DRM-encumbered music, because customers made it clear that they wouldn't stand for the inconvenience and loss of control over product that they had paid for. I can back up my iTunes library however I wish, and be reasonably confident that should anything happen to my Mac, I'll be able to restore the library on my new replacement Mac without downloading it all again from the ITMS.

Why can't I have that same confidence with the software products I license through the App Store? Apple aren't the only ones who think they should have a fair amount of control over "their" "stuff."

Comments?

A Blessing is ALWAYS A Curse (and Vice Versa)

I've recently started as Senior Architect at Savant Degrees, a Singapore-based Web consultancy. Quite often, I feel like my job is as much "Senior Curmudgeon" as anything. According to Wiktionary.org, a curmudgeon is "[a]n ill-tempered (and frequently old) person full of stubborn ideas or opinions." I'm undoubtedly the oldest person among any of my colleagues that I've met, and I was told fairly explicitly that I was being hired on the basis of the variety and depth of my experience — i.e., for my ideas and opinions. And, like most people, I think I'm only stubborn when I'm right.

Naturally enough, I'm doing a lot of learning, too. This is my first experience with Groovy, a nicely dynamic language which runs atop the Java VM. Like any language worth its weight, a rich ecosystem is growing around it. Like the ecosystems around most languages, there is at least one ultra-massive tool that takes new folk some time to wrap their minds around. With PHP, for instance, it's the Zend Framework, which arguably makes the C++ Standard Library look minimalistic. With Java, it's pretty much anything having to do with J2EE1 (to which Groovy itself is a reaction).

The biggest tool I've yet stubbed my toe on in the Groovy world is, unquestionably, Gradle2. In the grand Zend tradition, the PDF version of the user guide is nearly 300 pages long and, also like Zend, is considerably out-of-date. It's considerably easier to argue that Gradle justifies the bulk, however, since Gradle is a "let's build anything" build manager/dependency tracker/dessert topping/floor wax. The basics can be understood in half a day, even if you've only nodding familiarity with Groovy3. To truly master it, however, requires significant extra time — as much as you feel you derive benefit from.

And that (finally) brings me to the point of this post. The group I'm working with now has some very talented and moderately experienced developers in it; they're open to new ideas and new ways of doing things, more than is the local cultural norm. In our craft, that's not just a Good Thing™; it's necessary for continuing success4. But that enthusiasm is tempered by a keen consciousness of impending deadlines, particularly the "curse of success;" a month, later this year, when we will be spread extremely thin amongst several important projects. The ways we've done things have been successful, but we see the down-sides of success as well.

So, new ways of doing things, and the hope of new, increasing success. New things like continuous integration using automated builds and unit tests, among other such recently-popular tools and techniques. (They're popular because they generally work better than Ye Olde Ways.) I'm arguing for the team to use tools that will, at the click of a button or the tick of the clock, automatically check out, build, test and package the code and associated resources for our project. The team is new to this, having used "traditional" "waterfall" processes in the past. Knowing one is to be hanged, as Mark Twain said, tends to focus the mind wonderfully; a rational person (or team) will avoid being put in that position. My job, so to speak, is to make the tools that help us cut the hangman's rope, and teach their use. Presenting already-busy people with several hundred pages of documentation is probably not the most effective way to do that.

So, at the moment, the most pressing blessing is that we have such wonderful tools available, with more on the way. Most categories of tools have several at-least-usable alternatives available; it's very easy to go into information overload. When that happens, often you just choose to use what some exemplar, say a tool vendor, use themselves. It becomes tragically ironic when a development group pursuing a more agile process has their agility and effectiveness limited by their "agile" tools. (I don't mean to be commenting on any specific company or team here; I've seen this effect on numerous projects over the years.)

Choosing deliberately increases the odds that you will choose wisely, and those odds will increase along with your experience. Experience is perhaps the ultimate professional blessing-and-curse-combined; "you never learn anything from doing it right the first time." I prefer the Nietzsche:

That which does not kill you, makes you stronger.

We're not dead yet — so we must be strong, yes? Another "stubborn idea and opinion."

Comments?

Footnotes:

1. Now known as JEE, because "Java 2" (1.2) is so last decade; we're on 1.6 now. (Return)

2. No, I haven't looked at Grails yet. Being a close of Ruby on Rails, I expect that to be the proverbial "kitchen sink with subdivisions attached." Fortunately (?), we're a Tomcat shop. (Return)

3. But, of course, experience counts; Gradle rewards your groovy Groovy skills with more efficiently expressive power in your project automation. (Return)

4. There's another curmudgeonly opinion for you. (Return)

Windows? Check. Skype? Check. Quality? Oops…

Several cultures share variations of a proverb which reminds us that you cannot build a house on sand and have it endure the test of time — or even the next tide.

Skype learned that the hard way a couple of weeks ago. Like all peer-to-peer, or P2P, networks, Skype relies on (a varying subset of) its users' systems to handle routine network-housekeeping chores, including routing. Fine; that's the way many applications have done things for more than a decade. Skype have been living unusually close to the edge in two respects: they have an unusually large, widely-distributed number of simultaneous users relative to other P2P networks, and it implicitly trusts the reliability of its Windows-based application and the underlying Windows operating system when performing at any needed scale. This is what sent the waves crashing in upon the sand-borne house.

Microsoft Windows is, inarguably, a wildly successful software system; along with Microsoft's Office set of applications, it is the basis of Microsoft's billions of dollars a year in revenue and profit. But just because something is successful or because it's made by a major corporation doesn't necessarily mean it's well-designed or -built. (See, for example, the Ford Pinto.)

By relying on the (always-questionable) stability and reliability of each of the Windows PCs acting as a "supernode" in its network architecture, Skype was setting itself on a path to eventual inevitable, foreseeable large-scale failure.

Many of the well-known problems with keeping Windows stable and running properly have to do with the ridiculous ease of developing, distributing and using malware; in some countries (such as Singapore), estimated infection rates of Internet-connected Windows PCs routinely hit or exceed 90%. Besides the obvious threat to any "secure" information on those systems (credit card info, login IDs, and so on), those infected Windows systems have a relatively large share of their processing power diverted for the benefit of someone other than the legitimate user or of applications other than those that s/he has authorised.

Another cause of instability and other problems, as with the Skype outage, is that it is very difficult to develop, deploy and maintain even moderately complex applications under Windows with any real means of asserting that the software is reliable and robust. The only way to gain any such real assurance is by testing as many hardware and software combinations with your software as you can, and hoping that you've found all the "most likely" problems. (This is why Microsoft and others employ thousands of software testers, in addition to a massive automated-testing infrastructure.) And, though Microsoft have made some relatively amazing improvements in this area (compare, say, Windows 98 to Windows 7), even they can't catch everything — and their development process and system architecture essentially require them to. Every Windows usee in the last 25 years has seen the "Blue Screen of Death;" that is merely an acknowledgement by Windows that it's completely lost its mind, usually after innumerable random acts of lesser senility.

Skype were bitten by both ends of this issue: a new release of their client application for Windows had defects that were not caught during development or internal testing, and by deploying that defective application on an operating system that provides little-to-no protection against maliciously or inadvertently defective applications, a large "time bomb" started ticking away, unnoticed by all until its spectacular detonation on or around 22 December 2010.

For a Skype client to be a "supernode," it cannot be behind a "firewall" that uses network address translation to isolate its clients from the big, bad Net. (There are ways around that, too, of course; but they're generally very black-hat techniques that don't scale well on a commercial basis.) A client does not have to be running on Windows to be a supernode, but given the demographics of Skype's user base, and the larger PC user base in general, the majority are.

And so, when Skype's defective Windows client ran into mortal difficulties on Windows' defective platform, the ordinary user PCs that were acting as supernodes started dropping like flies. Other Skype clients, as with any proper P2P network, simply rerouted to other supernodes. However, since many (if not most) of the Windows clients were behind NAT firewalls and there were (and are) a relatively small percentage of non-Windows clients available to serve as supernodes, the number of "leaf node" clients chasing the decreasing number of supernodes put the remaining systems under an unsustainable load, and so those Skype clients (with their supernode capabilities) dropped off the network.

Had there been a more heterogenous mix of Skype clients so that those available to perform supernode duties were not almost exclusively Windows PCs, the failure might well have been mitigated if not eliminated altogether. Or, were it practical to develop a (reasonably) guaranteed-stable application on Windows, the defect that initiated the failure cascade might well never have occurred. But by "putting all their eggs in one basket" which was known to have large, gaping holes in it, Skype were guaranteeing that their metaphorical sidewalk would eventually be covered hip-deep in broken eggs.

Monocultures in technology, as in farming, are generally Bad Things; a threat which can attack any member of a monoculture can attack all such members. (Think of the Windows malware-infection rates as an example of this.) Single points of commonality (or of control) are always eventual single points of failure.

The Skype failure could well have been a reminder of these long-known basic truths. The commercial interests involved, however, ensure that such a lesson will go unheeded. It will therefore be repeated; very possibly not with Skype, but with some other aspect of our grossly defective information-technology ecosystem. Think of it as "Unsafe at Any Speed" meeting Groundhog Day until something is done to prevent yet another China Syndrome.

Keywords (aka "tags") ARE NOT Structure

And I understand that's the whole point. But sometimes, you're dealing with a concept or with data that is most naturally and properly organised in a structured hierarchy. While keywords1 are a very convenient tool for filtering ad hoc queries2, they're a poor stand-in for truly structured information.

One (possibly trivial) case in point: managing your browser bookmarks. What I want to be able to do is to organise bookmarks3 and be able to view them in the hierarchy and order that they will be in in the browser "Bookmarks" menu. That way, regardless of which file format4 is being imported from or exported to, while you're using the bookmark manager, you have an accurate view of how the ("folder"-based) organisation of the bookmarks will appear in your browser.

So why not, as several managers do, use keywords to imply structure? Let's take a look at that idea, with a scenario. Let's say you have 4,000-5,000 bookmarks in a fairly deep and wide hierarchy of about 800 folders, some nested five and six levels from the root. While that may sound like a 1996-era Yahoo!, that's roughly what I deal with, and I know quite a few long-time Web users who have significantly more.

OK, so you're going to do your internal organisation using keywords. You're a Web developer, so you might have a few bookmarks in a folder with the path Development/Web Development/PHP/Frameworks/Agavi. Each bookmark's reference in the bookmark-manager software would have the keywords Development, Web Development, PHP, and so on. It clutters up the keyword list, and it necessarily complicates the display and editing of those bookmarks within the software. It also means that more data is stored for each and every bookmark. Disk drives are cheaper than they have ever before been, but they're not free.

And, what's worse, if you accidentally (or otherwise) delete one of the keywords from one of the bookmarks, when you export your list again, that bookmark will be in a brand-new folder somewhere else in the hierarchy. Finding and fixing that, if you have a reasonably complex bookmark list, can be difficult. If you don't find and fix it, of course, then it won't be there when you go looking in what should be the right place.

And finally, you're twisting the whole structure of the way people (and browsers) view bookmarks; you're distorting the semantics of the system. This may seem a fine point in today's "get-it-done-yesterday!" way of doing everything. It's not. By not having a clean, self-consistent, efficient conceptual organisation for your software, you make it more difficult to learn, more difficult to use, more complex to develop, more expensive to maintain, and more likely to have more severe problems over the lifetime of the software5.

One final mental exercise. Think of that CD collection I alluded to at the beginning of this post, that's now stored on your computer using whatever music player you like. If I delete the Wakeup Music keyword from the track info for "Come Fly With Me," and then I play the album, nothing has changed. That's metadata. But if I change the bit-rate of the recording, or move the track into the folder for the album "My Way," that's different, isn't it? That's a change to the data, or to the organisation of the data. One is a matter of choice; the other is a matter of history.

By conflating data and metadata (literally, "data about data,") we do a disservice to both.

And now, if either of you happen to know of a (preferably not browser-based) bookmark manager that does things The Right Way™ — or reasonably close to it — I'm all ears.

Thanks for reading, and replying.

Footnotes

1. Though they're popularly called "tags" by all the Kool Kids™ these days, and "Labels" here on Blogger, I'm going to refer to the data items as "keywords" because that's what the library- and information-sciences worlds call them, as well as any programmer who started his Craft before, oh, about 1995. (Return)

2. "List all my Frank Sinatra album CD titles that I've tagged as Wakeup Music." (Return)

3. Each "bookmark" should store URL and page title at an absolute minimum; keywords are good to have; additional metadata optional. (Return)

4. XBEL, OPML, LSMFTML, or whatever the new "standard" is next month. (This would be a good feature to think about some sort of plugin architecture for.) (Return)

5. Look at practically any Windows program for examples of this, or, better, any three Windows programs together. It's not that they don't work, but after a few hours of use you can think of several ways that each could be more consistent with itself and with others. (Return)