Showing posts with label data. Show all posts
Showing posts with label data. Show all posts

Sunday, 31 March 2013

Yes, It's Nonsense.

No doubt profitable nonsense. What follows is a reply to a comment by one Andrew Webb on an InfoQ.com sales-pitch-as-technical-"journalism" puff piece. The piece was "reporting" on a study by Dr Donnie Berkholz of InfoQ.com. This kind of hucksterism is one of the most formidable barriers blocking our craft of software development from ever becoming a professional engineering discipline but, well, read on if you will.

Why here rather than on InfoQ? Simple: even though their reply-entry form states that <a> is an accepted HTML element in comments, it refused to accept any of the links I had in this post. Shoddy "journalism", neet shoddy Web development.


Worse than nonsense; this was written for PHBs, likely as a tool to sell consulting hours training teams in the "more expressive" languages. Anybody who remembers Java's transition from a language and VM into a marketing platform, circa 1997-1998, has seen this done before and better.

Note that I am not talking about Dr Berkholz' original study, the link to which was buried in this puff piece. But there appears to be a real problem with the data that the study was based on from Ohloh. From what I've been able to see, the data covers a period of at least 15 years (late 1990s-present). Subversion(!) was used for 53 percent of the projects in the data set; when you add in now-Palaeolithic CVS, the share rises to nearly 2/3 of the projects covered by the data set.

Let me beat that timeframe to death one more time: In the last 20 years, we've gone through at least two major revolutions in the techniques mainstream developers use. Twenty years ago, if you mentioned "OOP" in half the corporate development centres in North America or Asia, the response would have been "what's wrong?" We as a craft, we were just beginning to get our minds wrapped around object-oriented software development, moving it out of its metaphorical Bronze Age, when the larger revolution which that enabled, BDD/TDD/whatever your flavour of Agile, hit like a silver tsunami. Twenty years ago, I was writing "C/C++", Fortran and Ada code, using various odd bits of version control (anybody else remember how good SourceSafe was before Microsoft bought it?), and checking in massive commits, because centralised SCM systems like SourceSafe, CVS and RCS are a pain; seen less as a design/team-support tool than an insurance policy against Raj's hard drive being wiped by some virus or other. Network connectivity was not as omnipresent, reliable or fast as it is today, by several orders of magnitude.

Nowadays, regardless of what language you're in, you're (hopefully) using some form of agile process. You're strongly encouraged to tunnel in, specifying the smallest change that could possibly fail, and then implement just enough code to make that spec pass. You're not going to check in thousands (or even hundreds) of lines of code at one go; you're going to build that feature you're working on over several commits, on a branch of your SCM tree, and then merge it back into the mainline when it's done. The last ten PHP projects I worked on, over a six-year period, averaged less than 50 lines per commit – in one of the most overly verbose languages since ALGOL 68 and COBOL.

Changes in development practices, and tools, will hopelessly skew that data set unless additional controls are applied, and I could find no description of any. That, to me, says that this is, at best, an enjoyable-in-the-moment bit of data-mining with all the relevance to day-to-day developers' lives of the Magic 8-Ball.

This article was even worse; no examination of assumptions, no discussion of changing trends in the craft and industry, just a vacuous puff piece. An insult to the intelligence of the readers and, despite the possible flaws in the original study, to Dr Berkholz' work as well. If my mind could go take a hot shower to wash the oily residue off, it would. I used to think ZDNet was the bottom of the barrel for this sort of thing. I was wrong.


NOTE: Earlier, I'd previously written "Twenty years ago, I was writing "C/C++", Fortran and Ada code, using Subversion", which is an anachronism, of course. Subversion came out in 2000, and I was not in a time warp in 1994. It just feels like I'd been using (used by?) it for that long.

Tuesday, 21 December 2010

Keywords (aka "tags") ARE NOT Structure

And I understand that's the whole point. But sometimes, you're dealing with a concept or with data that is most naturally and properly organised in a structured hierarchy. While keywords1 are a very convenient tool for filtering ad hoc queries2, they're a poor stand-in for truly structured information.

One (possibly trivial) case in point: managing your browser bookmarks. What I want to be able to do is to organise bookmarks3 and be able to view them in the hierarchy and order that they will be in in the browser "Bookmarks" menu. That way, regardless of which file format4 is being imported from or exported to, while you're using the bookmark manager, you have an accurate view of how the ("folder"-based) organisation of the bookmarks will appear in your browser.

So why not, as several managers do, use keywords to imply structure? Let's take a look at that idea, with a scenario. Let's say you have 4,000-5,000 bookmarks in a fairly deep and wide hierarchy of about 800 folders, some nested five and six levels from the root. While that may sound like a 1996-era Yahoo!, that's roughly what I deal with, and I know quite a few long-time Web users who have significantly more.

OK, so you're going to do your internal organisation using keywords. You're a Web developer, so you might have a few bookmarks in a folder with the path Development/Web Development/PHP/Frameworks/Agavi. Each bookmark's reference in the bookmark-manager software would have the keywords Development, Web Development, PHP, and so on. It clutters up the keyword list, and it necessarily complicates the display and editing of those bookmarks within the software. It also means that more data is stored for each and every bookmark. Disk drives are cheaper than they have ever before been, but they're not free.

And, what's worse, if you accidentally (or otherwise) delete one of the keywords from one of the bookmarks, when you export your list again, that bookmark will be in a brand-new folder somewhere else in the hierarchy. Finding and fixing that, if you have a reasonably complex bookmark list, can be difficult. If you don't find and fix it, of course, then it won't be there when you go looking in what should be the right place.

And finally, you're twisting the whole structure of the way people (and browsers) view bookmarks; you're distorting the semantics of the system. This may seem a fine point in today's "get-it-done-yesterday!" way of doing everything. It's not. By not having a clean, self-consistent, efficient conceptual organisation for your software, you make it more difficult to learn, more difficult to use, more complex to develop, more expensive to maintain, and more likely to have more severe problems over the lifetime of the software5.

One final mental exercise. Think of that CD collection I alluded to at the beginning of this post, that's now stored on your computer using whatever music player you like. If I delete the Wakeup Music keyword from the track info for "Come Fly With Me," and then I play the album, nothing has changed. That's metadata. But if I change the bit-rate of the recording, or move the track into the folder for the album "My Way," that's different, isn't it? That's a change to the data, or to the organisation of the data. One is a matter of choice; the other is a matter of history.

By conflating data and metadata (literally, "data about data,") we do a disservice to both.

And now, if either of you happen to know of a (preferably not browser-based) bookmark manager that does things The Right Way™ — or reasonably close to it — I'm all ears.

Thanks for reading, and replying.

Footnotes

1. Though they're popularly called "tags" by all the Kool Kids™ these days, and "Labels" here on Blogger, I'm going to refer to the data items as "keywords" because that's what the library- and information-sciences worlds call them, as well as any programmer who started his Craft before, oh, about 1995. (Return)

2. "List all my Frank Sinatra album CD titles that I've tagged as Wakeup Music." (Return)

3. Each "bookmark" should store URL and page title at an absolute minimum; keywords are good to have; additional metadata optional. (Return)

4. XBEL, OPML, LSMFTML, or whatever the new "standard" is next month. (This would be a good feature to think about some sort of plugin architecture for.) (Return)

5. Look at practically any Windows program for examples of this, or, better, any three Windows programs together. It's not that they don't work, but after a few hours of use you can think of several ways that each could be more consistent with itself and with others. (Return)

Monday, 23 November 2009

Reuse, Renew, Recycle - Data Structures Edition

Everything gets recycled these days, including acronyms...

Whether you're developing Web pages in your language of choice, writing a new game to take on the world, or (insert your project here), at some point, you're going to deal with structured data.

You're quite likely, at some point, to deal with fairly complex, nested structured data - for configuration, to preserve the state of objects within your software, or whatever. In recent years, "complex, nested structured data" has often been coded up in XML. This can save your coding bacon; there are quite a few nice tools out there to work with XML, insulating you from having to deal with the "raw" <tag ...="" attribute="value">data<nested_tag>more data</nested_tag></tag> soup.

But at some point you may well end up having to hand-edit the thing — you may want to pull something out for reuse, or you suspect that one of the tools you're using is messing up in some way... it happens. And even during that bright, wonderful time when you're building a POC (proof of concept/ possibly overly c***tastic; see, I said we could reuse), you wind up humping a lot of "extraneous" data around that just really wraps a few small things. Ugh.

Even with compression, you're still taking a hit; you've just moved it from the pipe to the CPU. Twice.

I got handed a client project a little over a year ago that used this nifty notation called YAML. While YAML was apparently originally developed for Ruby apps, it's evolved; there are now support tools/libraries for everything from PHP to Visual Basic to Haskell. "YAML Ain't Markup Language," but it can express very complex data structures; nesting is defined by indentation, item types by decorations. (See the spec or this PHP implementation for more detail.)

For projects which have been using binary data storage for non-BLOB data, simply because XML was seen as overkill, this might well simplify things. Likewise, if you're pushing data around between different apps/platforms, thinking "I maybe oughta look at JSON", this is for you.

I've been using it primarily in PHP to encode various data and content models, and it has allowed me to dramatically simplify my code in several places - reducing complexity, inefficiency, and possible bugs. The Ruby and Python crowds have been using this for a while; PHP folk have tended to look at it suspiciously, since it came from Over in Those Languages.


Good tools work in any language.


Reuse? I was talking about recycled acronyms earlier?

At about the same time I found YAML the non-markup-language markup language, I also came across a nifty CSS framework with the highly original name of YAML. This code, which makes it pretty easy to style sites using Yet Another Multicolumn Layout, is a set of CSS (2.x) stylesheets, with minimal graphics and JavaScript thrown into the box. What it gives you is a nice, consistent layout on just about any browser (including hideously-broken variants of MS Internet Exploder) which you can slice and dice pretty much at will. It even degrades sensibly for non-CSS, non-JS browsers — like the Googlebot. If you know how to read CSS reasonably well and this takes you more than an hour to grok, you're working too hard. Browse the docs, the samples, the tutorials, or the online community rumblings, and you'll be productive in no time.

And yes, you can use both together on the same Web dev project. Just make sure you keep your tools straight. After all, YAML is not at all the same thing as YAML.

Monday, 21 August 2006

Projects and Data Formats, or Scratching a Standard Itch

Fair warning: This post was written in bits and pieces over a week that I spent mostly on my back in bed; it hits two or three hot-button issues that I've been running up against. In the fullness of time, I may come back and break it up, or write follow-on entries pontificating on one point or another, but for the nonce, your patience — and comments! are appreciated.

Real standards happen in one of two ways. One way involves an organisation like the World Wide Web Consortium (or W3C as it is commonly known) puts together different committees and working groups, and over the course of various meetings, seminars, forums, and other corporate expense-account sinkholes, massive sets of documents are ratified; if we're lucky, somewhere within that will be nuggets of information and wisdom around which useful things can be accomplished. Successful xamples of this include standards such as HTML, XHTML and CSS 2. Less sucessful examples include efforts such as WCAG 2. While it may safely be assumed that nothing in the new standard will disrupt the existing order of the Internet, the flip side of this is that there may be no actual working implementation of the new standard (to prove that such is practical), and it may well be that the new standard is not the most efficient or elegant solution to the problem. This may be described as the "top-down" approach.

The other way that standards happen in the real world is for a developer, or typically a small group of developers, to come up with something that works for them, open up community/public comment and collaboration, and eventually submit the standard definition (whihc by then has several working implementations) to standards bodies like the W3C or the Internet Engineering Task Force. This may be seen as the "bottom-up" approach. Its success is largely tied to how effectively it solves what it sets out to, and equally critically, whether it does so in a manner that doesn't convey inherent advantage to a subset of its audience (such as the company employing the creators of the standard). Successful examples of this include vCard and its successor hCard.

Stumbling across the description of hCard (from An Angry Fix by Jeffrey Zeldman, a well-known figure in the online Web-design industry and community) after I had been giving some thought to a problem I had been having with contact information in various formats. Namely, that the information was in various formats, for my (ancient) PalmPilot, each of two different Nokia phones, my email package (Mozilla Thunderbird), and so on, and so on.... Keeping everything synchronised — the mundane necessity of ensuring that any given contact was in each of the needed places with the most recently updated information — is a burden sufficient to preclude any further effort, such as actually communicating anything useful or interesting to those contacts. (Maybe they read this blog...)

What's needed is a free, open source bit of software to take these various directories in varyingly historical formats, apply updates and changes to a single, current-technology directory around something like vCard (or, better, hCard), and then to spit out various dumps of this data to suit the different devices and their differing format requirements. If you think about this for a while, you can think of all sorts of ways that synchronisation could be a real pain...which update gets applied if you enter the same information two different ways on two devices? Suppose that I get energetic and add data to the "Custom Fields" in my Palm to represent data that has specific fields for the phone or Thunderbird — but since I add different data at different times, it's not always consistent? And on and on...

I'm going to keep one eye open over the next few weeks or months for something that does this relatively painlessly (and, of course, if anybody knows of any, please let me know). Otherwise, it's likely to become Item 374 in my medium-priority queue for Tools I Intend To Write (Someday).

Implicit in the first paragraph, and alluded to more directly in the third (see Zeldman et al) is the fact that the W3C has spent the last 2-3 years making abundantly clear who its customers and stakeholders are, and telling those of us who are professionally tied to standard technologies but who are not ourselves multinational corporations flush with cash for endless junkets (and patent payoffs), to take a long walk off the shortest pier available. While this may be seen by some as an efficient use of resources, addressing the corporate sponsors who are the titans of the marketplace anyway, somebody made a good point along the way: the Microsofts and H-Ps and IBMs and so on of the world started out as small shops that nobody had ever heard of. Had the standards of the day been defined less for what made sense from an engineering perspective than a lock-out-the-small-guys marketing directive, the world would be a very different — and likely less advanced — place today. What goes around, comes around — and the W3C in particular is building up a lot of bad blood with the vitally "interested parties" who don't happen to (presently) be among the 200 or so largest corporations on the planet.

What will happen? On the one hand, we'll likely wind up with lots of easily available but proprietary "standards" like Adobe PDF; the word processors I've used for the last four years have supported publishing in PDF without Adobe asking for a dime. On the other hand, we'll have highly marketed, widely Diggable, proprietary-means-you-only-get-it-from-us packages. These may have lively add-on Astroturfed communities, but they won't deliver the business benefits of truly open software; you can't fork the product, you can't completely support yourself, every use you make of the product or technology, in perpetuity, will be subject to the dictates and whims of the company that owns the product. Well and good, you say; they do, in fact, own their product, and have a right to do whatever they like with it. True, but where does that leave customers who incorporate that product into critical business processes? A year or so ago, an American friend of mine told me of one of his clients, who had a hard drive in their accounting server fail. They swapped out the drive, restored from backups, and found they needed to reinstall the order-management package they used — to generate and track every single order from the day the company was founded right up to the guy who just got off the phone — needed a license key. Fine; they call the vendor's toll-free phone number, expecting to be back in business (literally) in a few minutes. Oops. The vendor was bought out by a much larger firm; their version is now three versions out of date and the (new) vendor requires htem to buy an upgrade — at retail — to access the data they've just restored from tape.

When people ask me what the business benefits of open systems are, they don't want to hear a Stallmanesque sermon on the virtues of individual liberties, real though they may be, or the geek chic of cool code, or the cheapskate appeal to "it doesn't cost a thing". It does — in time and effort to convert and adopt within the enterprise. But what you get from it at the end of the day is control over your own business processes; you can keep running a ten-year-old word processor if you choose to, or have your accounting package customised just so, or whatever you can create a business justification for — and it's going to be much easier to cost-justify relatively audacious projects because there are no hidden surprises. Transparency, auditability, control, economy -- those may not be terribly high on the Digg word list; they may not have dozens of ...For Dummies-style books in your local chain bookshop, but people who make their living, and their employees' living, by making the numbers come out right every quarter should understand what I'm talking about. It's about time.

A side note: Anyone who is considering setting up business in Malaysia rather than other nearby countries (Thailand, Vietnam) may well want to consider the level of technical efficiency, customer support, and attitude towards serice of the local telecom quasi-monopoly. For most people and businesses, Telekom Malaysia is the only game in town. As one of the subscribers/victims of their Streamyx ADSL "service" for the last three years, I have watched connection speed and reliabillity plummet as thousands of new subscribers are pushed onto steadily lower and lower tiers of service. I believe, for example, that they now offer a "broadband" connection at 128 Kbps; twice as fast as a standard dialup modem. I am paying for a 2 Mbps — 2,000 Kbps — connection; in the last two months, I have never witnessed transfer rates higher than 400 Kbps, and for the last week never higher than 80 Kbps. If I were living in a capitalist system with competitive markets, I would have choices. In a functionally Stalinist economy where competition against government-linked is tightly controlled, I have no usable choices. It has taken me well over two weeks of trying to post this blog entry. Selemat datang ke Malaysia! (Welcome to Malaysia!)