Making URL Shorteners Less "Evil"

The following is the text of a comment I attempted to post to an excellent post on visitmix.com discussing The Evils of URL Shorteners. I think Hans had some great points, and the comments afterward seem generally thoughtful.

This is a topic which I happen to think is extremely important, for both historical and Internet-governance reasons, and hope to see a real discussion and resolution committed to by the community. Thanks for reading.

---

I agree with the problem completely, if not with the solution. I was a long-time user and enthusiastic supporter of tr.im back in the day (up to what, a couple of months ago?) It was obvious they were doing it more or less as a public service, not as a revenue-generating ad platform; they were apparently independent of Twitter, Facebook and the other "social media" services (which is important; see below) and several other reasons. Unfortunately, since the First Law of the InterWebs seems to be that "no good deed goes unpunished," they got completely hammered beyond any previously credible expectation, and, after trying unsuccessfully to sell the service off, are in the process of pulling the plug.

I think it's absolutely essential that any link-shortening service be completely independent of the large social-media sites like Facebook and Twitter, specifically because of the kind of trust/benevolence issues raised in the earlier comments. We as users on both ends of the link-shortening equation might trust, say, Facebook because their policies at the time led us to believe that nothing dodgy would be done in the process. I think the events of the past few weeks, however, have conclusively proven how illusory and ill-advised that belief can be. Certainly, such a service would give its owner a wealth of valuable marketing data (starting with "here's how many unique visitors clicked through links to this URL, posted by this user"). They could even rather easily implement an obfuscation system, whereby clicking through, say, a face.bk URL would never show the unaltered page, but dynamically rewrite URLs from the target site so that the shortener-operator could have even MORE data to market ("x% of the users who clicked through the shortened URL to get to this site then clicked on this other link," for example). For a simple, benign demonstration of this, view any foreign-language page using Google Translate. (I'm not accusing Google of doing anything underhanded here; they're just the most common example in my usage of dynamic URL rewriting.)

Another security catastrophe that URL shorteners make trivially easy is the man-in-the-middle exploit, either directly or by malware injected into the user's browser by the URL-shortener service. The source of such an attack can be camouflaged rather effectively by a number of means. (To those who would say "no company would knowingly distribute malware", I would remind you of the Sony rootkit debacle.)

So yeah, I resent the fact that I essentially must use a URL-shortener (now j.mp/bit.ly) whenever I send a URL via Twitter. I also really hate the way too many tweets now use Facebook as an intermediary; whenever I see a news item from a known news site or service that includes a Facebook link, I manually open the target site and search for the story there. That is an impediment to the normal usage flow, reducing the value of the original link.

Any URL-shortening service should be transparent and consistent with respect to its policies. I wouldn't even mind seeing some non-Flash ads on an intermediate page. ("In 3 seconds, you will be redirected to www.example.com/somepage, which you requested by clicking on w.eb/2f7gx; click this button or press the Escape key on your keyboard to stop the timer. If you click on the ad on this page, it will open in a new window or tab in your browser.")

Such a service would have to be independent of the Big Names to be trustworthy. It's not for nothing that "that zucks" is becoming a well-known phrase; the service must not offer even the potential for induced shadiness of behaviour.

I'd like to see some sort of non-profit federation or trade association built around the service; the idea being that 1) some minimal standards of behaviour and function could be self-enforced, and especially 2) that member services that fold would have some ability/obligation to have their shortened link targets preserved. This way, there would still be some way of continuing to use links generated from the now-defunct service.

Since the announcement that the Library of Congress will be archiving ALL tweets as an historical- and cultural-research resource, and contemplating a future in which it is expected that URL-shortening services will continue to fold or consolidate, the necessity and urgency of this discussion as an Internet-governance issue should have become clear to everyone. I hope that we can agree on and implement effective solutions before the situation degrades any further.

A Slight Detour: Musing on Open Data Standards as applied to Social Entrepreneurship and Philanthropy

This started out as a conversation on Twitter with @cdegger, @ehrenfoss, @p2173 and other folks following the #opendata, #socent or #10swf hash tags. Twitter is (in)famous for being limited to 140 characters per “tweet”; with the extra hash tags and all, that's reduced to 96. I wrote a reply and used a text editor to break it into "tweets"; by the time I got to “(part 8/nn),” I knew it was crazy to try and tweet an intelligible response.

So, folks, here's what I think; I hope it's more intelligible this way. Comments appreciated, here or on Twitter.

---

What I see #opendata doing for #socent is to allow individuals or groups to build on and share information on opportunities, needs, donors, etc.. This collaboration would use open data formats and tools that iteratively improve philanthropy effectiveness.

Think of how a wiki enables text collaboration, shared learning and discovery, or how instant messaging allows both realtime and time-shifted conversation. Now expand that idea to a sort of "social database" that can be run like a less elitist Wikipedia mated with an RSS feed. Anybody can browse or search the information in the database (needs and offers). They can also get their own local copies of some/all data and have it be updated from the "upstream" source automatically. A smaller group of vetted people can update the "master" data which then gets pushed out to all viewers or subscribers.

This curation is needed to maintain data integrity and to (hopefully) eliminate attacks on or disruptions to the "social database" itself. The sad reality is that any public information on the Internet must have some sort of protection, or it will be damaged or destroyed. I see this as being especially true of a pioneering social-entrepreneurial system like this; there are enough people out there who have a vested interest in this sort of thing not working that security (authentication and validation) must be built in from the start. Otherwise, we will wind up with a situation akin to "spam" and "phishing" with email. Email standards were set in the early days, when the Internet was a primarily academic/scientific resource where all users could legitimately trust each other by default; the current state of the Net is far different. Any open data standards and protocols developed for the "social database" must take this into account.

These open data and protocol standards should be designed with the understanding that they are likely to change over time as the needs of users become better defined and as new opportunities to support those needs present themselves. The first version of a new system (like this) is almost never the simplest, nor will it be the most effective for its purpose. Lessons will be learned that should be folded back into revisions of the standards, in much the same way that later versions of standards like HTML built upon experience gained with earlier versions.

When evolving these data formats and protocols, it is vital that the process be fully transparent, with a balance between building progress and listening to the needs and concerns of the widest possible audience. It is entirely possible that no one standard in a given area will suit all stakeholders. In those instances, some sort of federation built on interchange of some common subset or intermediate format may be helpful. This should be seen as undesirable, however, as it limits the ability of casual or new users to make effective use of the entire distributed system.

The development, maintenance and ownership of standards developed for this project (including necessary legal protection such as copyrights) must be under the auspices of an organization with the visibility and stature to maintain control of the standards, lest they devolve into a balkanized mess that would be as unhelpful to the intended mission as not having any such standards at all. I would expect this organization to be a non-profit organization. Not only will this remove the fiduciary responsibility for monetizing investments made in the technology from the officers of the organization, but other non-profits/NGOs can be expected to cooperate more fully with the parent organization in developing, deploying and maintaining the standards – allowing them to remain open and unencumbered.

Finally, I think it's important to state that I don't see any one type of format as necessarily superior for developing this. I'm aware that there has been a lot of work done with various XML-based systems as part of the #socent build-out to date. After working with XML for approximately ten years now, I have seen some magnificent things done with it, and some absolutely misbegotten things done with it. Particularly with regards to the authentication and validation issues I mentioned earlier, and also with the sheer bulk and relative inefficiency of a large-scale XML data store, there are several issues I can think of. They're solvable, and they're by no means unique to XML, but they are issues that need to be thought about.

EDIT Sunday 18 April: I feel really strongly that one of the things our (distributed) information architecture is going to have to nail from the very beginning is the idea of authentication/verification; does a particular bit of information (what I'd been calling "opportunities, needs, [and] donors" earlier), otherwise we're just laying ourselves open to various black-hat cracking attacks as well as scams, for instance of the "Nigerian 419" variety. I think it's pretty obvious we're going to need some sort of vetting for posters and participants this in turn implies some (loose or otherwise) organization with a necessary minimum amount of administrative/curative overhead to maintain public credibility and apparent control over our own resources. Anybody would be allowed to read basic information, but I think we can all agree on the need for some sort of access control and/or obfuscation of data like individual contact information or some types of legal/financial information that gets tacked into the "social database." This could be pretty straightforward. One hypothetical approach might be as basic as having those who wish to publish information to go through a simple registration process that issues them some piece of private data (possibly even the open standard OAuth authentication that Twitter and others use for applications hooking into their infrastructure). Either alternatively or in conjunction, a public-key cryptography system such as GNU Privacy Guard could be used to prove data came from who it claimed to. For instance, the data to be published could be enclosed in a ZIP file or other archive, along with a "signature" and the identification of a registered publisher. (There's no way that I'm aware of to actually embed the 'signature' itself into the data file: the signature depends on the exact content of the data, and by adding the signature, the content is changed.)

To the non-technical user, the effects of such a system should be:

• The 'Foundation' (for want of a better term) can use already-proven, open standards to enhance member and public confidence in the accuracy and transparency of the content of the "social database". Attempting to "reinvent the wheel" in this area is an oft-proven Bad Idea™;

• The Foundation will be able to develop, deploy, manage and maintain a (potentially) widely-distributed philanthropic/social database architecture that can support a wide variety of organizational/use models;

• Having this sort of authentication and validation will aid in the evolution of the technical and "business" architectures of the system; new services can be layered on top of existing ones by different users as needed.

For instance, if a particular member has announced that they will publish information in a specific version of the schema for the "social database" (say, during registration), any later information purportedly from that member published in an older format should raise warning flags, as it may be a sign of actual or attempted compromise in security and data integrity. A benign incident, such as the member inadvertently using a software tool that submits data in an inappropriate format, can be quickly identified, communicated and rectified.

This will be vital if we are to create a data system which publicly distributes data pertinent to specific members outside those members' control that could include information that must not be altered by outside influences (such as, say, budget information) or information that, for general Internet-security reasons, should not be directly visible to all and sundry (for instance, contact information might be accessible to members but not the casual browsing public).

Again, Standards Matter

People who I've worked with, or worked for, or read my writing here and elsewhere, have probably figured out that I'm a huge fan of standards just about everywhere they make sense: data formats, user interfaces, and so on. After all, why should we have to relearn how to drive a car simply because we buy a new Ford in place of a Toyota that the Government doesn't want us driving anymore? (You see very few old — or even fully-paid-for — cars in Singapore.) The steering wheel, pedals, and other controls are in a familiar layout; any slight differences are quickly adapted to.

Not so with the Western world's most widely-sold word processing software (for instance); when Microsoft Word 2007 for Windows shipped with a different, unique ('innovative', whether or not you find it debatable, is beside the current point) interface. Bloggers bloviated, many users were deeply confused, and corporate help-desk calls (and support/training costs) spiked. People running Windows PCs were very rarely neutral about the change.

A year later, Microsoft shipped Word 2008 for the Mac. Although there were some interface changes, the points of loudest discussion in the Word:Mac user community seemed to be

• the omission of Visual Basic for Applications as an attempted cross-platform macro language; and
• the new semi-proprietary document format, which allowed flawless interchange with Windows users (VBA notwithstanding).

Interface changes, per se, didn't spark nearly as much angst as had the Windows version of the year before. While a certain amount of this should no doubt be attributed to Microsoft's experience with the earlier release, the main reason was both different and obvious.

When developing Mac applications, it's much easier and better to follow the Apple Human Interface Guidelines than to "roll your own" interface. Developers, including Microsoft, are well aware of the ways in which the Mac development tools make your life easier if you structure and style your app to meet the Guidelines (and user expectations), as opposed to how much scutwork needs to be reinvented from scratch to do things differently. Users benefit even more, as the amount of learning needed to use a new app, or a new version of an existing app, are much less than is the average under Windows or Linux. And, unlike far too many Windows programs, Mac programs are usually highly discoverable; the user may not know how to accomplish a desired action, but there is one (and preferably only one) obvious path to follow, and mis-steps are generally not heavily penalised.

Right, "everybody" knows this, so why did I spend five paragraphs restating the reasonably obvious? Because the real intent of this post is to draw your attention to a phenomenon which is a necessary outcome of that standardisation and discovery: it is much easier to switch from one Mac app that performs a given task to another than it is on Windows. Most Mac users periodically switch between different applications for a given purpose, even keeping two or three installed on their systems. When you ask them why, they don't (generally) point to bugs or deficiencies in one product over another; they merely switch between them as their use cases change. For example, though I have both Microsoft Office and Apple iWork on this Mac, I will often create or open smaller Word documents in a simpler application such as AbiWord instead. It doesn't have all the features of Word or Pages, but it has the basics, and it loads more quickly and uses fewer resources than its two "big brothers."

The average Mac user is also generally more willing to try new applications, and generally owns more applications, than is the average for Windows. Since she is confident in her ability to pick up and use a new program, generally without resorting to a manual or even online help, there is a much more open discussion between users and developers, since both have seen a good bit of "the competition" and know what they like and don't like.

More rarely than is the case elsewhere, but not rarely enough, this easy migration from one app to another is due to real or perceived defects in a previously-used program. This happened to me recently; the program I had been using for a few months as my main Twitter client was not showing me all the tweets of people I was following in the "mainline" stream that I would see when I looked at each person's stream individually. Once you start following more than about two or three people, the mainline becomes absolutely indispensable; you simply don't want to have to take the time to look at each stream in isolation. So, I moved to another client, Nambu (now in "private beta" for a new release; version 1.2.1 can be found via web search).

Two immediate observations: I already know how to use this, even though Nambu has a far more dense presentation than my earlier client. And, because of that "dense presentation", it now takes me about a fifth as much time to get through my morning and afternoon Twitter catchups as it did previously. (Multi-column view is the killer feature, guys; there's only one little thing I'd like to see different...)

Again, why make noise about this? Simple: I've been a Windows user (usee?) and developer quite literally as long as there's been a "Windows"; I ran across my 1.0-beta developer kit floppies (5-1/4", of course) a couple of weeks ago (thinking about having them bronzed...or mounted on a dartboard. Maybe both.) But the nasty truth is, I very rarely change applications that perform a given task in Windows. The pain level and the (hopefully temporary) hit on my productivity aren't worth it until the pain becomes absolutely excruciating. I don't have that problem with the Mac, at all. I can try out new applications at will, even daily-workflow applications, secure in the knowledge that

• I already know how to use this, at least well enough to get started, and
• I can go back — or on to another candidate — any time I want to.

There's a word for the feeling that having that kind of freedom, that control over your computing experience gives you:

Empowerment.