In Defense of Twitter

, , C. David Dent ,

Twitter made a faux pas. They changed the @reply model that they had in place. If you don't twitter, this is going to be both boring and incomprehensible. If you do, then hear me out a bit while I set some context for my argument.

Let's set aside the anger, blame, boneheadedness of twitter lying about the reasoning, and the half-assed "fix" they replaced it with. Let us set aside how much we liked what we had, and how much we will miss it for some reasoned examination. Also,Twitter does not discuss numbers, so a lot of this is complete speculation.

First, some terminology:

An @reply is what you send when you click the "reply" button. It references a particular tweet and has coding that allows you to follow the thread of a conversation back to where it started.

An @message is a message directed at someone. The tweet starts @username and is followed by the message. For all intents and purposes it is the same as a @mention, but it WAS handled like an @reply. The difference is it does not reference a previous tweet.

A @Mention is when someone references your name with an @username. Retweets are a form of @mention

Before this change you could choose between three @reply models

1) The "Run Silent" mode where you could not see @mentions, @replies or @messages at all.

2) The "Default" mode where you saw all @replies sent to you or those sent by people you followed to other people you followed. Also you saw @mentions for you and @messages from those you followed.

3) The "Kitchen sink" Mode where you saw all replies sent by people you followed as well as @mentions and @messages.

Twitter set the mode for everyone to ignore any @replies and @messages that weren't directed to you (although @mentions still came through) and said it was what most people wanted. Well it wasn't and the #fixreplies storm began.

Twitter responded by setting a new mode (let's call it "Compromise" mode) which created a distinction between an @reply and an @message. This effectively turned @messages into @mentions instead. The storm brewing on twitter is clearly indicating that this isn't what people wanted.

Okay, I am here to tell you that people need to back off their high horse a little. As a developer, I know the pressure they are under to keep things running smoothly. Unexpected growth can cause scaling problems you could not possibly foresee. But remember that this is a free service and that despite the many people on twitter who use every facet of the service as it was to connect, network, inform, socialize, and have fun...somebody has to plan for things to get bigger and better because the folks who enable you need to make bucks too.

But that aside, here's the problem: For every tweet you need to have additional information. When it was sent? How it was sent? Who sent it? Are their #tags in it? are there @mentions in it? There may be tons of information I've not even fathomed ... links, and all sorts of things. For every iota of information that Twitter tracks in order to link your tweet to useful information like twitter search, timelines, filters, and such is a series of indexes.

An index is basically a list that Twitter can refer to to reference a particular tweet. Since we are talking about tweets here that means that every tweet in the entire timeline has to be scanned for each user to find every tweet that is a @reply, a @message, a @mention or just a tweet. And each of those checks is a call to the database.

Finding who you follow is 1 index. Finding their tweets is another. Finding your mentions in all tweets is another. A fourth for filtering the list to find all replies to people in your list is a fourth. These add up fast.

Contrary to what they've told us (and lied about) it was default mode that was killing them. "Kitchen sink" mode is a cakewalk by comparison. But the logic to maintain three different user modes adds load and logic that is harder to streamline. In other words if everyone uses the same system you can optimize things easier.

If you are showing cross mentions for every user that means you are effectively making 4-5 index seeks for each tweet. This is just picoseconds of time to do, but multiplied by millions can add up to some significant CPU load. Not to mention data-seek time as Databases can make only so many requests at the same time (and it ain't millions, folks).

So imagine if you will when Ashton Kusher and CNN and Oprah all decide to begin promoting themselves in the same month. Twitter is suddenly front and center. You cannot buy this kind of publicity and so the number of twitter users increases by ...let say 10%.

This means effectively making 1 additional index seek for every tweet. That's a load increase of 20%-25% for a mere 10% increase in users. At this rate the fail rate will get worse and fast. Can somebody cue the fail whale?

So the Engineers dream up this "quick fix". They can make a single index seek to find everywhere you are mentioned or directly tweeted. It isn't so different than the default (although it is "slightly" different, but it means that 96% of the users will never notice.

But they have noticed. And to compound matters they have lied about it. Twitter is going to lose users over this (and likely more than the 10% they added...and power users at that). But the fix isn't as easy as going back to the old way.

What this will require is a complete re-think of how they index tweets. This is core-level change and when they roll it out it damn-sure better work. But the folks at twitter may (in light of the aforementioned lie) make it worse for themselves depending on what they do:

1) They may do the "delay shuffle" and ride out the storm and hope it goes away. Result= Twitter becomes friendfeed and disappears.

2) They reinstate the "Kitchen sink" option. This only affects a few users and most of those with 1000 or less followers. It will increase load some but it should not break the back of the servers like the old system did. Result= lots of grumbling, but API filtering to the rescue and 3rd party apps will pick up the slack.

3) They fix it right and proper and make good and restore most if not all of the old functionality. Result= Happy users and good press. But also users who think they can bully twitter to do what they want.

All of this is, however, made under an unfortunate case of bad timing. Twitter has announced that they are planning "paid services" and they are gearing up to enter the "search market". A lot of folks are grumbling that this is a way to squeeze cash from their users. I don't think it is. I do believe that better @reply options will be available to 'premium' users but I don't think that it'll be this simple. And I certainly don't blame them for getting their core service optimized before entering the search market.

My thinking is to wait 1 month to see if they reinstate the "kitchen sink" mode. If that happens I'll believe that they are moving toward the good and proper fix. I won't let them forget, however, that such largess is at my whim. And if after 4-5 months if it isn't fixed I'll probably drop twitter as another good idea lost to the money grubbing in the dot-com world.