TechCrunch just alerted me to this interview with the Twitter co-founders, Evan Williams (@ev) and Biz Stone (@biz), from this afternoon. Here’s the video, and below are some quotes, notes, and my analysis:
A major strain on their system is power users. For example, people who are followed by tens of thousands of people, and avidly post (Robert Scoble.) The math doesn’t scale well at all, so when you overlap these “power-user” qualities, you clearly get an exponential explosion of activity. I can see how this is nasty to scale. Now imagine what’ll happen when the mainstream finally figures it out! (Hint: it’s happening. Have you seen @comcastcares?)
“Twitter could be down for months.” They’re discussing the fact that they have a plan for fixing their scaling issues, but that the plan won’t be in place tomorrow. They’re complaining that the users won’t accept “months” as an answer, but they really didn’t give any details. I’d really like to hear about their proposed architecture! Will they ::gasp:: shed Rails? Adopt the cloud?
“The ratio of downtime this week is better than last week.” Apparently Twitter has been undergoing immense growth. March was South by Southwest, which is where Twitter was born 2 years ago, meaning an enormous uptake of usage and users. In fact, according to Evan, users doubled in the March-April time frame. They made it through all of it without a crash, supposedly due to some recent optimizations. Of course, the growth has continued (i guarantee it is exponential and the rate is exponential) and further changes are required.
“‘Why don’t you drop Ruby already?’ That’s not exactly a solution…”
“We have a lot of money in the bank, money in the bank doesn’t make the servers run better.”
“We haven’t had serious talks about open-sourcing our software. We have a small group of smart people who haven’t solved this kind of problem before… We have some great advisers… Senior Engineers at Google… too many cooks in the kitchen [if we add more] at this point.”
“…we turn off the really expensive parts of the site so the site will stay up.” Found that out the hard way recently, haven’t we? I call them “brownouts”.
“API requests are 10 or 20 times as much traffic as web requests.”
Talking about a stream: “Lots of copies on cache, only one copy on disk.” Sounds like a nightmare, but i’m sure he doesn’t mean “no backups” or anything so sinister.
“When we built Twitter, we didn’t know what it was.” This quote totally nails the power of Rails! I have to consider Twitter a Rails success.
Talking about what Twitter developers are capable of: “I can break this whole thing [Twitter], it’s fragile. It shouldn’t be, but it is.” Way to be candid there, guys.
Why don’t you just stop taking new users? “That’s a possibility…”, but new users aren’t nearly as expensive as maturing users, basically, so it isn’t terribly effective.
“Twitter was a side project at Odeo.” 4 guys worked on it on the side “as a lark.” This is why they didn’t build a huge infrastructure up front. Frankly, it wouldn’t have gotten built at all if they had stopped to think about scalability. (Rails definitely helped here.)
“Sometimes you have to build a prototype to get the point across.” (Rails again.)
About spammers: “We look a lot at the ratios of users’ activity.” Apparently people are doing “traditional email spam” with Twitter by unfollowing and re-following to keep triggering the “You are being followed by SpammaJamma” emails, over and over again, i suppose in hopes you follow them back.
Any of my Tweetin’ audience have thoughts?
well to be honest, if i were ev i’d just say point blank, no twitter user can have more than 2000 followers. after that it becomes less of what twitter is meant to be, which is a microblogging platform in the social sense. when people like scoble use twitter to puff up their reach, it becomes less social and more of a broadcast medium which is not what twitter is meant to be.
it’s a simple problem to solve if they had some balls and specified what the true point of seesmic was. they’re trying to be all things to all people and that is always a trap.
would you miss tweets from scoble? i surely wouldn’t. do you get irritated when twitter goes down sporadically, hell yeah!
which is more important to you?
Comment by sean808080 — June 21, 2008 @ 9:13 pm
i meant twitter of course. damn booze!
Comment by sean808080 — June 21, 2008 @ 9:14 pm
Thanks for your thoughts, Sean. A hard limit on followers would indeed mitigate the problem of power-users, but I have to admit that I think a lot of the power and value of Twitter is that it DOESN’T seek to tell you how to use it. This is shown to be a strength as we see more and more services leveraging the Twitter API, which is embodies the spirit of “do what you want with it.”
I think the real solution is a switch in the organization of their architecture. Twitter should be organized as a message passing system, not “blog style” where everything goes straight into the database and then gets served from the database. It’s actually quite tempting to spend a weekend reimplementing Twitter using Amazon SQS.
As for Scoble, I’ve unfollowed him already due to too much noise! Others, like Leo Laporte and friends, are much better, so how can you tell users they aren’t allowed to follow him because too many already are? Besides, wouldn’t they just start creating multiple accounts and cross-posting to them all? That’d probably be a whole new scaling problem…
Cheers!
Comment by loren — June 21, 2008 @ 11:02 pm