lorennorman.com

A couple videos of me sitting in the bed of Chad’s truck on the way home from the lab. Nothing terribly interesting, just playing with my new 3G wireless card and data plan. I really need to bring my camcorder so i can take better action shots on the way home (or broadcast it live, even!)

Ride Home 1 
Ride Home 2 

TechCrunch just alerted me to this interview with the Twitter co-founders, Evan Williams (@ev) and Biz Stone (@biz), from this afternoon.  Here’s the video, and below are some quotes, notes, and my analysis:

A major strain on their system is power users. For example, people who are followed by tens of thousands of people, and avidly post (Robert Scoble.) The math doesn’t scale well at all, so when you overlap these “power-user” qualities, you clearly get an exponential explosion of activity. I can see how this is nasty to scale. Now imagine what’ll happen when the mainstream finally figures it out! (Hint: it’s happening. Have you seen @comcastcares?)

“Twitter could be down for months.” They’re discussing the fact that they have a plan for fixing their scaling issues, but that the plan won’t be in place tomorrow. They’re complaining that the users won’t accept “months” as an answer, but they really didn’t give any details. I’d really like to hear about their proposed architecture! Will they ::gasp:: shed Rails? Adopt the cloud?

“The ratio of downtime this week is better than last week.” Apparently Twitter has been undergoing immense growth. March was South by Southwest, which is where Twitter was born 2 years ago, meaning an enormous uptake of usage and users. In fact, according to Evan, users doubled in the March-April time frame. They made it through all of it without a crash, supposedly due to some recent optimizations. Of course, the growth has continued (i guarantee it is exponential and the rate is exponential) and further changes are required.

“‘Why don’t you drop Ruby already?’ That’s not exactly a solution…”

“We have a lot of money in the bank, money in the bank doesn’t make the servers run better.”

“We haven’t had serious talks about open-sourcing our software. We have a small group of smart people who haven’t solved this kind of problem before… We have some great advisers… Senior Engineers at Google… too many cooks in the kitchen [if we add more] at this point.”

“…we turn off the really expensive parts of the site so the site will stay up.”  Found that out the hard way recently, haven’t we?  I call them “brownouts”.

“API requests are 10 or 20 times as much traffic as web requests.”

Talking about a stream: “Lots of copies on cache, only one copy on disk.”  Sounds like a nightmare, but i’m sure he doesn’t mean “no backups” or anything so sinister.

“When we built Twitter, we didn’t know what it was.” This quote totally nails the power of Rails! I have to consider Twitter a Rails success.

Talking about what Twitter developers are capable of: “I can break this whole thing [Twitter], it’s fragile. It shouldn’t be, but it is.”  Way to be candid there, guys.

Why don’t you just stop taking new users? “That’s a possibility…”, but new users aren’t nearly as expensive as maturing users, basically, so it isn’t terribly effective.

“Twitter was a side project at Odeo.” 4 guys worked on it on the side “as a lark.” This is why they didn’t build a huge infrastructure up front. Frankly, it wouldn’t have gotten built at all if they had stopped to think about scalability.  (Rails definitely helped here.)

“Sometimes you have to build a prototype to get the point across.”  (Rails again.)

About spammers: “We look a lot at the ratios of users’ activity.” Apparently people are doing “traditional email spam” with Twitter by unfollowing and re-following to keep triggering the “You are being followed by SpammaJamma” emails, over and over again, i suppose in hopes you follow them back.

Any of my Tweetin’ audience have thoughts?

Today I am attending Startup Riot, an event for startups who want to pitch themselves to venture capital and angel investor types. There’s something like 50 startups presenting today, so they each get 3 minutes and 4 slides. It is going to be brutal. Chad and I are here just to watch the bloodbath and do some networking.

UPDATE ABOUT THE VIDEO STREAM: Well, I was broadcasting this event live before lunch, but I was asked to stop by the organizer. (Something about some people still have day jobs, potential SEC trouble, etc. I’m sure i wouldn’t understand…)

UPDATE 2:  Here’s the videos i took until my battery died, neatly archived by Justin.tv.  I’m linking them here for posterity, but they really aren’t that interesting when they’re not live…