Re-scaling Twitter - some lessons for startups

I've been reading this post on scaling Twitter on the Twitter blog with a growing sense of Deja Vu. They note that:

Twitter is, fundamentally, a messaging system. Twitter was not architected as a messaging system, however. For expediency's sake, Twitter was built with technologies and practices that are more appropriate to a content management system. Over the last year and a half we've tried to make our system behave like a messaging system as much as possible, but that's introduced a great deal of complexity and unpredictability.

The Twitter blog then goes on to note that the best advice given to them from the blogosphere was from people who had done it, and pinted to this very good post from Hueniverse - who in one paragraph notes:

The idea that building a large scale web application is trivial or a solved problem is simply ridiculous. In a way it is surprising that there are so few companies out there dealing with commoditizing web developing scaling.

And here's the Deja Vu bit - Broadsight does this sort of stuff, and it was just over a year ago I went to the "Future of Web Apps" in London, and then a bit later a few of us went to the early "Open Coffee" meeting in London, and we got to chat to a whole lot of "Web 2.0" startups. I was left concerned by their technical skills - not because they weren't top class tech guys, but because most of them were what Andrew Orlowski later called "Presentation Layer" architects, who were now trying to build social network based systems, which are essentially based on network service technology, and thus rely hugely on "Infrastructure Layer" expertise.

I mildly suggested to the ones I met that they look in more depth at the transaction/messaging issues, and the issues around database reads etc because after N years in the Internet / Telco / Hosting / OSS game I knew this was where the scaling issues often are, and I knew scaling would be a big issue for them - with network services you can't just throw hardware at them because they behave in non-linear ways. (See this post at the time)

The response I got in too many cases was similar to what the Twitter guys now admit to above - they were not architecting the systems as messaging systems, they were building them as content management systems - because that's what they knew how to do. And - if I may be blunt - there was just a touch of ignorance (not knowing what they did not know) and arrogance (what could some Web 1.0 / Telco old farts tell us, its all different now) and too much "code zealotry" - Ruby on Rails worship was rampant at the time.....

So anyway, I strongly advise ANYONE in a startup building a social network type system to read the post from Twitter, and the accompanying post I linked to, and this one from Dare Obasanjo, so even if you don't believe old farts like us you can learn from these guys' experience - if you don't get network scaling infrastructure people in early, you risk building another Twitter.

And Twitter are being sensible here though.....

Our direction going forward is to replace our existing system, component-by-component, with parts that are designed from the ground up to meet the requirements that have emerged as Twitter has grown. First and foremost amongst those requirements is stability. We're planning for a gradual transition; our existing system will be maintained while new parts are built, and old parts swapped out for new as they're completed. The alternative - scrapping everything for "the big rewrite" - is untenable, particularly given our small (but growing!) engineering and operations team.

As we suggested in an earlier post, that is the best possible approach to take now.