broadstuff

I was reading my Economist on the train this afternoon, and there is a fascinating article on Datacentres (an area I spent some years in ). The online version is here. They were talking about the shifting location of datacentres. Its a nice article, but I didn't like this summary of history:

Until a few years ago, the location of servers was an afterthought, says Jonathan Koomey, a consulting professor of environmental engineering at Stanford University. Most sat in cupboards or under desks. The computers in corporate data centres were often housed in the firm's basement. And dedicated “server farms”, which came of age during the dotcom bubble and often housed the machines of internet start-ups, were mostly built in Silicon Valley and other high-tech hubs.

...so I thought I'd add in some of the background.

There have always been servers in backrooms and garages, and there still are - but serious datacentres even then were more carefully planned than one assumes.

Pre the DotCom era, Telco datacentres were far out of urban areas and put in remote places, and designed so they could almost withstand nuclear winter - some were even under ground. IT companies tended to be in big industrial estates.

Come the dotcom era, Web Hosting companies (anyone recall Exodus?) located initially near big backbone PoPs (points of presence) as access points to the internet was not at that point ubiquitous. Then came the cage-huggers - many dotcoms were so poorly managed (and to be fair, remote ops were much poorer) that they pretty much had to be near their server cages each day, so the demand for web hosting space right next door (and even desks in the datacentre) - Silicon valley, Silicon Alley, Soho in London. This phase too passed as more PoPs appeared, and the industry started to locate around huge telepresence hubs.

And then there was 9/11, and suddenly companies wanted datacentres anywhere except in large cities.

The big issue over the years has been the ever increasing packing of racks and thus the increasing power usage per square foot - and as processors are near-perfect power-wastage devices but most are still designed to work at room temperature, for every watt you put into a data centre, you pretty much have to remove a watt - and as cooling is not nearly 100% efficient, you wind up expending more energy on cooling as on driving. In cities in summer (like New York, or Palo Alto) the added heat can tip systems over the edge. The problem with putting datacentres in the boondocks however is that if you need to run powerlines long distances, the cost and wastage is high

Hence the extreme interest in places with (i) cooler climates and (ii) cheap natural power, that are still close enough to backbone PoPs - and as the Economist notes, those sort of sites are not that common.

Microsoft is looking for a site in Siberia where its data can chill. Iceland has begun to market itself as a prime location for data centres, again for the cool climate, but also because of its abundant geothermal energy. Hitachi Data Systems and Data Islandia, a local company, are planning to build a huge data-storage facility (pictured at top of article). It will be underground, for security and to protect the natural landscape.

Back to the pre dotcom world then - plus ca change

And like warehouse construction before, all sorts of towns, local governments etc will start to give grants to confuse the economic arguments. And like Warehouse towns before them, they will be sadly disappointed by the jobs they bring - a shed is a shed, whether for physical or information inventory.

The issue the Economist does not really mention is the increasing Green Lobby, which will start to also influence where datacentres can go. I also think they will have to start to develop chips that can run at higher temperatures, as every degree higher it can run saves a huge amount of cooling power.

I've been reading this post on scaling Twitter on the Twitter blog with a growing sense of Deja Vu. They note that:

Twitter is, fundamentally, a messaging system. Twitter was not architected as a messaging system, however. For expediency's sake, Twitter was built with technologies and practices that are more appropriate to a content management system. Over the last year and a half we've tried to make our system behave like a messaging system as much as possible, but that's introduced a great deal of complexity and unpredictability.

The Twitter blog then goes on to note that the best advice given to them from the blogosphere was from people who had done it, and pinted to this very good post from Hueniverse - who in one paragraph notes:

The idea that building a large scale web application is trivial or a solved problem is simply ridiculous. In a way it is surprising that there are so few companies out there dealing with commoditizing web developing scaling.

And here's the Deja Vu bit - Broadsight does this sort of stuff, and it was just over a year ago I went to the "Future of Web Apps" in London, and then a bit later a few of us went to the early "Open Coffee" meeting in London, and we got to chat to a whole lot of "Web 2.0" startups. I was left concerned by their technical skills - not because they weren't top class tech guys, but because most of them were what Andrew Orlowski later called "Presentation Layer" architects, who were now trying to build social network based systems, which are essentially based on network service technology, and thus rely hugely on "Infrastructure Layer" expertise.

I mildly suggested to the ones I met that they look in more depth at the transaction/messaging issues, and the issues around database reads etc because after N years in the Internet / Telco / Hosting / OSS game I knew this was where the scaling issues often are, and I knew scaling would be a big issue for them - with network services you can't just throw hardware at them because they behave in non-linear ways. (See this post at the time)

The response I got in too many cases was similar to what the Twitter guys now admit to above - they were not architecting the systems as messaging systems, they were building them as content management systems - because that's what they knew how to do. And - if I may be blunt - there was just a touch of ignorance (not knowing what they did not know) and arrogance (what could some Web 1.0 / Telco old farts tell us, its all different now) and too much "code zealotry" - Ruby on Rails worship was rampant at the time.....

So anyway, I strongly advise ANYONE in a startup building a social network type system to read the post from Twitter, and the accompanying post I linked to, and this one from Dare Obasanjo, so even if you don't believe old farts like us you can learn from these guys' experience - if you don't get network scaling infrastructure people in early, you risk building another Twitter.

And Twitter are being sensible here though.....

Our direction going forward is to replace our existing system, component-by-component, with parts that are designed from the ground up to meet the requirements that have emerged as Twitter has grown. First and foremost amongst those requirements is stability. We're planning for a gradual transition; our existing system will be maintained while new parts are built, and old parts swapped out for new as they're completed. The alternative - scrapping everything for "the big rewrite" - is untenable, particularly given our small (but growing!) engineering and operations team.

As we suggested in an earlier post, that is the best possible approach to take now.

It never ceases to amaze me how little people know about how much their privacy is being abused in closed worlds online - today CNet notes that many US companies spy on their own employees communication.

A new survey finds that 41 percent of large companies (those with 20,000 or more employees) are paying staffers to read or otherwise analyze the contents of employees' outbound e-mail.

In the study, which was commissioned by e-mail security provider Proofpoint and conducted by Forrester Research, 44 percent of the companies surveyed said they investigated an e-mail leak of confidential data in the past year and 26 percent said they fired an employee for violating e-mail policies, according to security portal Help Net Security.

Its an open story in the industry fer crying out loud - there are social network analysis companies making a living selling packages to analyse emails, draw up social graphs, define behaviours etc. What makes people think for a second that its only Facebook up to these stunts? And this just joins the ranks of surveillance tools like CCTV etc.

In Europe, because of our data protection laws it is (theoretically) illegal to analyse our emails etc, but if I were employed by a US outfit I'd make sure they are aware, just in case.

I read two interesting articles today, both on companies adjudicating the content of messaging:

Firstly, this story on Search Engine Land about Google maybe interdicting false linkbait stories, Matt Cutts saying:

My quick take is that Google's webmaster guidelines allow for cases such as this: "Google may respond negatively to other misleading practices not listed here (e.g. tricking users by registering misspellings of well-known websites). It's not safe to assume that just because a specific deceptive technique isn't included on this page, Google approves of it."

There's not much more deceptive or misleading than a fake story without any disclosure that the story is hoax.

Secondly, Facebook looking at censoring user messaging:

If you ever get the itch to use the word "yuwie" or perhaps make reference to "wadja.com" - don't bother. "Some of the content you included in this message is not allowed by Facebook," is the message you'll get in response. Both of the above are small social networks, but you can't even send a message about how something disgusting (like yuwie.com's site design) made you say "yuwie, that smells bad!" On principle, the whole thing stinks.

Obscenity isn't blocked by Facebook, I was able to send a message that said "John McCain is a f*cking Nazi" (asterisk free) with no problem.

(Wadja etc are Facebook competitors - f*cking Nazis are not)

Google is trying to be helpful, though it may be in real danger of getting the law of unintended consequences applying. Facebook is just being Facebook - Umair Haque would argue their DNA just started evil, no doubt.

The thing that interests me more about these happy groovy "Web 2.0" players is that they are even considering message interdiction - the "Web 0" usenet, and Web 1.0 tools (email, IM) by and large did not. Clearly though, there is something now in the Zeitgeist and that is concerning to anyone interested in user privacy and other rights

Update - here is another interdiction call - a woman who has been called nasty names and had identity details "outed" on Twitter wants to get her "stalker" banned from the service. Twitter had a look at the person's behaviour and its terms of service, and decided this was not its problem.

I think Twitter are right - this is another case where doing the politically correct / expedient / nice thing - interdicting the nasty-person - is probably the wrong thing to do big picture as it sets a dangerous precedent for the rights of all users. There have always been a**holes on the internet, and the "social net" means that most people - inadvisably in our view - have put more data about themselves on it than they should. I suspect we will see more and more of this on Social Networks as opponents, jilted lovers etc trawl the archives to expose your digital detritus. (Incidentally I see that the complainant also works for a Twitter competitor - hmmmmm )

Last week I ordered something off Amazon for my son - £50's worth - not too much - however, it was a pass through to an Amazon supplier, who then passed delivery through to one of the new private delivery companies, City Link, rather than using the Royal Mail.

Well...one week later, two attempts to get them to deliver on the day scheduled rather than a random day (hello...people work during the day and aren't at home) and it turns out I have to now go across town (London) to pick it up as in theory I have run out of allowances to have it delivered. One upset son, and after N bounces around a Customer Unhelp system in ACD meltdown, one infuriated father.

This is the second time we've been through an Amazon affiliate who has used these people, and the second time I've had to waste a lot of my time finally getting CityLink to step up to the mark.

Two strategic lessons:

(i) The attempt to create competition in Mail in the UK by opening the delivery market has clearly created an industry of low cost, low quality cowboys who are cherry picking the high volume urban routes, and undercutting services to the bone - leaving the Royal Mail trying to fund the mandatory deliveries to the Outer Hebrides while being scraped in the cities. This is not in the customer's interests - if the Royal Mail can't deliver and put the parcel in their depot, for most people its a short car ride to get it, not schlep across town after X hours on the phone dealing with people who clearly are not geared up to serve frustrated suck....customers.

Net net is that good delivery businesses are driven out, Royal Mail has to close down other services that are useful to customers (like post offices), cowboys selling crap services win, and total loss of benefit to end customer initially - but then also to the mail order businesses that use the CityLinks of this world, as - based on my experience anyway - people will be less and less willing to buy bulkier goods that can't come from a reliable source, and if they can't specify it they will go somewhere else.

(ii) Risk for Amazon - I am now - after the second time with this experience - rethinking what I buy through Amazon, as I don't want a 3rd repeat of buying stuff through an affiliate who then palms off delivery on a Cheap n Nasty service. I think they will need to (a) vet their affiliates delivery policies and probably (b) allow me to have greater control of specifying what I want than currently. In other words, they will need to take more control, or risk:

- parts of their extended business model withering away due to its unreliability

- the sh*t sticking to them - I am now sufficiently angry with the whole supply chain, and suspect that Amazon has over-extended itself in the pursuit of growth, to start to be more wary of someone I once loved and trusted. I'm blogging about it, so it enters the whole viral word of mouth chain, and I suspect I am far from alone.

The whole online shopping / shipping game is based on trusted, reliable physical delivery, and if that falters then the shock waves up the chain will be fairly hig impact.

( I was typing this while on the various ACD systems, I finally reached a harassed service rep, who has promised delivery tomorrow - we shall see.... )

Seems like we're not the only ones unhappy with the idea of people talking on their phones on 'planes - a Yahoo survey shows most people don't want it:

"There is an increasing debate about the use of mobile phones on airplanes," said Bruce Stewart, vice president and general manager, Connected Life Americas, Yahoo!. "This new survey overwhelmingly proves the desire of consumers to stay connected to the people, information, interests and passions that are important to them while in-flight, but they don't want to be forced to listen to the conversation of the passenger sitting next to them."

Across the U.S.A., but especially in the West, most consumers agree that mobile phone usage on planes should be restricted to non-talking features only:

-- West: 83 percent

-- Northeast: 72 percent

-- Midwest: 73 percent

-- South: 69 percent

More than half of consumers (60 percent) would want to use silent features while flying on an airplane such as:

-- Text message: 38 percent

-- Access e-mail: 28 percent

-- Play games: 29 percent

Age impacts consumers' mobile phone service preferences while in flight:

-- 35-44 year olds: 43 percent, more so than any other age group, would want to check e-mail on their mobile

-- 18-34 year olds: 62 percent would want to text and 29 percent would want to instant message (IM)

The Talking Zone

If usage of mobile phones was allowed while in-flight, more than two out of three (69 percent) consumers agreed that there should be a designated area of an airplane for passengers who want to talk on their mobile phones. In fact, older consumers (76 percent) and women (73 percent) are more likely to agree than their younger (64 percent) and male (64 percent) counterparts.

Lets hope this stops the Airlines, who have sadly sometimes been known to put profit ahead of customer benefit

I gave a talk at the Converged Services Congress this week, and it gave me an opportunity to reflect on “The Convergence” over the last 5 years or so, and on what we all thought would happen then versus now.

Convergence is a fascinating area as its where Telcos, IT Co's, IP Co's and Media Co's all...converge, and is thus a fairly frothy area. I don't know many people apart from us who work with Telcos, Media Co's Web Co's etc - at least I don't see many other people across these various spheres at conferences - but by and large they are all doing the same thing.

What is also interesting to me is to to see how "Convergence" changes over the years - my impression is it hit the top of the "Hype Curve" about 2 years ago and we are now in the downsliding, "reality finding" stage as people increasingly realise they are Crossing the Chasm.

Some thoughts from this year's conference:

1. The Future will come more slowly than you think

The first thing – clearly – is that it would happen much slower than the breathy predictions of 5 years ago. From IPTV to Mobile Web the Analysts’ projected growths of 5 years ago have been woefully short in most areas. There are a variety of reasons for this (as there always are) but a general rule of thumb – I propose – is to immediately halve any Analysts forecasts for 5 years out, for anything. Except good old broadband – for that, there is an insatiable demand.

2. The Future will happen differently to the best laid plans of mice, men and Mobile TV - and will be Over The Top

Things that were hot hot hot a few years ago have been quietly dropped by the wayside – IPTV, Mobile TV, Mobile Web, IMS, Femtocells in Europe were all examples. What is interesting is in each case there was a reason that, in hindsight, was clear – a service point not met, a difference in the way customers actually solved a problem rather than used a service, etc - that was clear at the time but hope (or desperation) over-rode experience.

Take Mobile TV for example - every Telco thought it was a saviour in 2004, as trialling in 2006, and quietly dropped by 2008. We predicted this in 2006 for a client (sadly before we started the blog) simply because the market confusion / cost benefit didn't work - but we summarised this last year in a post on the game theory in mobile and why it would fail here

And we predicted that the future of mobile TV would therefore in future be in over-the-top services in this post here - And we are now living in a world of Qik, Seesmic etc - its Mobile TV, but not as the Mobile Telcos know it.

3. You can't get convergence by just stapling together existing services

Stefan Volck of Deutsche Telkom presented what I thought was a very useful analysis of the future history of Convergence - he was talking about IPTV / Web TV specifically but I think its good generally I have reproduced it below with my generalised interpretation:




4. Advertising is not going to be a cure-all

Although advertising will be a part of the converged business model, there is increasing evidence recorded across the piece that (ab)using advertising kills the thing it loves - customers don't like it, no matter how hopeful the purveyors of targeted/ unique / personalised Adspam - and the over-reliane of advertising cripples the offering, much like overdeveloped beachfronts eventually drive sun seeking tourists away. (On a related note, I see that Facebook is showing its redesigned service less poking we are told)

5. The Convergence business model is not going to be a saviour driving large increases on new customers or even new ARPU

The evidence - from the conference anyway - is that much of the benefit at the moment is in increasing "stickiness" (ie reducing churn) - which for existing large players is a key thing as in a competitive market (think mobile, XDSL, mature Social Networking) the benefits of customer retention far outweighs the new customer acquisition costs

6. Its all about the Customer, stupid

More and more the realisation is that its all about giving the customer what they want, not want companies want to give them - customers have an irritating habit of ignoring stuff they don't like. I posted earlier on a good summary of lessons learned from successful services from Daniel Holle, over here. Wenka Booj of KPN gave a very good study of how messages from customer research can be subverted by internal interests within an organisation.

7. Twitter etc are interesting, but can these things scale?

Contrary to the (sometimes smug imho) view among the Web 2.0 fraternity that Telcos etc "don't get it". the reality is more that they do (most people had heard of Twitter, and this was besuited TelcoLand) - what they don't get is how you hope to succeed with networked services if they don't scale. They also know a thing or two about scaling, so services like Twitter interest them, but they scratch their heads as to why one wouldn't design such a service to scale from day one. (This is a classic tension between the IT heritage of Web 2.0, which is more the "get it out there and let customers fix the bugs", and of Telcos who are more concerned with "Right First Time")

8. Don't underestimate the Laptop

The laptop is moving from humble corporate workhorse to being a fashion item, and a mobile tool - nearly 90% of mobile 3G data in Europe is sold to laptops. The iPhone is essentially the closest current mobile that looks like a laptop (ie works on the mobile 'net without you noticing) but the Eee PC is the beginning of the Mobile PC play.

9. Overcomplicating converged services is a no-no

The example given was the humble washing machine - the industry has moved away from all singing, all dancing devices to simpler ones again, as customers just don't like that level of complexity. There is a subtrend, in that Moore's Law is driving devices to commodity pricing - how does the industry change when the $5 disposable phone emerges - which it will by 2010 we believe

10. Large companies will continue to be sold pups because they lie to themselves

A combination of not being able to match VC/Offset/Free backed business models with traditional business hurdle rates, plus unrealistic profit targets, internal politics, and "mainstream service" priorities will continue to make it very hard for large corporates to launch the correct ripostes to new startups, even though people in these companies know what they need to do. The only way to solve this is to develop services off to one side that cannot be smothered by the Mothership.

Course taken on Read Write Web over here, takes about 5 mins to read, PhD handed out at door on way out

Its a good starter for 101 and goes through all the usual suspects - Web 2.0, Blogs, Wikis, Facebook YouTube, Twitter etc etc - but as with so much Social Merdia stuff in the blogpress there is sweet FA on why you use X and not Y, when A works and not B, etc etc

The sort of analysis such as we sometimes expose, for example here and here and here

The problem with not making any attempt at analysis in Social Media U stories is that it makes the whole scene looks like one big happy fluffy cloud, without any real structure to where what lies in the Ecosystem, Value chain, etc etc, and it frustrates business minded people who look at all this miasma and say "Where's the ROI? What do I do first? What impact does X have over Y?

( Shameless punt of Broadsight services - we do 1/2 day, 1 day and 2 day workshops where we go through a lot more of our analysis in detail for clients, tailored to their industry - if you're interested, contact us)

...or rather the same basic story hits Techmeme twice....the one before was on May 14th - see here and was US based, now its gone global.

Cute PR by e-Marketer to get on Techmeme twice in a week by re-releasing basically the same stuff - it doesn't say much about us Techmeme readers, who clearly have the memory of a Dory-fish though

From the Wall St Journal:

Microsoft Corp. hopes to make gains on Google Inc. in the lucrative business of Internet search through a new service that pays consumers who buy items they find through the software company's search service.

They must've been reading Broadstuff - we noted last year, re matching YouTube, that organic growth via customer motivation was a real option:

Broadstuff Biz Case (expurgated version*)

Cost of Service build - $1m
Cost of Operation before breakeven - $100m
Cost of "persuading" critical mass of YouTube UGC videos onto new service = $5 / video x 50 million clips = $ 250m (added to their content)

No reason one can't apply it to search......for a while anyway.