Big Data - Big opportunity?

Interesting post from Nic Brisbourne, summarising a GigaOm article on Big Data case studies. Like Nic I am dubious about some areas so applied a classic 2x2 analysis (see diagram above) to parse where I thought value might lie. His summary is below, my comments are in italics:

I was interested when GigaOM put up a post this morning titled 10 ways big data changes everything. I read through the ten ‘case studies’, and summarised them below. I’ve put my opinion on the trend in italics after a summary of the GigaOM case study. There was, in my opinion, a lot of fluff in the examples they chose, and of the ten there were only two that really stood out to me as areas with the depth and breadth to be home to multiple successful startups, and they were business intelligence applications of big data and virtual assistants.

1. Mining social media and other music sites to predict the next Lady Gaga – GigaOM highlights Next Big Sound as a key company in this market. Will this idea work? So far Next Big Sound has ‘two undisclosed major record labels’ as customers for this service, making it too early to tell. (There was a fascinating book from a few years back called The Predictors, where one of the case studies was the conflict between the analytics vs the culture of the "personal nature" of the similar movie industry)

2. Finding opportunities for energy savings by comparing consumption patterns – observing yourself is powerful in any context and making consumers aware of how the different things they do effect their energy consumption is no different. Comparison with other others is one way to drive behaviour change, but simple displays showing realtime energy consumption may have more impact. (I think there is an opportunity for any business with zillions of fungible datapoints, but the value per is quite low)

3. Virtual assistants – software tools that help us with the more routine tasks we have to deal with. Siri is perhaps the most famous example, but there are many others. GigaOM chose to highlight a datacentre management assistant technology called Autopilot from a German company called Arago. Virtual assistants combine analysis of large volumes of data with artificial intelligence and will, I think, have a huge impact on many aspects of life over the next ten years. (The overall value in this market is potentially huge as it encompasses every type of business activity, and it is doing a relatively high value task - aggregating and simplifying)

4. Data fuelled recommendations – GigaOM gives the example of Foursquare which is making recommendations based on the data it gets from its users checking in and leaving tips and comments. Recommendations get more powerful when they have more data and I think the future will see recommendations based on data from multiple sources – Foursquare, Facebook, Twitter, and anything else that is getting traction with consumers. Ultimately I see this as a subset of the 3. Virtual assistants. (I agree with Nic, these are niche applications, and risk being one trick ponies in a market where real value comes from multi-factor aggregation - but it is a very large market they can impact so I'd say its my 3rd "interesting" sector)

5. Tracking disease epidemics – GigaOM tracks the story of how Twitter tracked a recent outbreak of cholera in Haiti and how that can be of use to aid agencies. There are numerous examples now of social media being used to good effect in crisis situations which is great. I’m not sure if it really counts as big data though. (Niche application of "Business Intelligence tools in my view, and fairly small as a market I suspect)

6. Business intelligence – GigaOM describes how a startup called Parse.ly is helping publishers analyse publishing data to see what content is driving traffic and predict what might work in the future. I think business intelligence is most obvious near term application for big data and that the most obvious short term startup opportunity is technologies and services for this market. Unsurprisingly this is where most of today’s hot big data companies are playing.

7. Mining cellphone billing records – GigaOM describes a number of use cases for this data including helping with malaria education in Kenya. I see cellphone billing records as one more dataset that can be minded to good effect. It is no different in principle to social media data and ultimately the successful services will be the ones that are able to draw value from multiple datasets. (This in my view is like utility analysis - huge amounts of data but relatively low value per unit)

8. Using data to predict and create video hits – Netflix is the case study this time. I am sure that by mining user data Netflix will be able to better guess how popular new video content will be, but it isn’t clear from the GigaOM article how much better it will be than traditional human based analyses which focus on success of similar content in the past.(See my notes on music - but this is a bigger market)

9. Touch screen interfaces for interacting with big data – touchscreens offer new possibilities for manipulating large datasets with novel graph and chart interfaces. I’m not so sure about this one. Funky charts have value in presentation situations, but I think that most of the insight from big data will be gleaned by analysts who work by looking at different cuts of the data and zomming in and zooming out.(I see this as the UI end of any of these systems, I don't think that Big Data will drive me to buy a second tablet - but it may drive replacement of many industrial terminals and PC screens, but IMO it is just upgrade/cannibalisation of the screen market)

10. Hospitals using big data to improve efficiency – GigaOM describes how electronic patient records have delivered some improvements, but that there is a way to go. There is huge opportunity for IT enabled cost savings in healthcare delivery. The challenges are more organisational and bureaucratic than technical though and the opportunity for startups is to build services highly tailored to the healthcare market in which they work rather than exploiting the latest big data technologies.(I see this as a niche application of Business Intelligence/VA/Utility monitoring, but as Nic says the issue here is data privacy)

Interesting its now suddenly so popular, when we built the data analytic MDE in 2007, theer was very little interest in deep analystics of the social media datastream, now there are (I heard at the FT Conference) "50 starving London startups looking for funding for Big Data".

Another point made by Nic is that:

One interesting thought which occurred to me whilst compiling this list is that it will likely get much easier for companies to realise the latent value in their data. Over the years we’ve seen a lot of plans from companies that put some value on data generated as a by-product of their main business and only a few of them succeeded in monetising that value.

With many years experience in this game i'd say the potential is there, but often the culture, mindset and busines models are too differemt/difficult for the company to easily assimilate them. Still, there is nothing like a spinoff.....