New Search

Scarcely a day goes by without yet another "real time search service" developer on the Twitter API declaring that they can do data analysis on the Twitter ecosystem, thus hopefully chasing the unbelievable riches that brand and influencer aggregators believe is there.

Unfortunately, Twitter has spotted it too and has decided if anybody is going to make money, its them. They are "Talking Separately to Microsoft and Google About Big Data-Mining Deals" sez BoomTown.

Sources said a number of scenarios are being discussed to compensate Twitter for its huge and potentially valuable trove of real-time and content-sharing information, generated from the data stream of billions of tweets from its 54 million monthly users.

These include a number of structures, including a payment of several million dollars to Twitter, along with various revenue-sharing proposals that would give Twitter a piece of the revenue made from search results.

The deals, stressed sources close to the situation, are nonexclusive, especially because Twitter’s management is keen to remain independent and also nonpartisan in the growing search battle between Google (GOOG) and Microsoft (MSFT).

This means Yahoo (YHOO)–which recently struck a search-technology and online-advertising partnership with Microsoft–could also license Twitter’s feed to make its search results even more robust, although Boomtown could not determine if the company is in talks with the San Francisco start-up.

What this means is that the true value of the data will be realised. What it also means is that they won't let it go to two-bit startups building their own businesses by giving twitter data away for free to build their own businesses.

From the Back to the Future Dept - Nielsen finds that:

In a nutshell, there is a segment of the online population that uses social media as a core navigation and information discovery tool — roughly 18 percent of users see it as core to finding new information. (See Chart Above - Ed)

Good heavens, who'd'athunk it! (Cough - this was why the whole world went into a tizzy about SocMed, the dream of it being able to capture Googlegold was a big part of the hype story in 2006/7, so well caught up Nielsen )

Anyway, as is the wont of marketing wonks, they divide the world into 3 groups:

At the root of the changing nature of content discovery is the sheer amount of information that is available on the Web. If you want to learn more about the latest smartphone released into the market, your favorite search engine is sure to provide you with hundreds, if not thousands, of articles about the device. But with the increasing number of resources available, it’s difficult to know what you should believe or take at face value. Socializers – those who spend 10 percent or more of their online time on social media – feel this effect more than others do. When asked, 26 percent feel that there is too much information available on the Internet, compared to 18 percent of people who predominantly use portals [Portalists]and just 5 percent of people who primarily use search engines [Searchers].

Can one be both a Socialiser and a Searcher, one wonders....? Seems to me that social media users are much more scared of navigating the sheer volume of info on the 'Net than pure searchers, and want it mediated heavily by people they trust. Portalists are somewhere in between.

Anyway, useful data to back up stuff you've known for 3 years - but the thing that astounded me was the size of the Portalist community. By 2009 I thought it would be in the minority!

I was one of the panellists at Chinwag's "Search is Dead, Long Live New Search" last night, where - ostensibly - we were talking about the rise of real time search and its impact on the older incumbents, ie Google. However, given that many on the panel (self excluded) and in the audience were from the SEO (Search Engine Optimisation) industry, it more turned into a "Future of SEO" talk at times, which I found fascinating in itself.

As I was on the panel (other members of panel can be found in the Chinwag link above), I didn't have time to take copious notes on this discussion but overall the narrative gist - as best I got it - is:

- its been great over the last few years having just one incumbent - Google - to optimise against
- complications are that (i) much of the "80/20" knowledge is leaking to clients, and (ii) there are now many SEO operators so margins are declining
- the industry is being churlish enough to churn itself up, with new "old style" search plays rising (Bing) and a plethora of small but rapidly growing New Search styles emerging, Real Time being the most worrying tight now. This is making optimisation more complex and costly
- Netizens are getting better and better at seeing phoney results, and the future will be in real conversations, not artificial juggling of keywords
- Even worse, all sorts of outsiders - PR Co's and others - are muscling in as SEO becomes part of the overall media plan, further eroding pure SEO margins.

Not a great prognosis, all in all, it would seem. Anyway, I was there to talk about Search itself, so this was the gist of my stuff in answer to the various points raised (I know roughly what I said as I wrote it down on the proverbial back of a paper napkin ):

1. Big Changes in Search in last 5 years

The 5 year look back was the arms race between Google and SEO companies as Google Gnomes battled to keep the Google algorithms one step ahead of the optimisers. Sob stories of online Mom & Pop stores kicked into penury by Google algorithm adjustments were staple fare. The 3 year horizon saw the emergence of new types of media - blogs, video etc - as valid searchable items, which blindsided old school SEO for quite a while (and also Google' algorithmatists). We saw the rise of Technorati and early "social network" based search. The 1 year horizon is the rise of Real Time search, driven mainly by the opportunity provide by the first real time ecosystem, Twitter. (There were real time search engines before, its just that they were searching specific niches or corporate data as no real time consumer system existed, apart from the blogosphere)

2. What do we think is going on now and in the near future

Real Time search is still small, but fast growing - it punches above its weight as it is "on the zeitgeist" - you can see this in real time as Facebook, Google et al have been forced to respond with major changes in thier own architectures to something that is so small. However, real time it is not a replacement for more backward looking search, over time they will blend. The 1 year horizon is the continuing rise of real time and massive innovation in the space. The 3 year horizon is integration, as well as Google coming under increasing margin pressure from these new searchers and new "old style" searchers such as Bing. In 3 years time they will be losing market share and margin (but will be the heavyweight for at least 5 years, probably more like 10). In my view Google is acting increasingly like an incumbent (The Borg, but fluffier as another panelist put it ) and increasingly outsourcing (and buying) innovation, but a number of the other panellists disagreed on this.

But, the key in search is no longer "search" per se - that is to an extent commoditising. The key is filtering to get relevant results, and - in my view anyway - this is where the real battleground for search will be going forward, as relevance is what drives customer usage (which is why real time punches so far above its weight)

3. And 5 years out into the Future?

The 5 year horizon is the ability to search in the "Deep Web" (stuff not yet visible to search engines) as companies make more data accessible, and Filtering to maximise relevance and minimise cr*p.

I also noted my co-panelists responses to this one:

- Structured Data (metadata, semantic data eyc ) allowing far more useful searching
- Social Feeds - helping people get more relevant results
- Pay more for prominence - simple SEO is over and algorithms will get better at culling spamming, flacking etc
- Big bandwidth changes all - look at Korea for lessons of the future

On Big Bandwith, there was quite an interesting discussion on Video search, someone mentioned that YouTube is now the second biggest search engine by search after Google proper. We also had a useful exchange at the time (and afterwards over drinks) about how search and discovery would work for Web TV (in a world of millions of "channels" what does the "Online EPG" look like? In addition there was an aside into the interesting trend for TV to blend with real time ecosystems, eg people commenting via Twitter on programs they were watching at the time (eg #bbcgqt - BBC Question Time - for example - see our notes at the time) and whether TV could align itself with real time systems to counter the Web TV threat.

As always, a fascinating evening, and conversations afterwards at the bar were also very good value - measuring the sociality of websites anyone? - and I look forward to seeing others' writeups which I'll link to as I see them.

Interesting article in Slatest on the memetic nature of news stories. News Dots visualizes the most recent topics in the news as a giant social network. Subjects—represented by the circles above — are connected to one another if they appear together in at least two stories, and the size of the dot is proportional to the total number of times the subject is mentioned:

Like a human social network, the news tends to cluster around popular topics. One clump of dots might relate to a flavor-of-the-week tabloid story (the Jaycee Dugard kidnapping) while another might center on Afghanistan, Iraq, and the military. Most stories are more closely related that you think. The Dugard kidnapping, for example, connects to California Gov. Arnold Schwarzenegger, who connects to the White House, which connects to Afghanistan.

It is essentially doing memeplex analysis (a memeplex is a cluster of memes - memes seldom travel alone, and memetic algorithms try and understand the shifting dynamics of their connections to each other):

How News Dots works

Step 1: Behind the scenes, News Dots scans all the articles from major publications—about 500 a day—and submits them to Calais, a service from Thompson Reuters that automatically "tags" content with all the important keywords: people, places, companies, topics, and so forth. Slate's tool registers any tag that appears at least twice in a story.

Step 2: Each time two tags appear in the same story, this tool tallies one connection between them. For example, a story about a planned troop increase in Afghanistan reform might return tags for President Obama, the White House, and Afghanistan. These topics are now connected:

Step 3: As this tool scans hundreds of stories, this network grows rapidly, and "communities" begin to form among the tags. Subjects that are highly connected—those that appear together in many stories—cluster together in the network. This occurs in the same way that a picture of the social network of your Facebook friends would reveal clusters of friends from high school, college, and work, with some unexpected connections between them when friends belong to multiple cliques.

Step 4: The news network that results is visualized using Slate's custom News Dots tool, which is built using an open-source Actionscript library called Flare. Tags are displayed if they appear in at least four stories, and connections are made if at least two stories link those two subjects. The visualization covers the previous three days of news and is updated daily.

We built a similar system two years ago for the BBC Innovation Labs, the other thing you will see is how over time the links wax and wane as the story twists and turns (there was a similar piece of analysis of just one memeplex story, the US presidential election).

Two years ago, if you mentioned Real Time Search, people laughed at you. That was because, except for very specific applications, there was no real time search ecosystem in Consumerville. Twitter changed that. And now, Wordpress has enacted the RSS Cloud feature - RWW:

RSSCloud is an element that's always been present in the RSS 2.0 spec but has drawn new attention with the rise of interest in the Real-Time Web. The element was just added to the WordPress code this afternoon.

..........

Supporting feed readers will now be able to request updates from WordPress blog feeds as soon as they become available, instead of polling a server periodically to check for updates. (Your blog posts typically get picked up by RSS aggregators 15 to 60 minutes after you posted them - this will change that.) The feature is already being rolled out, several WordPress users report seeing the cloud element in the source code of their RSS feeds.

The return of the Publish/Subscribe (Pub/Sub) model continues unabated.......

Google Reader, the dominant RSS aggregator on the market, began a limited implementation of a related protocol called PubSubHubbub last month. Facebook-acquired FriendFeed worked with Google on that system.

Now RSSCloud has a posse. Half a million blogs are created each month on WordPress and if Google Reader keeps taking its sweet time checking those blogs for updates instead of turning on support for RSSCloud, it's going to look slow as molasses.

In other words the blogosphere will now increasingly coalesce into a real time media ecosystem as well, because why would you wait 2 hours for your blog to be picked up. Which makes our piece yesterday on Technorati all the more poignant - someday, me boys, this should all of been yours....... Still could be, of course, but they are going in a different diretion.

Patenting the Google front Web Page - its like Da Vinci being able to patent head and shoulder portraits of women. Or, as Harrym put it:

Oh my. Really? Perhaps they're doing it on purpose to persuade EU that software patents really are a stupid idea...

There is a serious point here. When I was young and impressionable, a boss of mine remarked that you could tell companies who had stopped innovating (and thus whose long term value was in decline) by three main things:

- Fancy headquarters and hefty top executive perks vs the troops' benefits (beware of big bosses bonusses as they cut the staff's biscuits, I recall him saying)

- Defensive market behaviour rather than leadership (this included "assertive" use of patents )

- The Marcomms & Lobbying departments being closer to the Top Exec floor than the people who did stuff

I've found it a pretty good rule of thumb over 20 years. I'd add that another sign is when they move from in-house initiatives to acquisition for future growth.

Where d'you think the Googleplex scores today?

(Update - fair minded people have been pointing out to me that it could be a defensive move to stop patent trolls. Perhaps, but that is what prior art proof is all about..... )

(Update No 2 - The very smart Graeme Pieterz points out in the comments below that a patent gives the opportunity to countersue, whereas prior art does not. Am feeling very dim, I knew that but forgot about it )

According to Henry Blodget on SAI, Bing is Doomed to Fail

The reason almost 70% of folks use Google is that most folks like Google and most folks are used to Google. If there's a "better" search engine out there, most people probably don't know that it's better (because it's really hard to tell--in part because "better" is in the eye of the beholder), and many people who can tell probably don't care.

I recall similar said for Netscape, IE could never win.....lets see what happens when Bing is packaged as the default search in every WinTel machine.

Reports in the New York Post that Google is taking Bing seriously:

..Co-founder Sergey Brin is so rattled by the launch of Microsoft's rival search engine that he has assembled a team of top engineers to work on urgent upgrades to his Web service, The Post has learned.

Brin, according to sources inside the tech behemoth, is himself leading the team of search-engine specialists in an effort to determine how Bing's crucial search algorithm differs from that used by the company he founded in 1998 with Stanford University classmate Larry Page.

"New search engines have come and gone in the past 10 years, but Bing seems to be of particular interest to Sergey," said one insider, who spoke on the condition of anonymity.

As we wrote at the time Bing was launched this is a serious play compared to previous search engines, Google could be the next Netscape:

The issue for Bing is that while it is – in my view anyway – pretty close to Google in output quality, the question is why would anyone switch over to use it? Ah – it could of course be the default search engine in the IE browser of course. Also, Google is no longer as "cool" as it once was, there has been a bit too much evil leaking out the system for that, so I suspect many will try out the Bing system for a while.....and I suspect that many people - especially regulators etc - will be quote glad to promote a Google competitor.

In fact, the parallels between this and the Browser wars are quite pertinent – by attacking access to search, Microsoft is striking at the heart of Google’s revenue model – no Google search, no Ad revenue. No Ad revenue, no ability to support all these other expensive adventures. Declining revenue also impacts margins, as does competition for Adword style advertising. And Microsoft can load Bing as default on every new Windows machine sold…..

Interesting that its taken 2 weeks or so for the story to come out that Google was seriously worried. We suspect this is because they were waiting to see what its usage settled down to after the launch razmatazz - and the Post records that:

Early statistics show Bing increasing Microsoft's market share by two percentage points, to about 11 percent -- but that the gains largely didn't come from Google or Yahoo!

This is strategically interesting for another reason too - up till now, the Google/Microsoft fight has been one of Google attacking Microsoft's high margin areas - but this is a major push by Microsoft into the heart of Google's economics (as it takes away Ad revenues while also pushing up Google's costs to compete). It implies a change of strategic policy/thought from Microsoft, so the next few years will be fascinating.

(Hat tip CNet for link)

Delicious irony - Googe's Chrome browser will have to offer ad blocking to be competitive, thus shooting its parent's business model in the foot. Los Angeles Times:

Asked for comment, Google did not directly address the issue.

"We are designing Google Chrome's extensions to be flexible enough to support all different types of features, and we are encouraged by the development that we've seen in this area so far," a Google spokesperson wrote in an e-mail.

Surely AdSweep, specifically, isn't the source of that encouragement, right?

"Put a different way, we are encouraged by the work that developers are doing as they experiment with building tools on our extensions platform," the spokesperson wrote.

Google, being an advertising company at its core, probably won't benefit from ...

... this particular extension, which, if it ever became popular, could bleed money from Google's ad revenue. After all, the wide array of free tools the company puts out couldn't exist if not for the ad revenue that supports them.

Realistically though, this probably won't be the catalyst that propels ad blockers beyond the avid Web surfer niche.

"I don't think this really puts a dent into Google's revenue," wrote AdSweep developer Charles-Andre Landemaine in an e-mail. "There are so few users of Chrome, let alone AdSweep, so the loss of revenue for Google is peanuts, really."

I love the System Dynamic of this - the more your Mozilla Breaker grows, the more it breaks your own business model - and if you don't do it, the less likely it is to grow. Talk about eating tour own lunch

We’ve been trying Microsoft’s new search engine, Bing, for the best part of the day and – almost surprised to say – it’s rather good.

The obvious comparison is “how does it rate vs Google” – we’ve been using theis compare and contrast site at BlackDog (see above picture) and throwing various terms at it.

This is no scientific test, but impression are:

1. The initial screen – colours and text – looks more up to date
2. It’s quick, a bugbear of may wannabee search engines
3. It works as well as Google as far as I can see, in terms of results. It gives slight differences (see below) but there is nothing that puts me off.

a. Google seems to have more results in the space, but not provably more relevance (have they compromised too much with Ad requirements?)
b. Google still has a larger indexed area I think, but Bing will no doubt make this up over time
c. The output display is pleasing
d. I even like the term "to bing"

And its not just us, as TechCrunch points out, most people are finding the same.

The issue for Bing is that while it is – in my view anyway – pretty close to Google in output quality, the question is why would anyone switch over to use it. Ah – it could of course be the default search engine in the IE browser of course. Also, Google is no longer as "cool" as it once was, there has been a bit too much evil leaking out the system for that, so I suspect many will try out the Bing system for a while.....and I suspect that many people - especially regulators etc - will be quote glad to promote a Google competitor.

In fact, the parallels between this and the Browser wars are quite pertinent – by attacking access to search, Microsoft is striking at the heart of Google’s revenue model – no Google search, no Ad revenue. No Ad revenue, no ability to support all these other expensive adventures. Declining revenue also impacts margins, as does competition for Adword style advertising. And Microsoft can load Bing as default on every new Windows machine sold…..

Is Google the next Netscape?

(Or even funnier - in 5 years time will we be Binging using Chrome as our browser )