New Search

You may recall we wrote a story just after the Christmas holidays about the "Increasing Uselessness of Google Search" - well, it got so far as being mentioned on TWIG (where Jeff Jarvis roundly dissed us for not knowing what we were talking about*, despite a lot of agreement by ordinary users that they too had experienced a similar thing). Well, today, Google admitted - in their own roundabout "there was no problem, and here are some changes to fix the non-problem" way, that there was a problem:

First, the "there is no problem" bit - Matt Cutts:

Reading through some of these recent articles, you might ask whether our search quality has gotten worse. The short answer is that according to the evaluation metrics that we’ve refined over more than a decade, Google’s search quality is better than it has ever been in terms of relevance, freshness and comprehensiveness. Today, English-language spam in Google’s results is less than half what it was five years ago, and spam in most other languages is even lower than in English.

Tut - silly users - the algorithm says there is No Problem, so....anyway, then comes the "so now we are fixing the problem that doesn't exist" bit:

As we’ve increased both our size and freshness in recent months, we’ve naturally indexed a lot of good content and some spam as well. To respond to that challenge, we recently launched a redesigned document-level classifier that makes it harder for spammy on-page content to rank highly. The new classifier is better at detecting spam on individual web pages, e.g., repeated spammy words—the sort of phrases you tend to see in junky, automated, self-promoting blog comments. We’ve also radically improved our ability to detect hacked sites, which were a major source of spam in 2010. And we’re evaluating multiple changes that should help drive spam levels even lower, including one change that primarily affects sites that copy others’ content and sites with low levels of original content. We’ll continue to explore ways to reduce spam, including new ways for users to give more explicit feedback about spammy and low-quality sites.

.......

As “pure webspam” has decreased over time, attention has shifted instead to “content farms,” which are sites with shallow or low-quality content. In 2010, we launched two major algorithmic changes focused on low-quality sites. Nonetheless, we hear the feedback from the web loud and clear: people are asking for even stronger action on content farms and sites that consist primarily of spammy or low-quality content.

While I hear all this, and am heartened by it, I do believe there is a little bit of trying to tell us that we didn't see what we saw. To reiterate what we observed empirically, in late 2010 and early 2011, if you clicked on something monetisable (like buying a consumer product) you found that most of Page One would lead to sites that basically served more links and Ads, and others that basically copied Wikipedia text and links and Ads, and I wrote the article because my family (and some friends) also remarked on it, and judging by the response a lot of other people noticed it as well. (searching for say "Byzantine Iconoclasts" does not contain spam, but then there is b*gger all money in that, these days anyway). So, I am delighted that these steps are being taken, but I suspect that Google's attention has been refocussed in early 2011 by what was, essentially, a massive user outcry. I have just searched again for the things I was looking for in November/December, and there is a marked improvement today - seeing the camera brand name's website up first again is soooo 2006 :-0
.

So while I am heartened, I am pretty certain these algorithms have mainly come in very late 2010/early 2011, and may I hypothesize that it was mainly due to the user outcry - and I would also hypothesize that this is why I am reading Matt Cutt's post on 21 January 2011, not say 07 January 2011

Also, perish the thought that we may wonder if they may have been making money in this (potentially) symbiotic relationship:

To be crystal clear:

- Google absolutely takes action on sites that violate our quality guidelines regardless of whether they have ads powered by Google;
- Displaying Google ads does not help a site’s rankings in Google; and
- Buying Google ads does not increase a site’s rankings in Google’s search results.

These principles have always applied, but it’s important to affirm they still hold true.

Well, whatever the reason, I am delighted at the changes made so far, and I look forward to proof positive that they are not making money from the spamsites. A rapid deterioration again in a few months would be a "thing that made me go "Ummm".

*In fairness, he then went on a rant about Buzz, which I enjoyed

Techmeme is now trawling Twitter as well as blogs and the online news world to get an indication of what is being talked about - Wired:

On Thursday, Techmeme chief Gabe Rivera announced that the site would now be aggregating tweets — the 140-character micro-posts found on Twitter.

The move represents another milestone in Twitter’s seemingly inexorable march toward becoming a central news water cooler of the internet era.

“I think this is yet another step in Twitter becoming a mainstream news service, since more people will recognize it as such now,” Rivera said in an email to Wired.com. “Content-wise, tweets will never play the leading role, but they’ll play a supporting role, and will act as glue and early warning.”

Why do it, and why now - after all, Twitter has been around awhile? I think the answer is here:

Rivera said he’s not sure if the move will boost traffic, but he said the goal of the shift is to improve the site, “which should expand our readership.”

Au contraire, I suspect the issue Techmeme has been having is that people are linking direct from their Twitter aggregators to stories, rather than going to Techmeme first, a trend which could eventually make one's Twitter aggregator the "Go To" news site (and which is what Tweetmeme et al are trying to do.) If you have the real time search and aggregation already, adding Twitter is not hard to do*

So are blogs therefore dead, as Business Insider argues - on Twitter?

“Now it’s official. Blogs are dead.”

No, we would argue, and that twt shows why - its a statement, without any reasoning or backup. I for one am not going to use a service that just points me to one liners like that. There has to be some beef somewhere, and blogs are the best way of delivering that.

*A bit of an aside - we actually built a blog/news search system a few years ago (see here), and looked at turning it into a Twitter search (it's not that hard) but were darned if we could see a sustainable business model for just reading twts.

The meme of Google's sputtering search product has certainly taken off, and that of course brings the apologistas out of the woodwork - one of the most amusing is this one on MediaBeat, which somewhat circularly argues that:

I saw it coming first:

I can only ask: What took so long? I first blogged about Google’s increasingly terrible search results in October 2007. If you search for any topic that is monetizable, such as “iPod Connectivity” or “Futon Filling”, you will see pages and pages of search results selling products and very few that actually answer your query. In contrast, if you search for something that isn’t monetizable, say “bridge construction,” it is like going 10 years back into a search time machine.

You are all wrong, Google is working on it - but they can't win

But here’s what these late-blooming critics miss: Yes, Google’s search results do indeed suck. But Google’s fixing it.

......

The folks at Google have not been asleep at the wheel. They are well aware that their search results were being increasingly gamed by search marketers and that this was not a battle they were going to win. The answer has been to dump the famous blue links on which Google built its business.

Umm..actually, most observers have been noticing this for some time, certainly since 2006 when we first wrote on New Search - the reason the meme has taken off like wildfire now is that it has got noticeably worse since 2009! Anyway, the argument is that Gooogle is now putting in Niche Search un-noticed by all us dumbos, so even though its "not a battle they could win", they have in fact won:

But they have won anyway

Over the past couple of years, Google has progressively added vertical search results above its regular results. When you search for the weather, businesses, stock quotes, popular videos, music, addresses, airplane flight status, and more, the search results of what you are looking for are presented immediately. The vast majority of users are no longer clicking through pages of Google results. They are instantly getting an answer to their question.

Google is in the unique position of being able to learn from billions and billions of queries what is relevant and what can be verticalized into immediate results. Google’s search value proposition has now transitioned to immediately answering your question, with the option of sifting through additional results. And that’s through a combination of computing power and accumulated data that competitors just can’t match.

For those of us who have watched this transition closely and attentively over the past few years, it has been an amazing feat that should be commended. So while I am the first to make fun of Google’s various product failures, Google search is no longer one of them.:

So, all the current people who are seeing a problem are clearly wrong, its gone now - move on, nothing to see here. Unfortunately for this argument, SEO Watch did a survey at the same time and found that (surprise) there was a problem, Bing getting better results by 63 to 52 points.

There is the start of a good debate for New Search aficionados on Stowe Boyd's post here:

We will find everything through social relationships: what washing machine to buy, or the best Thai restaurant in Beacon NY, or the company that makes the horizontal corduroys. people that care about these issues, and to who we matter, will share meaning with us: they have beliefs that they can justify, also called knowledge.

Google is only the echo of our linking behavior, a second-order derivative of our combined gestures. But generally, we would be happier with fewer results from trusted sources, and the rise of social tools makes that almost as fast as Google search.

Interesting response on Hacker News to my and others' posts on Google Adspam:

Given the current high-ranking thread about spammy sites in Google results, it strikes me that a very simple solution would be to let logged-in users blacklist sites.

Bam, no more wareseeker or efreedom.

This would solve a lot of people's complaints in one fell swoop.

There are greasemonkey etc. scripts to do this, but they're tied to a single browser on a single machine. A global filter (like in gmail) would be so much more useful.

Would this be particularly hard to do?

Comments are interesting too - firstly, there is the exploration of a social graph blacklisting:

I think a personal black-list would be ideal initially as those most motivated would be most helped, i.e. the majority of people who might not care about the status quo are then not impacted at all.

As mentioned above, then introducing the shared-ranking via the social graph would be the next logical step. It could be something opt-in'd to ease adoption.

Then, ideally (and this is my personal 'white whale' problem) it would be great to imagine something where the user could whitelist through no action of their own rather than have to do any work to block, i.e. use the result set 'hit' of what's clicked in the results to act as a personal ranking upvote.

There's some interesting engineering issues of per-user indexing though, but hey, you wanted to work at Google right?

......

The blacklist does not only have to affect me, just throw in the blacklists of my social graph too. These are the people I trust.

Other ideas are a "report spam" system:

I want a search results page similar to the "Priority Inbox" we got recently in gmail. Set sane defaults and let me override them with "Important/Notimportant" buttons (or thumbs up/down or whatever) next to results. Let it learn what I think is a good result for my needs. If you make it a little bit social, make sure you weight other people's opinions by how much they agree with my own in other areas (making it harder for sockpuppets to muddy the waters)

All in all it goes to a theme we have been banging on for quite a while about, ie allowing user-defined filtering for web-services (Whether Google, Twitter etc). But as many point out, Google is conflicted here, as these sites generate fast churn of click through pages so a high Ad turnover - so are probably not motivated to do it:

A million mailboxes were seemingly seeded with Quora invites over the last few days (grassroots or astroturf? the jury is still out, the question unanswered), a million flowers thenceforth bloomed, and our twitterstream filled up with irritating autotwts..

A quick aside on Crowdsourced Q&A services - they have been touted as "alternate search" for years, but crowdsourciing answers has its own problems. Their history is one of being useful at first, but dying under the weight of spurious questions, cr*p answers and pure spam over time. But it is a good time now, as those that launched before did it against a background of Google search being effective, but that is not so much the case anymore - so timing is propitious.

Here is the Broadstuff's take on its probable future therefore:

1. Quora is just the next iteration for the "group answering" service, it's hard to tell if it is any better in workflow terms to what has gone before - nothing jumps out right now.

2. Right now its fun because its new and small, there is thus a minimum of spam and spurious answers, and all the "cool people" are on it talking about what they like to talk about, so it's the new new shiny thing

3. If the past is anything to go by, however, if it takes off it will fill up with dumb punters and spam.

4. In that case, if not controlled, the answers will become less useful (see Yahoo Answers et al) and people will stop using it

So the key task for the team is to build the system such that dumb punter impacts are mitigated, that spam is minimised, and that - above all - the service gives enough good answers to questions. Do this, and there is an alternative search system in the making, Fail to do do and it joins the ghost town question services out there, with their tumbleweeds of hanging, unanswered questions except for spam replies. Personally I have always felt that the way to do this is rank the answerers as well as the answers and allow user filtering.

However, I do fail to see the difference between these services and a million "vertical" system on Yahoo groups and various web services, with the exception that you only join one service here rather than all those verticals - but then you also have to deal with unfocussed answers and grockles (Having read quite a few threads, I cannot say that the answers are of higher quality than I have seen anywhere else, but at least at the moment there are answers).

So good luck....they will need it.

Update - Good Lord - a Techcrunch story critiqueing it a tad (TechCrunch has been promoting Quora something awful!)

Update - and now, 3 weeks later, a GigaOm story in the face of universal "meh" We salute thee, PR team of Quora, but you are turning it into a bit of case study!

Practice is starting to look like theory. When Social Networking started, it was clear that people would use it to find things that way as well as using search engines. Its just that the volumes were small and Social Networks, by and large, had crap comms to share things by. Twitter changed all that, and now Facebook - having implemented far better and simpler comms - is finding the same:

According to Web measurement firm Compete Inc., Facebook has passed search-engine giant Google to become the top source for traffic to major portals like Yahoo and MSN, and is among the leaders for other types of sites.

For what its worth this blog has had more referrals from Twitter than Google for some time, which has made us far less impressed with all the arcania of Google SEO).

But now that its happening in scale, it puts Google's model under pressure as it is simply cruder for Advertising:

David Berkowitz, director of emerging media and client strategy for the digital marketing firm 360i of New York, said the importance of search engines isn't going away.

"But there's always been one downside to search," he said. "Consumers only spend about 5 percent of their time online searching and the other 95 percent of the time at the destination. Social media is quickly accounting for a large percentage of that 95 percent. Google's biggest acquisitions, DoubleClick and YouTube, have been all about playing a big role in the rest of consumers' Web usage."

But Social search just isn't scalable without the Signal to Noise ratio getting unmanageable - so we expect to see a whole raft of Social Algorithms emerge to help automate, speed up and filter the process. How do we know this - because we, and a whole lot of other people we know, are fiddling with them, and they show a lot of early promise.

Steve Rubel hypothesized Buzz was launched half baked and half cocked to get out there before Facebook launched its FBMail service, and that may be so - but to our mind the real Buzz is in the algorithms dealing with the blend of social graph and social data.

Not only can one tell a lot about you by the searches you make, seems a lot can also be deduced from the search engine we use, as a WPP survey of 17,000 US internet users shows - Ad Age:

What does your search engine say about you? Well, if it's Bing, you're probably an early adopter, but you also visit, shop and ultimately make purchases from Walmart more than other search-engine users. Google searchers, on the other hand, are partial to Target and Amazon, and Yahoo searchers have a strong preference for wireless service from AT&T and Sprint.

Google users are more likely to book a flight online at JetBlue or make a reservation on Hotwire. They are also more likely to do research on a Lexus, while Bing users tend more toward Toyota. Aside from uncovering the research and purchasing habits of consumers, the study claims to have identified a host of demographic and psychographic information on search-engine users.

For instance, AOL customers feel less intellectual than their peers, are 55 and older, spend their money more responsibly, want to blend in to the crowd, feel like they've gotten a raw deal out of life, expect less from their future and, believe it or not, still use dial-up modems. Bing users are middle-aged, highly educated tech-savvy individuals who consider themselves to be average and spend more than 10 hours a week online.

Googlers tend to be the average internet Joe, according to the study. The search leader's loyalists are conventional people yet open to trying new things, believe in following rules and don't consider themselves any smarter or less intelligent than the person next to them. Yahoo users tend to be 55-plus, reserved and a less-independent group with little faith in imagination. They feel they have little control over their future and are skeptical and cautious of new or untried ideas.

Fascinating. I've used Dogpile for nigh on 10 years, it searches all the other engines and aggregates their result for you. I love the "parasite search engine" design (I wonder what that says about me ). Interesting that it hasn't really taken off though.

Update - realised there is a bigger story here. In an age where we see Google moving to mainstream advertising at the Superbowl (while Pepsi eschews it), clearly which search engine you use is becoming as much a branding battle as for any other commodity product - ie there is little inherent technical differentiation left. Increasingly search engines will try and identify demographics they want for Advertising purposes, and set out their brands to recruit them. Next step is diversofoed brands for different segments - Google Lite, Bing Premier etc - and celebrity sponsorship. Oh joy.....

I wondered how long it would take someone to build a system that obfuscates your data from Google. We looked at how to do it a few years ago, but the best short term default seemed to use Dogpile, a "parasite" search engine, or similar - yes, they have your data but its a far smaller operation. But that was just search. . However, a smart hacker has come up with a way of doing it. Forbes:

Now an independent security researcher who goes by the name Moxie Marlinspike is making Web users a counter-offer: Take Google's ( GOOG - news - people ) giveaways and keep your privacy too.

On Tuesday, Marlinspike launched a service he calls Googlesharing, a plug-in for Firefox designed to give users access to Google's online offerings while cloaking their identity from the company's data collection tools. By hosting a proxy server with a collection of Google "identities," the privacy software, which can be accessed at Googlesharing.net, will allow users to temporarily route their traffic through another computer that masks their identity by mixing their online actions with those of other users.

"Each identity looks like a normal user, but everything is mixed up between identities so Google can't track any individual," says Marlinspike. That means users can exploit any of Google's offerings that don't require logins, such as search, maps or news, without allowing Google to assemble a profile of their activities that can be used for advertising targeting--or, as some users might fear, information that could be subpoenaed by government investigators.

You can get the Firefox addition here. Marlinspike is offering up his code so you can build your own, so well worth a look methinks. I predict a large increase in Google services that have to be signed in for. I wonder if the Open ID people may have looked at this, but they have a different gig right now - but maybe we have a bigger need for Private rather than Open ID.

You all know by now that Google is pulling out of China (actually, its "May pull out if non censorship request is not met", but that is very unlikely). The reason given is a massive attack on the accounts of Chinese dissidents in Google (and other) companies' servers. But, as was pointed out on the BBC Today program this morning, everyone told Google that would happen when they went in 4 years ago. The Google response this morning was that they hoped that by engaging, Chinese behaviour would change

Another reading - the RealEkonomik reading - is that Google knew this would happen, but that it could not afford to ignore such a huge emerging market so went in with eyes wide open, albeit possibly with rose tinted glasses. As Shefaly Yogendra points out, strategy in China is a new Great Game.

So what's changed? We suspect that as well as an increase in attacks on the servers (Why is China doing it only a problem now, though?*), three other factors have driven them:

- Their reputation in the far more valuable US and European Union market is diving, and both areas are looking at regulation to curb their activity.

- Nothing builds confidence in your new Cloud strategy like knowing its massively hackable by anyone with bit of will and nous

- Penetration in China has been disappointing for Google, its Chinese management has been turbulent, ARPU is still low but costs are not, so its not a great economic benefit at this point

Let's be clear - a publicly owned corporation does not do things on points of ideological principal, Google is doing this as it believes it serves its own best economic interests here and now. In other words, in the short term it would hopefully do Google a lot of PR good (and allay some legal activity) to give China short shrift. In the long term this may prove to be short sighted, but its a reversible decision and in the long term we are all dead - and the current management will long since have taken their pension pots into retirement. Besides, doors are still open....

All that said, it is still thrilling - regardless of the reasons why, they are doing the ethical thing.... Interesting Times are surely ahead.

* It will be interesting to see if other companies like Microsoft and Yahoo report increasing attacks. Also, surely this is a government to government issue if as widespread as claimed? (Update to this - a statement from Hilary Clinton)

Techmeme comes up with their top 10 stories of the year - and 7 out of 10 are Apple or Google stories. The other 3 are TechCrunch (with an Apple substory), Friendfeed (ex Googlers) and eBay/Skype. How was such a monochrome palette selected - sez TM:

Techmeme has distilled its very own Top 10 story list, utilizing headline ranking data archived throughout the year. While we employ a terrific team of news editors to guide our story selection, this list draws on many factors outside our direct influence, making it a … nearly objective sampling of what was important in 2009.

Hmmm... to an extent what you get out of it depends on what you put into it. I've noticed that TM is biassing more and more towards mainstream sources, so maybe that is an explanation.

But even so - seriously - no Twitter, Hulu etc? Or the AT&T/4Chan and PirateBay stories that hit top spots on Digg? If I look at the traffic on our blog, we covered all the above stories and they were not the biggest runners (see here). Be interesting to see what the top runners on TechCrunch, Mashable etc. were.

One wonders about the exact selection criteria! (Update - Techmeme's Gabe Rivera has commented below, explaining its based on link count. That makes more sense as I suspect Google and Apple have more fanboi 2.0's than say Microsoft, but also - in my humble opinion - shows the risks of pure link based ratings)

Now I really like TechMeme, and am a great fan of its model, I've always seen them as a leading example of how media may work in the future - but the risk they run here is becoming Yet Another Mainstream Tech News Organ..... so why?

Does becoming a me-too storyboard system create sufficient value from the (hypothesized) increased volumes? One also therefore starts to wonder more deeply about how they are funded, and the implications thereof!