Directions in New Search

TechCrunch on the gradual failure of Old Search:

As the Web swells with more and more data, the predominant way of sifting through all of that data—keyword search—will one day break down in its ability to deliver the exact information we want at our fingertips. In fact, some argue that keyword search is already delivering diminishing returns

TechCrunch also quotes Nova Spivack, Radar Networks CEO who believes semantic search is the answer.....:

Keyword search engines return haystacks, but what we really are looking for are the needles . The problem with keyword search such as Google’s approach is that only highly cited pages make it into the top results. You get a huge pile of results, but the page you want—the “needle” you are looking for—may not be highly cited by other pages and so it does not appear on the first page. This is because keyword search engines don’t understand your question, they just find pages that match the words in your question.

...and then asks:

So how do we get beyond keyword search and Google’s PageRank? There are many approaches being tried: social search, tagging, guided search, natural-language search, statistical methods, open search, semantic search, and (way out there) artificial intelligence. They all have their problems.

How Indeed ?

The issue increasingly is not finding - as anyone who is using aggregators like FriendFeed will know, you wind up with a firehose - the issue is filtering to get the needles you want.

We too are building new search systems, we believe we have learned quite a lot since building a specific type of search system to a BBC requirement last year.

This is going to be a fascinating space over the next few years.