Status Quo

Darwin's theory of evolution is based on five key observations and inferences drawn from them. The first is summarized thus: “Species have great fertility. They make more offspring than can grow to adulthood.” (Wikipedia) And so it is with the virtual speciation that has taken place in the online variation of search spiders, bots, and crawlers. A handful of the litter that burst forth from the fertile womb of search necessity in the mid to late 1990’s have survived and fewer have thrived. Notable among the victor’s, and ranked here by search market share are Google, Yahoo Search, MSN Search, and Ask (formerly AskJeeves). Some of the victims not with us today died of starvation while others became food for the emerging.

Late on the scene was Google, who’s informal corporate motto is “Don’t Be Evil”, took up the noble task of, in Google’s words, “organizing the world’s information”. It’s the rare online citizen who does not recognize the Google name today. And while Google critics are perhaps justified in arguing that Google needs to get back to its “Don’t Be Evil” roots, seemingly abandoned for the almighty dollar soon after becoming a publicly traded IPO, none can argue that it has been Google raising the evolutionary bar. Google was the first to develop algorithms to rank a website’s importance based on the number and quality of inbound hyperlinks. And while motivated by advertising dollars, Google is the first to implement on a Web wide scale the algorithmic analysis of a websites content to determine the semantic topic spaces to which it naturally belongs. In spite of the non-altruistic, financial motivation, no, because of the competitive market conditions in which Google is competing to survive, Google has evolved search technology to the point where search relevancy is more accurate than ever. Search engines are no longer just associatively accurate based on database keyword matching. They now remember where things reside on the Web and through instant association, take you to the most semantically matched sites with the greatest value as determined by democratic peer support (quality inbound links).

It is right to give Google thanks and praise, but it is not the only player. Both MSN Search and Yahoo! Search are forcing Google to constantly update their associative and semantic search algorithms. And Ask has risen like a phoenix from the ashes of an engine that started with great promise, but collapsed under the weight of its advertising components that destroyed user confidence as it became apparent that relevance with AskJeeves was directly proportional to the amount of money that advertisers were willing to spend. Ask has reinvented itself to the acclaim of Search critics whose majority claim that Ask already returns more relevant results than any of the big three. These challenges to Google’s authority are forcing Google to play its Latent Semantic Indexing (LSI) hand prematurely. Already a part of the Google algorithmic Shepard’s Pie, LSI is being given heavier weighting in relevancy returns in an attempt to stay on top.

In an interesting side note, a parallel between what is happening today in Search and what happened 20 years ago in national television broadcast networks can be drawn. Barry Diller is the man credited with turning a small upstart network called Fox into a national force to be reckoned with, bringing the big networks from three to four. Barry Diller is the man behind the reinvention of Ask, and early predictors indicate that history will repeat itself.

Back to the subject of current Search survivors, apart from the big three (soon to be four) at the top of the food chain, other engines that carry on are niche specific or topical engines and meta engines which aggregate the results from multiple engines, returning the best averages with the premise that skimming the cream from the top of many sources will produce the most balanced results. These niche and meta engines possess only a small percentage of the search market, but will likely continue to be useful contributors until one or another of the food chain carnivores develop an appetite for these smaller producers.

Such is the status quo as the smartest and strongest search engines roam the vast fertile savannahs of the Web, sometimes harvesting and consuming from disorderly but fruitful fields of information in countless Web sites, and sometimes hunting down to consume and incorporate the essence of smaller engines which will add to the strength and skill of the dominant engines.