TCS Daily

Yahoo! Takes on Google

By Jay Currie - March 29, 2004 12:00 AM

Yahoo! has announced it is dropping Google as its search engine and is going to crawl the web to provide its own search results. This is being cast as a competition and, of course, it is. It is also a crucial step in making the internet even more useful. If Yahoo! is ambitious, this may be the moment when the real power of the net is multiplied.

Google has become many net users' default search engine. How Google and its webcrawling robots see the net is how millions upon millions of people also see it. Entire websites and discussion groups buzz every time Google tweaks its search algorithm. The more or less monthly Google dance, when website rankings twitch and change, causes consternation, paranoia and despair as people see their website, their information, ranked lower than their competitors'.

Dark theories abound: Google likes commercial sites more than non-commercial, put AdSense on your site and Google will spider you faster and boost your site up its rankings, Google loves blogs. Successful companies exist for no other purpose than to "optimize" websites for Google.

It is difficult to overestimate Google's perceived importance to everyone from porn merchants to plastics manufactures because if a site is not on the first page of the Google rankings it is missing traffic, a lot of traffic. And traffic equals sales.

At the non-commercial level, Google has structured the internet for millions of users. Through a series of partnerships with AOL, Yahoo! and other search engines, Google results have become the de facto internet standard.

None of which is a bad thing. After all, one of the biggest issues of the early internet was simply trying to find reliable information about what was actually on the net. Google went a long way to solving that problem.

If Google has a single fault it was that it was too authoritative. Sites which Google ranked low or missed altogether effectively vanished from the net. Indeed there were occasional calls for Google to be somehow regulated. Here's one from Businessweek,

"'It's the kind of question the Internet is putting to us again and again,' says Jonathan Zittrain, co-director of the Berman Center for Internet & Society at Harvard Law School - 'the Net feels awfully public, but in fact almost every element of it isn't public or government-run or even subsidized. When these private parties start making individualistic decisions about who's favored and who's not, it does create serious legal questions.' Zittrain argues that some regulation of Web searching might prove necessary should Google kill all its competition and become the only viable search provider."

Google looked like an emerging information monopoly and while there was no clear evidence that it was abusing this position, the possibility existed. Worse, as Google headed towards its much anticipated IPO, concern grew its commercial interests might override its original business model of simply being the best search engine on the internet.

Enter Yahoo!. For years Yahoo! has run an internet directory operation -- a list of sites rather than a true search engine -- and supplemented its results through a partnership with Google. It has also been buying up a number of second tier search engine outfits -- Inktomi, Fast Search & Transfer and AltaVista, which were owned by Overture which was also bought by Yahoo!. On February 18, Yahoo! announced it was cutting its ties to Google and that it would be crawling the internet on its own.

Half of all web searches world wide originate from Yahoo!. It claims 260 million users worldwide, with 100 million of those users registered. By withdrawing from the Google partnership, Yahoo! is removing a significant fraction of Google's search traffic.

Yahoo! is promising to avoid a "me too" approach to crawling the net. Rather than simply trying to replicate the Google search algorithm, Yahoo! is looking to craft a search engine which delivers a better user experience. How?

Watch those registered users. Yahoo! is promising to integrate the information it has gained from the Yahoo! Portal on its members' surfing habits into its search engine results. What this is likely to do in the medium term is enable Yahoo! to calibrate its results to the user experience of particular websites. In practice, this might be as simple as aggregating the average time Yahoo! users spend at site "A" versus site "B" and then using that average time as a variable in the Yahoo! ranking algo.

At a technical level the Google and Yahoo! crawls of the internet will likely start out with fairly similar results. You can see how similar by going to Yahoo-vs-google or look at side by side results at After all there are only so many ways to determine the content and relative importance of a website. However, those results will tend to diverge over time and that divergence will come to represent two very different versions of the vast array of information the net can provide.

Even subtle differences in the weight the two search algorithms assign to such things as inbound links, metatags, keyword densities, size, search engine optimization strategies (aka "spam"), internal links and the value of a given inbound link will mean that the results of the two engines will begin to diverge. Adding additional variables may also cause sharp differences to appear. Across time, divergence will be self-reinforcing. If Yahoo! ranks a site more highly than Google, links from that site will have a greater value in the Yahooverse and the divergence will cascade through the Yahoo! results.

Fairly quickly, the Google map of the internet and the Yahoo! map of the internet will become maps of different internets. Many of the major landmarks are likely to be the same, however, the detail may very well emerge in strikingly different contours.

Other entrants, including Microsoft, are preparing to offer their own search algorithms, their own maps. And with each new map, the Internet will be re-invented.

Reshuffling information changes how we access and understand that information. It alters the perceived connections and importance of particular information. It creates the opportunity to check the results of one search against the results of another. Having two full bore, modern search engines means that fewer websites and webpages will be missed.

Best of all, real competition between Yahoo! and Google will drive both companies to consistently work at improving their search techniques, presentation and business models. Real price competition for items like pay per click keyword advertising and paid search engine inclusion can't help but benefit e-commerce. Most importantly, two full scale search engines will ensure new sites have a fighting chance to be found by search engine users.

Jay Currie last wrote for TCS about Virtual Israelis. His online home is at


TCS Daily Archives