« The GapingVoid on Business Models | Main | Attending Syndicate this Tuesday »

December 13, 2005


Greg Linden

It's a good point, but be careful of overstating it.

On the one hand, open source solutions like Nutch and now this Alexa service make it easier to get crawl and index data on the web.

On the other hand, comprehensiveness and timeliness are still issues. A partial, stale index of the web has a lot less value.

By the way, it is important to distinguish between the index and the relevance rank. Relevance rank is what gets the right information to the right people. Search is all about relevance, and relevance rank is where most of the value is.

Jeff Clavier

Greg> Good points, and I should have expanded on my comment regarding the distribution of the 5 billion pages indexed by Alexa re the freshness.

What is interesting here is that Alexa is making available a huge infrastructure as a service - whereas Nutch is a piece of software that you have to run. But you are right, we don't know what is the update rate of the crawler(s) and whether it is possible to "force" a refresh of a specific set of web sites.

And regarding relevance, this is where application built on top of the index will be able to introduce some level of differentiation. It might however not be optimized at all because of the implied layering of data/processing.


Jeff, if Alexa and others start providing their search engine infrastructure for everyone to create their own search engines in this way, I don't feel these newly upcoming search engines will become that famous even as alexa itself! Developing and offering a service is one thing and marketing and promoting and establishing the same in it;s target market is another. And while we have big giants such as google, yahoo, msn and others not giving up their billion dollar infrastructure and their yearly revenues, I don't think these upcoming hundreds of thousands of search engines will make any sense or get any popular. Is this just a source that will offer free content that one can harness along with contextual advertising options? Again the ball is in google's court!

Cheers! The earth is round!

The comments to this entry are closed.

On Plancast


On the Web

  • www.flickr.com
    This is a Flickr badge showing public photos from jeffclavier. Make your own badge here.