C. Lee Giles, the David Reese Professor of Information Sciences and Technology who led the research team which developed a new search engine—BotSeer—for the study, found the result "surprising".
Giles said, "We expected that robots.txt files would treat all search engines equally or maybe disfavor certain obnoxious bots, so we were surprised to discover a strong correlation between the robots favored and the search engines’ market share".
Some of the results are:
More than any other search engine, Google's is most favored and called most often.

Take a look at the chart below and guess who the prodigal one is.

Apparently, the study's title is, "Determining Bias to Search Engines from Robots.txt (PDF)," Barry Schwartz has a local copy for easy download.
Comments
RSS
Home
Newsletter
Advertising

















