Quantcast

Web Crawlers Articles

New “Bingbot” Will Crawl Non-optimized Sites More Easily
· 4

Microsoft has announced that it will be bringing the Bing web crawler out of beta on October 1st. It will be rebranded as "the Bingbot" and replace the existing msnbot. "It will still honor robots.txt directives written for msnbot, so no change is required to robots.txt file(s)," a Bing representative tells WebProNews.

"Improvements to the bot enable more efficient crawling, and increase the ability to crawl content on sites not optimized for search," he says.
 

Google States Case for Online News in WSJ
· 1

Update: The Wall Street Journal is running a piece from Google CEO Eric Schmidt on how Google can help newspapers. It’s an interesting read.

Yahoo Slurps Somewhere Else

The migration of the Slurp is complete, says Yahoo. Over the past few weeks, the search engine has been transitioning its crawler, dubbed (disgustingly) "Slurp," to a new address at crawl.yahoo.net. Adjust your server logs as necessary and join the curmudgeons who are unimpressed.

Webmaster Claims Spider Entered Contract In Suit
· 8

The Web and artificial intelligence have brought about some surreal, science fiction like questions. The most recent mind-bending concept is whether or not robots can enter into contracts – that is, is a Web crawler implicitly entering a contract posted on a website announcing copyright conditions?