All Posts Tagged Tag: ‘Web Crawlers’
Microsoft has announced that it will be bringing the Bing web crawler out of beta on October 1st. It will be rebranded as "the Bingbot" and replace the existing msnbot. "It will still honor robots.txt directives written for msnbot, so no change is required to robots.txt file(s)," a Bing representative tells WebProNews.
"Improvements to the bot enable more efficient crawling, and increase the ability to crawl content on sites not optimized for search," he says.
The migration of the Slurp is complete, says Yahoo. Over the past few weeks, the search engine has been transitioning its crawler, dubbed (disgustingly) "Slurp," to a new address at crawl.yahoo.net. Adjust your server logs as necessary and join the curmudgeons who are unimpressed.
The Web and artificial intelligence have brought about some surreal, science fiction like questions. The most recent mind-bending concept is whether or not robots can enter into contracts – that is, is a Web crawler implicitly entering a contract posted on a website announcing copyright conditions?