All Posts Tagged Tag: ‘Web Crawlers’
Microsoft has announced that it will be bringing the Bing web crawler out of beta on October 1st. It will be rebranded as "the Bingbot" and replace the existing msnbot. "It will still honor robots.txt directives written for msnbot, so no change is required to robots.txt file(s)," a Bing representative tells WebProNews.
"Improvements to the bot enable more efficient crawling, and increase the ability to crawl content on sites not optimized for search," he says.