All Posts Tagged Tag: ‘Web Crawlers’

New “Bingbot” Will Crawl Non-optimized Sites More Easily

Microsoft has announced that it will be bringing the Bing web crawler out of beta on October 1st. It will be rebranded as "the Bingbot" and replace the existing msnbot. "It will still honor robots.txt directives written for msnbot, so no change is required to robots.txt file(s)," a Bing representative tells WebProNews.

"Improvements to the bot enable more efficient crawling, and increase the ability to crawl content on sites not optimized for search," he says.

Read More