Quantcast

New “Bingbot” Will Crawl Non-optimized Sites More Easily

Bing's New Web crawler Bingbot Coming in October

Get the WebProNews Newsletter:
[ Search]

Microsoft has announced that it will be bringing the Bing web crawler out of beta on October 1st. It will be rebranded as "the Bingbot" and replace the existing msnbot. "It will still honor robots.txt directives written for msnbot, so no change is required to robots.txt file(s)," a Bing representative tells WebProNews.

"Improvements to the bot enable more efficient crawling, and increase the ability to crawl content on sites not optimized for search," he says.
 
Robot - This is not the real Bingbot, but it will be here in October.Rick DeJarnette has more about the change on the Bing Webmaster Blog:

Instead of the old msnbot 2.0b showing up in your server logs, the updated user agent will be:

Mozilla/5.0 (compatible; bingbot/2.0 +http://www.bing.com/bingbot.htm)

The HTTP header From field will also change as shown below:

From: msnbot(at)microsoft.com

will become

From: bingbot(at)microsoft.com

If Bing finds separate sets of directives for Bingbot and for other crawlers, directives for bingbot will take precedence, the company says.

I find the part about increasing the ability to crawl content on sites not optimized for search to be particularly interesting. I wouldn’t exactly call this an invitation to ignore SEO. Obviously Google is still the biggest search engine anyway, but even as far as Bing is concerned, good SEO practices will likely still help your rankings.

Also keep in mind that optimizing for Bing is becoming increasingly important. Not only is Facebook giving more reason for people to search (where Bing provides the web results), but the Yahoo/Bing integration will be here (likely) before the holidays.

New “Bingbot” Will Crawl Non-optimized Sites More Easily
Top Rated White Papers and Resources
  • http://www.snerdey.com Snerdey

    Is this how the internet is adapting to the burned out designers or hard to read websites? Instead of the web owners working hard at keeping sites up to date they are going to pull sites out of the pits and bring them to light.. hmm, wonder how that’s going to impact results for those who spend their weeks, months at following all the rules they can get their fingers on to make the top results?

    Will they also constantly change the results every few months or so like Google, is there a pattern here? I wonder.. why not bring non seo developed web site results up.. why? Heck, it’s all about money. Looks like the ones who work hard at organic results will now have to PPC to get noticed.

    My 2 Cents :)
    Snerdey

    Follow me on Twitter!
    http://www.twitter.com/snerdey

  • http://offshoreally.com/ offshoreally

    Yes Snerdey it’s all about money,still I think I prefer google rather than bing as they are best at what they do..

  • Douglas Gross

    I have to wonder if this means Bing will be more efficient, or will traditional SEO techniques remain in the game? Real SEO (not the commercial crap that some try to sell) is just good practice. Also, I have to wonder if that means we won’t be able to tell Bing not to index pages in frames or index pages that were not optimized because they aren’t intended to be indexed? There could be some loss of control here. It could be just another attempt by Microsoft to do something right that yields bad results, but I like Bing.

  • MikeGlenn

    ping my website http://www.addenda.us so many time, even its daily updated, new content there but no crawl