"Improvements to the bot enable more efficient crawling, and increase the ability to crawl content on sites not optimized for search," he says.
 
Rick DeJarnette has more about the change on the Bing Webmaster Blog:

Instead of the old msnbot 2.0b showing up in your server logs, the updated user agent will be:

Mozilla/5.0 (compatible; bingbot/2.0 +http://www.bing.com/bingbot.htm)

The HTTP header From field will also change as shown below:

From: msnbot(at)microsoft.com

will become

From: bingbot(at)microsoft.com

If Bing finds separate sets of directives for Bingbot and for other crawlers, directives for bingbot will take precedence, the company says.

I find the part about increasing the ability to crawl content on sites not optimized for search to be particularly interesting. I wouldn’t exactly call this an invitation to ignore SEO. Obviously Google is still the biggest search engine anyway, but even as far as Bing is concerned, good SEO practices will likely still help your rankings.

Also keep in mind that optimizing for Bing is becoming increasingly important. Not only is Facebook giving more reason for people to search (where Bing provides the web results), but the Yahoo/Bing integration will be here (likely) before the holidays.

Original Article: Google has created a new web crawler specifically for Google News. What this means is that publishers who do not want Google News to index their content can more easily control that. That also applies to publishers who don’t wish to completely cut out indexing, but wish to limit/restrict certain elements of their content from being indexed.

Google offers this new crawler at a time when Google’s relationship with online news is a heavy focus of discussion throughout the industry, with the FTC’s meeting of the media minds taking place. This week Google already announced some changes to how it handles paid content (by offering a five-article limit for the "first click free" plan). Now the company appears to be further extending its olive branch to concerned publishers (whether or not that will be enough is another discussion).

In the past, publishers have been able to block Google from content via robots.txt and the Robots Extension Protocol (REP). They have also been able to keep content out of Google News and stay in Google Search, by using a contact form provided by Google. Now, Google is making it so publishers don’t even have to contact them.

"Now, with the news-specific crawler, if a publisher wants to opt out of Google News, they don’t even have to contact us – they can put instructions just for user-agent Googlebot-News in the same robots.txt file they have today," says Google News Senior Business Product Manager Josh Cohen. "In addition, once this change is fully in place, it will allow publishers to do more than just allow/disallow access to Google News. They’ll also be able to apply the full range of REP directives just to Google News. Want to block images from Google News, but not from Web Search? Go ahead. Want to include snippets in Google News, but not in Web Search? Feel free. All this will soon be possible with the same standard protocol that is REP."

"While this means even more control for publishers, the effect of opting out of News is the same as it’s always been," says Cohen. "It means that content won’t be in Google News or in the parts of Google that are powered by the News index. For example, if a publisher opts out of Google News, but stays in Web Search, their content will still show up as natural web search results, but they won’t appear in the block of news results that sometimes shows up in Web Search, called Universal search, since those come from the Google News index."

Cohen says Google News users shouldn’t notice any difference in their experience with the service. It will be interesting to see the reaction from disgruntled publishers, and whether or not this will make any significant difference in how they view Google News.

Related Articles:

> Google Changes How it Handles Paid Content

> Minds of the Media Gather to Discuss Future of News

> Google Okay With Blocking News Corp.

> Is it Really Crazy to Block Google?

The Yahoo Search Blog reads:

…all machines crawling as Slurp are now in crawl.yahoo.net. You can see this change in your web server logs, where the page accesses from inktomisearch.com are being fully replaced by crawl.yahoo.net contacts. Note that this does not cover other Yahoo! crawlers, such Yahoo! China, and other verticals, like Yahoo! Shopping, Yahoo! Travel, etc., which have their own user-agent.

Don’t fret though; there is no need to change your robots.txt file because the crawler user-agent is still Yahoo! Slurp. If you use IP based filtering, there is no need to change that either, since the IP addresses from which we crawl remain the same. However, please ensure that your network or firewall setup does not keep crawl.yahoo.net out as we won’t be able to include your content in our results.

Be sure to click that link to get more enumerated information.

Over at WebmasterWorld, the crowd is a bit mixed about it (but only a bit), the loudest complaint, from "IncrediBill," who notes not only is it the move a year-and-a-half too late, but has gone overboard.

Why do we need to allow an army of Yahoo spiders to redundantly abuse our servers?

Is it a conceptual problem that Yahoo can’t share pages already downloaded?

When I posed that question to one of their engineers I was given a lame excuse that the various crawlers had different needs….

Funny, Google managed to make some of their crawlers share CACHE, so we know it can be done.

Negativity often rings louder and truer than other things, but there is at least one voice in that forum who thinks Yahoo’s update is "a small evolutionary improvement above" Google.

Even if in our hearts, we know that’s not true.
 

A little while back, we explored the idea that RSS, as an automatic distribution agent, could imply permission to republish. But that involves two human parties, essentially, with a technical agent in between.

A court battle in Colorado, however, focuses on claims brought by Suzanne Shell against the Internet Archive’s Wayback Machine, which holds in searchable perpetuity pages that appear on the Web, for future historical reference.

Shell owns the website www.profane-justice.org, devoted to providing information and support for people who feel they’ve been unlawfully targeted by state agents (like police or child services organizations) or unfairly accused of child abuse.

A notice appears on the site stating that users copying or distributing the content on the site automatically agree to the terms of a contract. Failure to abide carries a fee of $5,000 per page copied; $250,000 per occurrence of unauthorized use, and a charge of $50,000 for each occurrence of failure to pay, plus costs and triple damages.

There was no mechanism in place on the site (such as in the robots.txt file) to prevent Internet Archive’s robot from scanning, copying, and storing the pages on Shell’s website. She discovered that the Wayback Machine had reproduced the contents of her website about 87 times in five years, "and displayed her entire website to the public daily during that period." 

Shell sued the company for conversion, civil theft, breach of contract, and violations of the Racketeering Influence and Corrupt Organizations Act (RICO) and the Colorado Organized Crime Control Act (COCCA).

As might be guessed, most of these claims were dismissed. But the breach of contract claim is still under consideration, awaiting more information. The question that will be decided, ultimately, is whether a web crawler that is not blocked by a website can legally be bound by a contract posted on the site.

The outcome of that question could also have important impact on Web-crawling and Internet copyright law itself. Search engines like Google have leaned on Fair Use principles when scouring the Web (and off-line libraries) for information. The Google Book Search project defenders have claimed that publishers can "opt out" of their network by letting Google know their desire to do so.

Of a similar vein, then, is Web crawling. Webmasters must opt-out of indexing via the robots.txt file, preventing the spider from crawling the site. If the courts continue to back search engines’ and other Internet companies’ right to copy and index at will, then the whole copyright system, by default, it would seem, goes opt-out, too.

Copyright law blogger John Ottaviani, on Eric Goldman‘s blog, goes into the Internet Archive v. Shell case and its implications in greater detail.