Google announced today that it will no longer be using a separate crawler for Google News, and will now start using Googlebot.
“Google News recently updated our infrastructure to crawl with Google’s primary user-agent,Googlebot. What does this mean? Very little to most publishers,” says Google News Product Specialist David Smydra. “Any news organizations that wish to opt out of Google News can continue to do so: Google News will still respect the robots.txt entry for Googlebot-News, our former user-agent, if it is more restrictive than the robots.txt entry for Googlebot.”
“Although you’ll now only see the Googlebot user-agent in your site’s logs, no need to worry: the appearance of Googlebot instead of Googlebot-News is independent of our inclusion policies,” says Smydra. “You can always check whether your site is included in Google News by searching with the “site:” operator. For instance, enter “site:yournewssite.com” in the search field for Google News, and if you see results then we are currently indexing your news site.”
As far as analytics, you’ll still be able to differentiate traffic from Google Search and traffic from Google News, Google says.
Sites using Google’s metered subscription model or the first click free model won’t have to make any changes, but sites that require registration, payment or login before reading the full article, Google News will only be able to crawl and index the title and snippet that’s shown on the page.
Google stresses that the change will not affect how it crawls your News sitemaps.
More info in the help center.
