Tag: Googlebot

Google Says It Will Follow Five Redirects At The Same Time When Crawling
About a year ago, Google put out a Webmaster Help video d...
Read More
Trouble With Googlebot Fetching Robots.txt? Use Fetch As Googlebot Tool
Google has released a new Webmaster Help video in response to a question from a user who has been having trouble getting Google to fetch their robots.txt file. Here's what the user...
Read More
Cutts Does Customer Support In Latest Webmaster Video
Google has put out a new webmaster help video with Matt Cutts. The basis of the user-submitted question isn't even accurate, and Cutts still took the time to make the video and ans...
Read More
Is Googlebot Getting More Human-Like?
Google may be getting better at crawling javascript and Ajax. In a Tumblr post, developer Alex Pankratov...
Read More
Matt Cutts On The Hardware & Software That Power Googlebot
Google uploaded a new Webmaster Help video from Matt Cutts, which addresses a question about the hardware/server-side software that powers a typical Googlebot server....
Read More
Google Gets Better At Crawling Smartphone Content
Google announced some improvements for how it indexes smartphone content in mobile search. Googlebot-Mobile will now crawl with a smartphone user-agent in addition to its previous...
Read More
Making The Googlebot Fetch With URL Submission
Is there some new content and/or a new site you want Google to notice sooner rather than later? Well, there's an official Google utility for that. With the Fetch as Googlebot U...
Read More
Webmasters: Googlebot Caught in Spider Trap, Ignoring Robots.txt
Sometimes webmasters set up a spider trap or crawler trap to catch spambots or other crawlers that waste their bandwidth. If some webmasters are right, Googlebot (Google's crawler)...
Read More
Get the WebProNews newsletter delivered to your inbox

Get the free daily newsletter read by decision makers

Subscribe
Advertise with Us

Ready to get started?

Get our media kit

Advertise with Us