All Posts Tagged Tag: ‘Googlebot’

How To Use Google’s New Blocked Resources Report

Google just introduced a new Webmaster Tools feature called the Blocked Resource Report, aimed at helping webmasters find and resolve issues where Google can’t use images, CSS, or JavaScript that has been blocked. Blocked resource prevent pages from rendering properly, and Google wants to make sure you’re only blocking what you really want/need to be. The report provides the names …

Read More

Google Adds Crawling, Indexing Of Locale-Adaptive Pages

Google announced that it’s introducing new locale-aware crawl configurations for Googlebot for pages it detects may adapt their content based on the request’s language and perceived location. “Locale-adaptive pages change their content to reflect the user’s language or perceived geographic location,” Google says in a blog post. Since, by default, Googlebot requests pages without setting an Accept-Language HTTP request header …

Read More

How Googlebot Treats Multiple Breadcrumbs On E-Commerce Pages

Google has a new “Webmaster Help” video out about e-commerce pages with multiple breadcrumb trails. This is the second video in a row to deal specifically with e-commerce sites. Last time, Matt Cutts discussed product pages for products that are no longer available. This time, he takes on the following question: Many of my items belong to multiple categories on …

Read More

Google Says It Will Follow Five Redirects At The Same Time When Crawling

About a year ago, Google put out a Webmaster Help video discussing PageRank as it relates to 301 redirects. Specifically, someone asked, “Roughly what percentage of PageRank is lost through a 301 redirect?” Google’s Matt Cutts responded, noting that it can change over time, but that it had been “roughly the same” for quite a while. “The amount of PageRank …

Read More

Trouble With Googlebot Fetching Robots.txt? Use Fetch As Googlebot Tool

Google has released a new Webmaster Help video in response to a question from a user who has been having trouble getting Google to fetch their robots.txt file. Here’s what the user said: “I’m getting errors from Google Webmaster Tools about the Googlebot crawler being unable to fetch my robots.txt 50% of the time (but I can fetch it with …

Read More

Cutts Does Customer Support In Latest Webmaster Video

Google has put out a new webmaster help video with Matt Cutts. The basis of the user-submitted question isn’t even accurate, and Cutts still took the time to make the video and answer it. This goes to show that there is a solid chance that Google will answer your questions when you send them. The question was: The Webmaster Tools …

Read More

Is Googlebot Getting More Human-Like?

Google may be getting better at crawling javascript and Ajax. In a Tumblr post, developer Alex Pankratov wrote this week about spotting an “ajax request issued from document.ready() callback of one website’s pages.” “This means that the bot now executes the Javascript on the pages it crawls,” Pankratov wrote. “The IP of 66.249.67.106 is crawl-66-249-67-106.googlebot.com and the A record is …

Read More

Matt Cutts On The Hardware & Software That Power Googlebot

Google uploaded a new Webmaster Help video from Matt Cutts, which addresses a question about the hardware/server-side software that powers a typical Googlebot server. “So one of the secrets of Google is that rather than employing these mainframe machines, this heavy iron, big iron kind of stuff, if you were to go into a Google data center and look at …

Read More