All Posts Tagged Tag: ‘Crawling’

How To Use Google’s New Blocked Resources Report

Google just introduced a new Webmaster Tools feature called the Blocked Resource Report, aimed at helping webmasters find and resolve issues where Google can’t use images, CSS, or JavaScript that has been blocked. Blocked resource prevent pages from rendering properly, and Google wants to make sure you’re only blocking what you really want/need to be. The report provides the names …

Read More

Google Adds Crawling, Indexing Of Locale-Adaptive Pages

Google announced that it’s introducing new locale-aware crawl configurations for Googlebot for pages it detects may adapt their content based on the request’s language and perceived location. “Locale-adaptive pages change their content to reflect the user’s language or perceived geographic location,” Google says in a blog post. Since, by default, Googlebot requests pages without setting an Accept-Language HTTP request header …

Read More

Google Says It Will Follow Five Redirects At The Same Time When Crawling

About a year ago, Google put out a Webmaster Help video discussing PageRank as it relates to 301 redirects. Specifically, someone asked, “Roughly what percentage of PageRank is lost through a 301 redirect?” Google’s Matt Cutts responded, noting that it can change over time, but that it had been “roughly the same” for quite a while. “The amount of PageRank …

Read More

Google Continues To Improve Crawling Capabilities For Smartphone Content

Throughout 2013 and into 2014, Google has been making various improvements to the way it handles website content on mobile devices. That continues this week with the announcement of a new user-agent for crawling smartphone content. Google says it’s retiring “Googlebot-Mobile,” for smartphones, which has been used to refer to various mobile-specific crawlers that index content for both feature phones …

Read More

Bing Details How To Control BingBot Crawling Behavior

BingBot is the cute name that Microsoft gives the bot that crawls the Web indexing the content is returned to users who use its search function. BingBot might start to index unwarranted content or it might index enough content sometimes. Microsoft has some tips to make your Web site BingBot friendly so that the right amount of content is always …

Read More

Google Details How They Crawl Images, Offers Tips

It’s pretty clear now how Google crawls Web sites to get the results you search for. Heck, we can’t go a week without Google updating us on how they’re changing the algorithm that determines search results. Unfortunately, there hasn’t been much on how Google crawls for images – until now. In a lengthy post on the Webmaster Central Blog, Google …

Read More