All Posts Tagged Tag: ‘Crawling’

How To Use Google’s New Blocked Resources Report

Google just introduced a new Webmaster Tools feature called the Blocked Resource Report, aimed at helping webmasters find and resolve issues where Google can’t use images, CSS, or JavaScript that has been blocked. Blocked resource prevent pages from rendering properly, and Google wants to make sure you’re only blocking what you really want/need to be. The report provides the names …

Read More

Google Adds Crawling, Indexing Of Locale-Adaptive Pages

Google announced that it’s introducing new locale-aware crawl configurations for Googlebot for pages it detects may adapt their content based on the request’s language and perceived location. “Locale-adaptive pages change their content to reflect the user’s language or perceived geographic location,” Google says in a blog post. Since, by default, Googlebot requests pages without setting an Accept-Language HTTP request header …

Read More

Google Says It Will Follow Five Redirects At The Same Time When Crawling

About a year ago, Google put out a Webmaster Help video discussing PageRank as it relates to 301 redirects. Specifically, someone asked, “Roughly what percentage of PageRank is lost through a 301 redirect?” Google’s Matt Cutts responded, noting that it can change over time, but that it had been “roughly the same” for quite a while. “The amount of PageRank …

Read More

Google Continues To Improve Crawling Capabilities For Smartphone Content

Throughout 2013 and into 2014, Google has been making various improvements to the way it handles website content on mobile devices. That continues this week with the announcement of a new user-agent for crawling smartphone content. Google says it’s retiring “Googlebot-Mobile,” for smartphones, which has been used to refer to various mobile-specific crawlers that index content for both feature phones …

Read More

Bing Details How To Control BingBot Crawling Behavior

BingBot is the cute name that Microsoft gives the bot that crawls the Web indexing the content is returned to users who use its search function. BingBot might start to index unwarranted content or it might index enough content sometimes. Microsoft has some tips to make your Web site BingBot friendly so that the right amount of content is always …

Read More

Google Details How They Crawl Images, Offers Tips

It’s pretty clear now how Google crawls Web sites to get the results you search for. Heck, we can’t go a week without Google updating us on how they’re changing the algorithm that determines search results. Unfortunately, there hasn’t been much on how Google crawls for images – until now. In a lengthy post on the Webmaster Central Blog, Google …

Read More

Google Rolls Out Breadcrumb Display in SERPs

Update: Google announced today that it will now be rolling out the use of breadcrumbs in seach results on a global basis. They will only be used in place of some URLs – mainly the ones that don’t give the added context of a link the way that the breadcrumbs do. Google says:

Read More

Google to Start Crawling Google Docs Documents

Google has quietly announced that Google Docs documents that are published will soon be crawlable. This means if you have published documents as web pages, or used the publish/embed option for a document, and it has been linked to on the web, it can be indexed by Google and other search engines.

Read More

Tips for Getting Crawled Faster by Google

Probably the most important step in getting your site found in a search engine is the one in which the search engine crawls it. There are things that can be done and things that can be avoided to make this process as painless as possible for the search engine, which will in turn, make it as painless as possible for the webmaster.

Read More

Google Improves Flash Indexing Capabilities

Last year, Google began crawling and indexing Flash content, but now Google has announced that it can also index external resource loading. In other words, Google can index external content that loads within an SWF file, and associate it with that file, so that it will appear in search results.

For example, a site that loads something like this in Flash:

Transporter Flash file

..might appear in a Google SERP like this:

Read More

Google Counts First Link, Not Second, Says SEO

Got a couple of links on one page to another page? Google only has love for the first one, no matter what you do with it.

Read More

Much ado about Delicious robots

Bots arriving from Google,, and MSN to sample pages on the bookmarking site Delicious hit a robots.txt block. As do Yahoo’s Slurp bots too. It’s no big deal.

Read More