Quantcast

Crawling Articles

Google Says It Will Follow Five Redirects At The Same Time When Crawling Google Says It Will Follow Five Redirects At The Same Time When Crawling

About a year ago, Google put out a Webmaster Help video discussing PageRank as it relates to 301 redirects. Specifically, someone asked, “Roughly what percentage of PageRank is lost through a 301 redirect?” Google’s Matt Cutts responded, noting that it …

Google Continues To Improve Crawling Capabilities For Smartphone Content Google Continues To Improve Crawling Capabilities For Smartphone Content

Throughout 2013 and into 2014, Google has been making various improvements to the way it handles website content on mobile devices. That continues this week with the announcement of a new user-agent for crawling smartphone content. Google says it’s retiring …

Bing Details How To Control BingBot Crawling Behavior Bing Details How To Control BingBot Crawling Behavior
· 4

BingBot is the cute name that Microsoft gives the bot that crawls the Web indexing the content is returned to users who use its search function. BingBot might start to index unwarranted content or it might index enough content sometimes. …

Google Details How They Crawl Images, Offers Tips Google Details How They Crawl Images, Offers Tips
· 1

It’s pretty clear now how Google crawls Web sites to get the results you search for. Heck, we can’t go a week without Google updating us on how they’re changing the algorithm that determines search results. Unfortunately, there hasn’t been …

Google Rolls Out Breadcrumb Display in SERPs
· 66

Update: Google announced today that it will now be rolling out the use of breadcrumbs in seach results on a global basis. They will only be used in place of some URLs – mainly the ones that don’t give the added context of a link the way that the breadcrumbs do. Google says:

Google to Start Crawling Google Docs Documents
· 3

Google has quietly announced that Google Docs documents that are published will soon be crawlable. This means if you have published documents as web pages, or used the publish/embed option for a document, and it has been linked to on the web, it can be indexed by Google and other search engines.

Tips for Getting Crawled Faster by Google
· 28

Probably the most important step in getting your site found in a search engine is the one in which the search engine crawls it. There are things that can be done and things that can be avoided to make this process as painless as possible for the search engine, which will in turn, make it as painless as possible for the webmaster.

Google Improves Flash Indexing Capabilities
· 14

Last year, Google began crawling and indexing Flash content, but now Google has announced that it can also index external resource loading. In other words, Google can index external content that loads within an SWF file, and associate it with that file, so that it will appear in search results.

For example, a site that loads something like this in Flash:

Transporter Flash file

..might appear in a Google SERP like this:

Google Counts First Link, Not Second, Says SEO
· 10

Got a couple of links on one page to another page? Google only has love for the first one, no matter what you do with it.

Much ado about Delicious robots

Bots arriving from Google, Ask.com, and MSN to sample pages on the bookmarking site Delicious hit a robots.txt block. As do Yahoo’s Slurp bots too. It’s no big deal.