Quantcast
Read WebProNews
With Friends!
 
Trouble With Googlebot Fetching Robots.txt? Use Fetch As Googlebot Tool

Trouble With Googlebot Fetching Robots.txt? Use Fetch As Googlebot Tool

By Chris Crum November 26, 2012

Google has released a new Webmaster Help video in response to a question from a user who has been having trouble getting Google to fetch their robots.txt file. Here’s what the user said: “I’m getting errors from Google Webmaster Tools …

Making The Googlebot Fetch With URL Submission Making The Googlebot Fetch With URL Submission
· 1 Comment

Is there some new content and/or a new site you want Google to notice sooner rather than later? Well, there’s an official Google utility for that. With the Fetch as Googlebot URL submission tool, site owners can now directly request …

Webmasters: Googlebot Caught in Spider Trap, Ignoring Robots.txt Webmasters: Googlebot Caught in Spider Trap, Ignoring Robots.txt
· 4 Comments

Sometimes webmasters set up a spider trap or crawler trap to catch spambots or other crawlers that waste their bandwidth. If some webmasters are right, Googlebot (Google’s crawler) seems to be having some issues here. In the WebmasterWorld forum, member …

Tips for Getting Crawled Faster by Google
· 28 Comments

Probably the most important step in getting your site found in a search engine is the one in which the search engine crawls it. There are things that can be done and things that can be avoided to make this process as painless as possible for the search engine, which will in turn, make it as painless as possible for the webmaster.

Love Letters from Googlebot
· 3 Comments

In a mildly weird fashion, Google answered some questions regarding HTTP status codes and "if-modified-since" from users who were either made up or had their names changed to protect their anonymity. Names like Little Jimmy, Temp O’Rary, Janet Crinklenose, and Frankie O’Fore (my apologies to any Janet Crinklenoses who may be out there, but I think I’m safe in my assumption that the name is fictitious).

Google Provides Valuable Information On Webserving Techniques

At the Google Webmaster Central Blog, Google has released some valuable information about webserving techniques, especially related to Googlebot.

Googlebot Gets Candid

Googlebot is like a dream which knows us all , , and soul. Here in this interview, Maile Ohye as the website and Jeremy Lilley as the Googlebot from Google Central Webmaster blog would answer all those questions that you ever had.

Question & Answers

Googlebot: ACK

Noteworthy Videos
Some New Tags To Play With
· 1 Comment

HighRankings.com’s Jill Whalen broke some interesting news about tags webmasters can use to control how their content is shown in Google results. The information comes from Dan Crow, director of crawl systems at Google, speaking at the Search Engine Marketing New England event in Rhode Island.

Controlling How Your Site Is Indexed

Over at the Google Blog, Dan Crow is publishing a series of how-to’s about controlling what the Googlebot says about your site through the Robots Exclusion Protocol. This is the cheap-and-easy version.

Don’t Index That Link!

Get That Site Back Up Before It’s Delisted

If you’re site goes down for whatever reason, get it back up quickly or risk being de-listed from Google’s index.

Though it has been thought there may be somewhat of a grace period, it now seems robots only notice what’s there when they arrive.

GoogleBot Crawls CSS Files

Barry Schwartz spotted a thread in the Cre8site Forums discussing a report by Ekstreme.com that Googlebot requested a CSS file. This is a very interesting, but not surprising development.

Google Enhances Webmaster Tools
· 1

Vanessa Fox unveiled new features included with Google’s Webmaster Tools in an entry this week on the Official Google Webmaster Central Blog.

Googlebot Wields New Crawl Tools

Bringing your website into the voluminous BigTable of Google’s index involves sending out pieces of code, referred to as spiders or crawlers affectionately named “Googlebot,” to pick up your site. Google Webmaster Central has some tips for webmasters showing how they can learn more about Googlebot.

When Was the Last Time You Saw the Googlebot?

The Google crew cleared up a bit about how the Googlebot caches webpages. The date at the cached page reflects the last time the page was modified.

Quoth The Googlebot, “304!”

The latest news from Google Webmaster Central disclosed how the search engine’s spider saves a webmaster from unnecessary bandwidth costs.

Matt Cutts Reins In Googlebot

Ahead of Search Engine Strategies 2006 conference in San Jose, Google’s best-known engineer suggested some basic approaches to making a visiting Googlebot behave when it drops by your website.

Google Sitemaps Wants Your URLs
· 2

Vanessa Fox, a technical writer for Google’s Sitemaps product, thinks you should be using Sitemaps with your site today. Here’s why.