Google announced some improvements for how it indexes smartphone content in mobile search. Googlebot-Mobile will now crawl with a smartphone user-agent in addition to its previous feature phone user-agents. This, the company says, will allow it to increase its coverage …

Is there some new content and/or a new site you want Google to notice sooner rather than later? Well, there’s an official Google utility for that. With the Fetch as Googlebot URL submission tool, site owners can now directly request …

Sometimes webmasters set up a spider trap or crawler trap to catch spambots or other crawlers that waste their bandwidth. If some webmasters are right, Googlebot (Google’s crawler) seems to be having some issues here. In the WebmasterWorld forum, member …

Probably the most important step in getting your site found in a search engine is the one in which the search engine crawls it. There are things that can be done and things that can be avoided to make this process as painless as possible for the search engine, which will in turn, make it as painless as possible for the webmaster.

In a mildly weird fashion, Google answered some questions regarding HTTP status codes and "if-modified-since" from users who were either made up or had their names changed to protect their anonymity. Names like Little Jimmy, Temp O’Rary, Janet Crinklenose, and Frankie O’Fore (my apologies to any Janet Crinklenoses who may be out there, but I think I’m safe in my assumption that the name is fictitious).

At the Google Webmaster Central Blog, Google has released some valuable information about webserving techniques, especially related to Googlebot.

Googlebot is like a dream which knows us all , , and soul. Here in this interview, Maile Ohye as the website and Jeremy Lilley as the Googlebot from Google Central Webmaster blog would answer all those questions that you ever had.

Question & Answers

Googlebot: ACK

HighRankings.com’s Jill Whalen broke some interesting news about tags webmasters can use to control how their content is shown in Google results. The information comes from Dan Crow, director of crawl systems at Google, speaking at the Search Engine Marketing New England event in Rhode Island.

Over at the Google Blog, Dan Crow is publishing a series of how-to’s about controlling what the Googlebot says about your site through the Robots Exclusion Protocol. This is the cheap-and-easy version.

Don’t Index That Link!

If you’re site goes down for whatever reason, get it back up quickly or risk being de-listed from Google’s index.

Though it has been thought there may be somewhat of a grace period, it now seems robots only notice what’s there when they arrive.

Barry Schwartz spotted a thread in the Cre8site Forums discussing a report by Ekstreme.com that Googlebot requested a CSS file. This is a very interesting, but not surprising development.

Vanessa Fox unveiled new features included with Google’s Webmaster Tools in an entry this week on the Official Google Webmaster Central Blog.

Bringing your website into the voluminous BigTable of Google’s index involves sending out pieces of code, referred to as spiders or crawlers affectionately named “Googlebot,” to pick up your site. Google Webmaster Central has some tips for webmasters showing how they can learn more about Googlebot.

The Google crew cleared up a bit about how the Googlebot caches webpages. The date at the cached page reflects the last time the page was modified.

The latest news from Google Webmaster Central disclosed how the search engine’s spider saves a webmaster from unnecessary bandwidth costs.

Ahead of Search Engine Strategies 2006 conference in San Jose, Google’s best-known engineer suggested some basic approaches to making a visiting Googlebot behave when it drops by your website.

Vanessa Fox, a technical writer for Google’s Sitemaps product, thinks you should be using Sitemaps with your site today. Here’s why.

A funny story is circulating in tech circles about how Googlebot inadvertently destroyed the database of a content management system (CMS) based site that took months of work to build.

Googlebot loves surfing the web. He follows from link to link, discovering new pages along the way and adding them to Google’s index.

There has been endless webmaster speculation and worry about the so-called “Google Sandbox” – the indexing time delay for new domain names – rumored to last for at least 45 days from the date of first “discovery” by Googlebot. This recognized listing delay came to be called the “Google Sandbox effect.”

Now that the long-awaited PR update has come and gone, there are reports of the Google’s indexing spider continuing to perform comprehensive site scans. Recently, WebProNews ran an article discussing these types of scans and the majority of the speculation pointed towards the upcoming PR update. Now that this quarter’s update is complete, why is the Googlebot continuing to bombard sites?

There has been a great deal of speculation about a shift coming in Google’s ranking algorithms over the past few weeks.