This, the company says, will allow it to increase its coverage of smartphone content, which means a better search experience for smartphone users.

“The content crawled by smartphone Googlebot-Mobile will be used primarily to improve the user experience on mobile search,” explains software engineer Yoshikiyo Kato. “For example, the new crawler may discover content specifically optimized to be browsed on smartphones as well as smartphone-specific redirects.”

“One new feature we’re also launching that uses these signals is Skip Redirect for Smartphone-Optimized Pages,” adds Kato. “When we discover a URL in our search results that redirects smartphone users to another URL serving smartphone-optimized content, we change the link target shown in the search results to point directly to the final destination URL. This removes the extra latency the redirect introduces leading to a saving of 0.5-1 seconds on average when visiting landing page for such search results.”

Google stresses that webmaster should treat each Googlebot-Mobile request as they would a human user with the same phone user-agent. Google references a blog post from the Webmaster Central blog earlier this year about making sites more mobile friendly.

In that, Webmaster Trends Analyst Pierre Far wrote, “To decide which content to serve, assess which content your website has that best serves the phone(s) in the User-agent string.”

The new smartphone user-agent strings are as follows:

Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

With the Fetch as Googlebot URL submission tool, site owners can now directly request that Google send the web-crawling/indexing Googlebot to the URL that was submitted. While Google’s index will usually find these new pages/sites–especially if they are backlinked–this method speeds up the process. In fact, according to the Google Webmaster Central Blog, URL that have been submitted with Fetch as Googlebot are crawled within a day.

There are obvious benefits with this technique, especially if Google’s index has out-of-date pages for your site and you’d like to see the content updated. Clearly, the same is true for new site launches as well. The sooner it’s in the Google index, the better. Quality backlinks and good content are still the key to gaining rank in the index, but not knowing your site will be indexed almost as soon as its launched is a boon for site owners and SEOs, alike.

The blog post details the steps in order to submit a URL to Fetch, and it’s really quite simple. So much so, in fact, it would be foolish not to take advantage of the option.

How to submit a URL
First, use Diagnostics > Fetch As Googlebot to fetch the URL you want to submit to Google. If the URL is successfully fetched you’ll see a new “Submit to index” link appear next to the fetched URL

Once you click “Submit to index” you’ll see a dialog box that allows you to choose whether you want to submit only the one URL, or that URL and all its linked pages.

When submitting individual URLs, we have a maximum limit of 50 submissions per week; when submitting URLs with all linked pages, the limit is 10 submissions per month.

In other words, you can submit 50 pages a month or 10 sites. The post goes on to say that is if your wanting to submit content like images and/or video, use their Sitemap. The Fetch as Googlebot is intended for content that appears in the web search results, also known as text.

Another update allows users to submit unverified URLs to the Googlebot as well. The difference being, with verified submissions, the person submitting most confirm ownership of the site/URL being submitted. Unverified submissions, obviously, do not require the same proof. There’s even a link provided for these kinds of unverified submissions, which takes you to the Crawl URL page.

The video that leads this post features Google’s Matt Cutts discussing how long it took Googlebot to recrawl a page, and it was posted on May, 2010 on the Google Webmaster Help YouTube page. While not specific, the answer for sites that frequently update content was “a few days,” but now, with Fetch as Googlebot, if it’s that important that your new content is indexed on an even more rapid basis, well, there’s a utility for that.

In the WebmasterWorld forum, member Starchild started a thread by saying, “I saw today that Googlebot got caught in a spider trap that it shouldn’t have as that dir is blocked via robots.txt. I know of at least one other person recently who this has also happened to. Why is GB ignoring robots?”

Another member suggested that Starchild was mistaken, as such claims have been made in the past, only to find that there were other issues at play.

Starchild responded, however, that it had been in place for “many months” with no changes. “Then I got a notification it was blocked (via the spidertrap notifier). Sure enough, it was. Upon double checking, Google webmaster tools reported a 403 forbidden error. IP was google. I whitelisted it, and Google webmaster tools then gave a success.”

Another ember, nippi, said they also got hit by it 4 months after setting up a spider trap, which was “working fine” until now.

“The link to the spider trap is rel=Nofollowed, the folder is banned in robot.txt. The spider trap works by banning by ip address, not user agent so its not caused by a faker – and of course robots.txt was setup up correctly and prior, it was in place days before the spider trap was turned on, and it’s run with no problems for months,” nippi added. “My logs show, it was the real google, from a real google ip address that ignored my robots.txt, ignored rel-nofollow and basically killed my site.”

We’ve reached out to Google for comment, and if and when we receive a response.

Meanwhile, Barry Schwartz is reporting that one site lost 60% of its traffic instantly, due to a bug in Google’s algorithm. He points to a Google Webmaster Help forum thread where Google’s Pierre Far said:

I reached out to a team internally and they identified an algorithm that is inadvertently negatively impacting your site and causing the traffic drop. They’re working on a fix which hopefully will be deployed soon.

Google’s Kaspar Szymanski comment on Schwartz’s post, “While we can not guarantee crawling, indexing or ranking of sites, I believe this case shows once again that our Google Help Forum is a great communication channel for webmasters.”

Since Google dominates the search market share by such a large market share, it is always a good idea to listen to what they have to say about such matters. So when they post a presentation with tips on optimizing crawling and indexing, you’ll probably want to pay attention.

Google has done just that, highlighting things to stay away from, and things you can do to enhance your site’s crawlability. Here is that presentation with specific examples of URLs.

"The Internet is a big place; new content is being created all the time," says Google Webmaster Trends Analyst Susan Moskwa. "Google has a finite number of resources, so when faced with the nearly-infinite quantity of content that’s available online, Googlebot is only able to find and crawl a percentage of that content. Then, of the content we’ve crawled, we’re only able to index a portion."

"URLs are like the bridges between your website and a search engine’s crawler: crawlers need to be able to find and cross those bridges (i.e., find and crawl your URLs) in order to get to your site’s content," continues Moskwa. "If your URLs are complicated or redundant, crawlers are going to spend time tracing and retracing their steps; if your URLs are organized and lead directly to distinct content, crawlers can spend their time accessing your content rather than crawling through empty pages, or crawling the same content over and over via different URLs."

If you want to get crawled faster by Google, you should remove user-specific details from URLs. Specifics of this can be viewed in the slideshow.  Basically, URL parameters that don’t change the content of the page, should be removed and put into a cookie. This will reduce the number of URLs that point to the same content, and speed up crawling.

Google says infinite spaces are a waste of time and bandwidth for all, which is why you should consider taking action when you have calendars that link to infinite numbers of past/future dates with unique URLs, or other paginated data.

Tell Google to ignore pages it can’t crawl. This includes things like log-in pages, contact forms, shopping carts, and other pages that require users to perform actions that crawlers can’t perform themselves. You can do this with the robots.txt file.

Finally, avoid duplicate content when possible. Google likes to have one URL for each piece of content. They do recognize that this is not always possible though (because of content management systems and what have you), which is why the canonical link element exists to let you specify the preferred URL for a particular piece of content.

"Sorry, guys – the fluffy distraction of the dating theme stopped me reading it. I’ll find another, concise article to read about this," says one comment on Google’s post. I guess this isn’t the one he’s looking for since I’m already in my second paragraph about it. But point taken. Let’s try to pick through this nonsense.

One letter talks about cleaning up a site, deleting some old pages, and whether or not 404 pages are ok. "404s are the standard way of telling me that a page no longer exists," says….umm, the GoogleBot. "I won’t be upset—it’s normal that old pages are pruned from websites, or updated to fresher content. Most websites will show a handful of 404s in the Crawl Diagnostics over at Webmaster Tools. It’s really not a big deal. As long as you have good site architecture with links to all your indexable content, I’ll be happy, because it means I can find everything I need."

The post then goes on to address similar items like 301 and 302 redirects for pages linked to by ohers that no longer exist or have been moved, and dynamic pages with changing content.  The GoogleBot says for example:

"Once you’re indexed, it’s the polite way to tell your visitors that your address is still the right one, but that the content can temporarily be found elsewhere. In these situations, a 302 (or the rarer ’307 Temporary Redirect’) would be better. For example, orkut redirects from http://orkut.com to http://google.com/accounts/login?service=orkut, which isn’t a page that humans would find particularly useful when searching for Orkut***.

It’s on a different domain, for starters. So, a 302 has been used to tell me that all the content and linking properties of the URL shouldn’t be updated to the target – it’s just a temporary page.

You can find more info in the post like how supporting the "If-Modified-Since" header and returning 304 can save bandwidth, and you may find the answers to redirect-related questions you have. That is if you can stomach the presentation of the information.

1. Geolocation: It is the process of serving targeted or different content to users on the basis of their locations. Webmasters have the tools to determine a user’s location from preferences stored in their cookies. This information is related to the user’s login or their IP address. Such as, if your website is about theater, then you can always use geolocation techniques to highlight Broadway for a user in New York.
2. IP Delivery: It is the process of serving targeted or different content to users on the basis of their IP address. IP addresses are meant to provide geographic information. IP delivery is quite similar to geolocation, therefore, the techniques are almost the same.
3. Cloaking: It is the method (unethical though) of serving different content to users than to Googlebot. However, this step is considered to be unethical and Google Webmaster Guidelines prohibit Webmasters from using it. If the file that Googlebot crawls is different from the file served to the user then a Webmaster is coined as being in a high-risk category. A program such as md5sum or diff can compute a hash to verify that two different files are identical.
4. First Click Free: If the Webmasters follow Google First Click Free Policy. Then they would be able to include their premium or subscription-based content in Google’s websearch index without violating Google’s quality guidelines. Webmasters can allow all users who find their page using Google search to see the full text of the document, even if they have not registered or subscribed. The user’s first click to the content area is free. But, if the user jumps to another section of the website, then a Webmaster can block the user’s access to the premium or subscribed content with a login or a payment request.

There is also a thread at the Webmaster Help Group, that would be quite interesting for all the Webmasters out there.

Comments

Question & Answers

Googlebot: ACK

Website: Googlebot, you’re here!

Googlebot: I am.

GET / HTTP/1.1

Host: example.com

Connection: Keep-alive

Accept: */*

From: googlebot(at)googlebot.com

User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Accept-Encoding: gzip,deflate

Website: Those headers are so flashy! Would you crawl with the same headers if my site were in the U.S., Asia or Europe? Do you ever use different headers?

Googlebot: My headers are typically consistent world-wide. I’m trying to see what a page looks like for the default language and settings for the site. Sometimes the User-Agent is different, for instance AdSense fetches use "Mediapartners-Google":

User-Agent: Mediapartners-Google

Or for image search:

User-Agent: Googlebot-Image/1.0

Wireless fetches often have carrier-specific user agents, whereas Google Reader RSS fetches include extra info such as number of subscribers.

I usually avoid cookies (so no "Cookie:" header) since I don’t want the content affected too much by session-specific info. And, if a server uses a session id in a dynamic URL rather than a cookie, I can usually figure this out, so that I don’t end up crawling your same page a million times with a million different session ids.

Website: I’m very complex. I have many file types. Your headers say "Accept: */*". Do you index all URLs or are certain file extensions automatically filtered?

Googlebot: That depends on what I’m looking for.

If I’m indexing for regular web search, and I see links to MP3s and videos, I probably won’t download those. Similarly, if I see a JPG, I will treat it differently than an HTML or PDF link. For instance, JPG is much less likely to change frequently than HTML, so I will check the JPG for changes less often to save bandwidth. Meanwhile, if I’m looking for links as Google Scholar, I’m going to be far more interested in the PDF article than the JPG file. Downloading doodles (like JPGs) and videos of skateboarding dogs is distracting for a scholar—do you agree?

Website: Yes, they can be distracting. I’m in awe of your dedication. I love doodles (JPGs) and find them hard to resist.

Googlebot: Me, too; I’m not always so scholarly. When I crawl for image search, I’m very interested in JPGs. And for news, I’m mostly looking at HTML and nearby images.

There are also plenty of extensions (exe, dll, zip, dmg…), that tend to be big and less useful for a search engine.

Website: If you saw my URL, http://www.example.com/page1.LOL111, would you (whimper whimper) reject it just because it contains an unknown file extension?

Googlebot: Website, let me give a bit more background. After actually downloading a file, I use the Content-Type header to check whether it really is HTML, an image, text, or something else. If it’s a special data type like a PDF file, Word document, or Excel spreadsheet, I’ll make sure it’s in the valid format and extract the text content. Maybe it has a virus; you never know. If the document or data type is really garbled, there’s usually not much to do besides discard the content.

So, if I’m crawling http://www.example.com/page1.LOL111 with an unknown file extension, it’s likely that I would start to download it. If I can’t figure out the content type from the header, or it’s a format that we don’t index (e.g. mp3), then it’ll be put aside. Otherwise, we proceed indexing the file.

Website: My apologies for scrutinizing your style, Googlebot, but I noticed your Accept-Encoding headers say:

Accept-Encoding: gzip,deflate

Can you explain these headers to me?

Googlebot: Sure. All major search engines and web browsers support gzip compression for content to save bandwidth. Other entries that you might see here include "x-gzip" (the same as "gzip"), "deflate" (which we also support), and "identity" (none).

Website: Can you talk more about file compression and "Accept-Encoding: gzip,deflate"? Many of my URLs consist of big Flash files and stunning images, not just HTML. Would it help you to crawl faster if I compressed my larger files?

Googlebot: There’s not a simple answer to this question. First of all, many file formats, such as swf (Flash), jpg, png, gif, and pdf are already compressed (there are also specialized Flash optimizers).

Website: Perhaps I’ve been compressing my Flash files and I didn’t even know? I’m obviously very efficient.

Googlebot: Both Apache and IIS have options to enable gzip and deflate compression, though there’s a CPU cost involved for the bandwidth saved. Typically, it’s only enabled for easily compressible text HTML/CSS/PHP content. And it only gets used if the user’s browser or I (a search engine crawler) allow it. Personally, I prefer "gzip" over "deflate". Gzip is a slightly more robust encoding — there is consistently a checksum and a full header, giving me less guess-work than with deflate. Otherwise they’re very similar compression algorithms.

If you have some spare CPU on your servers, it might be worth experimenting with compression (links: Apache, IIS). But, if you’re serving dynamic content and your servers are already heavily CPU loaded, you might want to hold off.

Website: Great information. I’m really glad you came tonight — thank goodness my robots.txt allowed it. That file can be like an over-protective parent!

Googlebot: Ah yes; meeting the parents, the robots.txt. I’ve met plenty of crazy ones. Some are really just HTML error pages rather than valid robots.txt. Some have infinite redirects all over the place, maybe to totally unrelated sites, while others are just huge and have thousands of different URLs listed individually. Here’s one unfortunate pattern. The site is normally eager for me to crawl:

User-Agent: *

Allow: /

Then, during a peak time with high user traffic, the site switches the robots.txt to something restrictive:

# Can you go away for a while? I’ll let you back

# again in the future. Really, I promise!

User-Agent: *

Disallow: /

The problem with the above robots.txt file-swapping is that once I see the restrictive robots.txt, I may have to start throwing away content I’ve already crawled in the index. And then I have to recrawl a lot of content once I’m allowed to hit the site again. At least a 503 response code would’ve been temporary.

I typically only re-check robots.txt once a day (otherwise on many virtual hosting sites, I’d be spending a large fraction of my fetches just getting robots.txt, and no date wants to "meet the parents" that often). For webmasters, trying to control crawl rate through robots.txt swapping usually backfires. It’s better to set the rate to "slower" in Webmaster Tools.

Googlebot: Website, thanks for all of your questions, you’ve been wonderful, but I’m going to have to say "FIN, my love."

Website: Oh, Googlebot… ACK/FIN.

Also, for more information on Googlebot read- Is GoogleBot Crawling CSS Files?

Comments

Lay off my snippet: Webmasters can prevent Google from showing certain text in the search results by using a simple "nosnippet" tag. Not that you wouldn’t want your snippets showing – just that there may be a reason for it out there somewhere.

Caching out: The "noarchive" tag stops Google from showing a cached version of a webpage.

Limited time only: Jill seemed pretty excited about a forthcoming tag from Google called "unavailable_after" which sets a self-destruct timer on page information. This is good for coupons, limited time offers, articles intended to be viewed as previews only.

Don’t Index That Link!

Crow says if you don’t want the links on specific page indexed, you can use the NOINDEX tag for that page. If, however, doing this requires you to continually add NOINDEX and remove it, say for a continually updated and redirected news page, then you may want to go another way and save yourself some grief. If that news page is found through a gateway page, add a NOFOLLOW tag to the entry page instead so the Googlebot will stay put.

The code should look like this: <META NAME="ROBOTS" CONTENT="NOFOLLOW">

Don’t Say That!

Sometimes webmasters don’t want cached versions of webpages showing up in the search results, especially on the chance that the information is dated, or has been updated. Sometimes, webmasters don’t want the usually-important "snippet" displayed in the results either.

To keep the Googlebot from creating a cached version on Google’s servers, Crow says to use the NOARCHIVE tag.

It looks like this: <META NAME="GOOGLEBOT" CONTENT="NOARCHIVE">

For snippets, a NOSNIPPET tag is useful, and kills two birds at once. The NOSNIPPET tag automatically prevents archiving as well.

That code is: <META NAME="GOOGLEBOT" CONTENT="NOSNIPPET">

For the long version, click here. For the first robots.txt tutorial, click here.

Add to Del.icio.us | Digg | Reddit | Furl

Bookmark WebProNews:

Though it has been thought there may be somewhat of a grace period, it now seems robots only notice what’s there when they arrive.

SEORoundtable points us to a webmaster struggling with a hosting company’s reliability raised the question in a Google Groups thread about why the webmaster’s well-ranking site had disappeared from the search results.

Google’s Vanessa Fox responded verifying that the site had indeed been de-listed:

If the host is down when Googlebot tries to access your pages, then those pages may disappear from the index until Googlebot can crawl them again. . .

Googlebot can’t know if or when a page will return when it gets an error response (whether that’s a network down, 404, or other error) and since our primary aim is to have quality search results for users, we don’t want to keep pages that return these types of errors in our index.

The best way of handling these situations, therefore, is to remove the pages, but continue to try to access them, then return them to the index once we get a valid server response.

How long it takes before a site to de-listed is unclear, but Fox confirmed that Googlebot would make a few passes before declaring it dead.

This is not a case for a re-inclusion request, however, as once the site is back up and running, it will be crawled again – though no mention was made as to whether rankings would be restored.

"[I]t does make sense for Google to remove the site, because it may never come back," said a commentator who really, really wants you to know his name is Vince.

Matt Cutts chimes in and agrees with the Vincemeister:

VinceVinceVince makes a good point: sometimes temporarily down pages turn into truly-gone-forever pages, so we have to drop those pages at some point.

But it’s also true that we go back and revisit those pages pretty often and try to re-crawl them in case the site comes back up.

Add to Del.icio.us | Digg | Reddit | Furl

Bookmark WebProNews: