He has followed that up with a full-fledged post on the official Google blog about the new tag. The unavailable_after META tag informs the Googlebot when a page should be removed from Google’s search results:

This information is treated as a removal request: it will take about a day after the removal date passes for the page to disappear from the search results. We currently only support unavailable_after for Google web search results.

After the removal, the page stops showing in Google search results but it is not removed from our system.

Fully removing something from Google still requires the URL removal tool, found as one of Google’s Webmaster Central tools.

Google also extended some control over assets beyond web pages to webmasters. Those who publish PDF, audio, video, or other file types can direct the crawler on how Google should manage access to them from its index.

“We’ve extended our support for META tags so they can now be associated with any file,” said Crow. “Simply add any supported META tag to a new X-Robots-Tag directive in the HTTP Header used to serve the file.”

Supported META tags include options like noarchive, nosnippet, noindex, and unavailable_after. Google sees these as offering enough flexibility to satisfy site publishers; we imagine they have organizations like AFP and Copiepresse in mind here.

The first thing I noticed is how damn close the adsense adverts are to the rating system. I accidently clicked on one while voting quickly. I wonder how many others have done the same?

Which brings about the question – Where should Google draw the line between adverts that honestly integrate as part of a website and those that go out of there way to optimise specifically for the accidental click…

How much are these ads pulling in? I would love to see the conversion rate on them. How many of those accidental clicks like mine are getting filtered and classed as invalid?

Would you trust your ads sitting there next to a voting system?!

From what I can see the ad placement is not breaking Googles terms and conditions. Some advertisers just need to use site exclusion.

Comments

In an article over at the New York Times, Kim Malone Director of online sales and operations for Google Adsense spoke of the big change.

In the next few months, Google’s advertiser reports will begin listing the sites where each ad runs, Ms. Malone said. She added that advertisers on the Google networks would soon be able to bid on contextual ads on particular Web sites rather than simply buying keywords that appeared across Google’s entire network. Still, Ms. Malone said she did not see much of consequence coming from the changes. “We don’t expect a lot of demand for that placement targeting,” she said. “It’s the brand, the display advertisers who care where they run.”

Well, I can see a LOT of consequences coming from these changes, with Googles site exclusion tool becoming more meaningful & important to all advertisers…

More chatter over at the SEW Forums.

Comments

Tag:

Add to Del.icio.us | Digg | Reddit | Furl

Bookmark WebProNews:

It imparted important details to the web publishers about how they can control indexing and accessing of sites by search engines and Google itself. The important tool for the same purpose is the robots.txt file. Robots.txt file gives powerful control to site owners on how the site is searched.

The more recent post on robots exclusion protocol provides more details and examples of mechanisms to control access and indexing of your website by Google.

This post simplifies the procedure of preventing Googlebot from following a link. “Usually when the Googlebot finds a page, it reads all the links on that page and then fetches those pages and indexes them. This is the basic process by which Googlebot "crawls" the web. This is useful as it allows Google to include all the pages on your site, as long as they are linked together.” It further says that one can add the NOFOLLOW tag to a  page which tells the Googlebot not to follow any links it finds on that page.

Further on, the post intricately explains how to control caching and snippets. “Usually you want Google to display both the snippet and the cached link. However, there are some cases where you might want to disable one or both of these. For example, say you were a newspaper publisher, and you have a page whose content changes several times a day. It may take longer than a day for us to reindex a page, so users may have access to a cached copy of the page that is not the same as the one currently on your site. In this case, you probably don’t want the cached link appearing in our results."

To know more on how robots exclusion protocol can assist read the complete post.

Comments

Reddit | Furl

Bookmark WebProNews:

If I disable Google to my Web site, its possible Google.com erase or drop down my Web site for his directory?

Many thanks for your time and keep up the good work.

Answer: Many thanks for posting this question because Web server issues and excluding robots are a very important aspect of search engine marketing (SEM). The reader did not specifically state how he kept Googlebot from spidering his site. I am assuming that the reader used the Robots Exclusion Protocol.

Robots Exclusion Protocol

The Robots Exclusion Protocol is a means of instructing robots (or spiders) from crawling a site. With the Robots Exclusion Protocol, Web site owners can instruct search engine spiders to not index individual Web pages, subdirectories, or even an entire site. Instructions can also be tailored for individual search engines.

There are two types of robots exclusion: a meta tag or a text file.

To let Google know that you do not want a page crawled, you can create the following meta tag:

<META NAME="GOOGLEBOT" CONTENT="NOINDEX, NOFOLLOW">

To let all search engine spiders know that you do not want a page crawled, you can create the following meta tag:

<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

For this tag to be effective on a whole site, you will have to place this tag on every page of your site. This process can be quite boring and time consuming. For that reason, I prefer to use the robots exclusion text file, commonly referred to as robots.txt, because it can easily be applied to an entire site.

The robots.txt is a text file that you place on your server that instructs search engine spiders to NOT record the information in specified areas on your Web site, and not to follow the links on your Web site. In other words, text file lets the search engine spiders know which sections of your site are off limits.

I usually create my robots.txt files in NotePad (PC) or SimpleText (Mac). But you can create simple text files in HTML software such as Dreamweaver.

Google will request the robots.txt file before trying to index any page within your site. For example, if do not want Google to record any of the information on the site, type the following text into a text editor:

User-agent: Googlebot
Disallow: /

Be sure to save the file as robots.txt. Do not use any other file extension. If you save the file as a Word document and call it robots.doc, Google will ignore that file.

When search engines crawl to frequently

I understand the reader’s concern about bandwidth. If Google or any search engine crawls a site too frequently, it takes up bandwidth. All of us pay for bandwidth.

However, when you instruct Google (or any search engine) to not crawl your site, you are essentially communicating, “Don’t show my Web pages in your search results.”

I do not believe the reader’s intention was to exclude all of his Web pages from Google search engine results pages (SERPs). He just wants Google not to request pages from his server so often.

Google actually has a Web page with this information and an email address. This is a direct quote from Google’s Webmaster FAQs page:

“Please send an email to googlebot@google.com with the name of your site and a detailed description of the problem. Please also include a portion of the weblog that shows Google accesses, so we can track down the problem more quickly on our end.”

The URL for the information on this page is at http://www.google.com/webmasters/faq.html.

When to use the Robots Exclusion Protocol

Some content is not important to site visitors and search engines, such as items in a CGI-BIN directory. When your target audience searches for information, they are not interested in your site’s programs that generate your forms or your drop-down menus. They are not interested in a section of a Web site that is under construction. They are not interested in redundant content, either. Using the Robots Exclusion Protocol ensures that unnecessary information is not shown in search results pages.

For more details about the Robots Exclusion Protocol, please visit: http://www.robotstxt.org/wc/faq.html.

Shari Thurow is Marketing Director at Grantastic Designs, Inc., a full-service search engine marketing, web and graphic design firm. This article is excerpted from her book, Search Engine Visibility (http://www.searchenginesbook.com) published in January 2003 by New Riders Publishing Co. Shari can be reached at shari@grantasticdesigns.com.

Shari Thurow Answers SEO Questions: Click Here For Free Answers