The first tool to get the boot is the Subscriber stats feature. The reported “the number of subscribers to a site’s RSS or Atom feeds.” Google already has the same features included in their Feedburner tool so they suggest users of the current Subscriber stats feature switch to that.

The second removal is for the Create robots.txt tool. This allowed Web sites to generate a robots.txt file that would block a section of a Web site from being crawled by the Googlebot. The reason for removal is that it got very little use. Google says that those people who did use the feature can easily create their own since there are a multitude of other services that create robots.txt files.

The last feature hitting the cutting block is the Site performance feature that’s part of the Webmaster Tools Lab. It let Webmasters check out the average load time of a site’s pages. The reason for its removal is the same as the last – low usage. If you need to check your site’s performance, Google provides the same features in the Site Speed feature in Google Analytics or Google’s PageSpeed tool.

As you can see, these removals are more about removing redundancy than any kind of breaking changes. The retiring of these features only means that Webmasters have to switch to one of the many other options available. Chances are you’re already using one of those alternative options. If you are still using one of the above features, you have two weeks to say your goodbyes.

Google’s Matt Cutts spoke about a few options for these files in the latest Webmaster Help video, in response to a user-submitted question: “Is it better to have a blank robots.txt file, a robots.txt that contains User-agent: *Disallow:” or no robots.txt file at all?”

“I would say any of the first two,” Cutts responded. “Not having a robots.txt file is a little bit risky – not very risky at all, but a little bit risky because sometimes when you don’t have a file, your web host will fill in the 404 page, and that could have various weird behaviors. Luckily we are able to detect that really, really well, so even that is only like a 1% kind of risk.”

“But if possible, I would have a robots.txt file whether it’s blank or you say User-agent: *Disallow nothing, which means everybody’s able to crawl anything they want is pretty equal,” said Cutts. “We’ll treat those syntactically as being exactly the same. For me, I’m a little more comfortable with User-agent: * and then Disallow: just so you’re being very specific that ‘yes, you’re allowed to crawl everything’. If it’s blank then yes, people were smart enough to make the robots.txt file, but it would be great to have just like that indicator that says exactly, ‘ok, here’s what the behavior is that’s spelled out.’ Otherwise, it could be like maybe somebody deleted everything in the file by accident.”

“If you don’t have one at all, there’s just that little tiny bit of risk that your web host might do something strange or unusual like return a ‘you don’t have permission to read this’ file, which you know, things get a little strange at that point.,” Cutts reiterated.

All of this, of course, assumes that you want Google to crawl your site.

In another video from Cutts we looked at yesterday, he noted that Google will sometimes use DMOZ to fill in snippets in search results when they can’t otherwise see the page’s content because it was blocked by robots.txt. He noted that Google is currently looking at whether or not it wants to continue doing this.

In fact, WebProNews has articles dating back to 2005 discussing this very subject. However, according to the latest appeal loss, the saga may finally be coming to an end. According to a post over at CNet, the latest attempt by Perfect 10, one that seeks to punish Google for being a search engine that works as its supposed to, has been denied.

Here’s the gist:

The Ninth Circuit ruled that Perfect 10, a porn studio with a long history of filing copyright suits against Internet companies, rejected a request for a preliminary injunction against Google. The court said that Perfect 10 didn’t present enough evidence to prove that it would suffer irreparable harm from the photos.

You see, Perfect 10′s content is primarily hidden behind a pay wall, meaning, in order to see their index of naked women, you have to pay for it. Unfortunately for the Perfect 10 web developer, who apparently didn’t understand how to manipulate a robots.txt file, apparently, Perfect 10 images began appearing in Google’s image search results.

Regardless of the fact that there are an unending amount of tutorials and instructional sites that inform developers how to keep their paid content from appearing in free image searches, for some reason, Perfect 10 felt it was Google’s fault their paid content was going out to the world for free.

In fact, Perfect 10′s claim was Google’s image search cost them something in the area of $50 million. Disregarding the fact that, again, the blame should’ve been placed directly on the head of the Perfect 10 web developer, the company tried, and tried, and tried again to make Google (and others) pay for their design inadequacies.

Each time, these attempts did little but clog up a court system that’s already bursting at the seams.

There was, apparently, a slight moment of victory when another judge upheld a Perfect 10 filing against Megaupload, a file-sharing site that allows others to swap files via email or direct download. Granted, Megaupload doesn’t have the money Google does, but even the smaller victories count, right?

It should also be noted that when a “Perfect 10” search is conducted in Google Images, the amount of content originating from the site in question is negligible, even if SafeSearch is turned off. This mean that, even though the Perfect 10 web developers finally figured out how to protect their paid content, the company still wants to nail Google to the cross.

A semi-recent post on the Perfect 10 blog reveals as much. The title, “Google Is Destroying The Entertainment Industry” reeks of a “give me back my money” approach, courtesy of Mel Gibson and South Park:

In the WebmasterWorld forum, member Starchild started a thread by saying, “I saw today that Googlebot got caught in a spider trap that it shouldn’t have as that dir is blocked via robots.txt. I know of at least one other person recently who this has also happened to. Why is GB ignoring robots?”

Another member suggested that Starchild was mistaken, as such claims have been made in the past, only to find that there were other issues at play.

Starchild responded, however, that it had been in place for “many months” with no changes. “Then I got a notification it was blocked (via the spidertrap notifier). Sure enough, it was. Upon double checking, Google webmaster tools reported a 403 forbidden error. IP was google. I whitelisted it, and Google webmaster tools then gave a success.”

Another ember, nippi, said they also got hit by it 4 months after setting up a spider trap, which was “working fine” until now.

“The link to the spider trap is rel=Nofollowed, the folder is banned in robot.txt. The spider trap works by banning by ip address, not user agent so its not caused by a faker – and of course robots.txt was setup up correctly and prior, it was in place days before the spider trap was turned on, and it’s run with no problems for months,” nippi added. “My logs show, it was the real google, from a real google ip address that ignored my robots.txt, ignored rel-nofollow and basically killed my site.”

We’ve reached out to Google for comment, and if and when we receive a response.

Meanwhile, Barry Schwartz is reporting that one site lost 60% of its traffic instantly, due to a bug in Google’s algorithm. He points to a Google Webmaster Help forum thread where Google’s Pierre Far said:

I reached out to a team internally and they identified an algorithm that is inadvertently negatively impacting your site and causing the traffic drop. They’re working on a fix which hopefully will be deployed soon.

Google’s Kaspar Szymanski comment on Schwartz’s post, “While we can not guarantee crawling, indexing or ranking of sites, I believe this case shows once again that our Google Help Forum is a great communication channel for webmasters.”

But first, Warden’s story. Read the whole thing in his words here for more context about what he wanted to do with the data, but to make a long story short, he was building a tool to bring data from email and various social networks into one place to make it easier for users to manage their contacts, and he crawled Facebook. He says he checked Facebook’s robot.txt, and that "they welcome the web crawlers that search engines use to gather their data," so he wrote his own. He was able to obtain data like which pages people were fans of and links to a few of their friends. He created a map showing how different countries, states and cities were connected to each other and released it so that others could use the information. Once Facebook caught wind of this, they threatened legal action. Warden writes:

Their contention was robots.txt had no legal force and they could sue anyone for accessing their site even if they scrupulously obeyed the instructions it contained. The only legal way to access any web site with a crawler was to obtain prior written permission.

Obviously this isn’t the way the web has worked for the last 16 years since robots.txt was introduced, but my lawyer advised me that it had never been tested in court, and the legal costs alone of being a test case would bankrupt me. With that in mind, I spent the next few weeks negotiating a final agreement with their attorney. They were quite accommodating on the details, such as allowing my blog post to remain up, and initially I was hopeful that they were interested in a supervised release of the data set with privacy safeguards. Unfortunately it became clear towards the end that they wanted the whole set destroyed.

Facebook Public Policy Communications Manager Andrew Noyes tells WebProNews, "Pete Warden aggregated a large amount of data from over 200 million users without our permission, in violation of our terms. He also publicly stated he intended to make that raw data freely available to others. Warden was extremely cooperative with Facebook from the moment we contacted him and he abandoned his plans."

"We have, and will continue to, act to enforce our terms of service where appropriate," adds Noyes.

Noyes pointed to Facebook’s Statement of Rights and Responsibilities, which states that "You will not collect users’ content or information, or otherwise access Facebook, using automated means (such as harvesting bots, robots, spiders, or scrapers) without our permission." That’s under the safety section, by the way.

"I’m bummed that Facebook are taking a legal position that would cripple the web if it was adopted (how many people would Google need to hire to write letters to every single website they crawled?), concludes Warden. "And a bit frustrated that people don’t understand that the data I was planning to release is already in the hands of lots of commercial marketing firms, but mostly I’m just looking forward to leaving the massive distraction of a legal threat behind and getting on with building my startup."

Hearing some of what both parties have to say on the issue, what are your thoughts? Discuss here.

If we hear back from Warden or if Facebook offers us more insight into the situation, which I’m told may still happen, I’ll update this article.

Original Article: Google has created a new web crawler specifically for Google News. What this means is that publishers who do not want Google News to index their content can more easily control that. That also applies to publishers who don’t wish to completely cut out indexing, but wish to limit/restrict certain elements of their content from being indexed.

Google offers this new crawler at a time when Google’s relationship with online news is a heavy focus of discussion throughout the industry, with the FTC’s meeting of the media minds taking place. This week Google already announced some changes to how it handles paid content (by offering a five-article limit for the "first click free" plan). Now the company appears to be further extending its olive branch to concerned publishers (whether or not that will be enough is another discussion).

In the past, publishers have been able to block Google from content via robots.txt and the Robots Extension Protocol (REP). They have also been able to keep content out of Google News and stay in Google Search, by using a contact form provided by Google. Now, Google is making it so publishers don’t even have to contact them.

"Now, with the news-specific crawler, if a publisher wants to opt out of Google News, they don’t even have to contact us – they can put instructions just for user-agent Googlebot-News in the same robots.txt file they have today," says Google News Senior Business Product Manager Josh Cohen. "In addition, once this change is fully in place, it will allow publishers to do more than just allow/disallow access to Google News. They’ll also be able to apply the full range of REP directives just to Google News. Want to block images from Google News, but not from Web Search? Go ahead. Want to include snippets in Google News, but not in Web Search? Feel free. All this will soon be possible with the same standard protocol that is REP."

"While this means even more control for publishers, the effect of opting out of News is the same as it’s always been," says Cohen. "It means that content won’t be in Google News or in the parts of Google that are powered by the News index. For example, if a publisher opts out of Google News, but stays in Web Search, their content will still show up as natural web search results, but they won’t appear in the block of news results that sometimes shows up in Web Search, called Universal search, since those come from the Google News index."

Cohen says Google News users shouldn’t notice any difference in their experience with the service. It will be interesting to see the reaction from disgruntled publishers, and whether or not this will make any significant difference in how they view Google News.

Related Articles:

> Google Changes How it Handles Paid Content

> Minds of the Media Gather to Discuss Future of News

> Google Okay With Blocking News Corp.

> Is it Really Crazy to Block Google?

Cutts says Google gets a lot of complaints from webmasters who say the search engine is violating their robots.txt files, with which they intend to keep Google from crawling certain pages. Sometimes those URLs still end up in search results.

According to Matt, what is happening in most cases is that when someone’s saying "I blocked example.com/go" in robots.txt, it turns out that the snippet Google returns in search results just brings back a URL with no text for the snippet. The reason for this is that Google didn’t actually crawl the page.

"It did abide by robots.txt. You told us this page is blocked, so we did not fetch this page," says Matt. It is a URL reference. "We saw a link to it, but we didn’t fetch the page itself," he explains.

Google didn’t actually fetch the page itself, and that’s why there’s no text snippet. In case you were wondering what the point of showing them at all is, Cutts breaks out an example looking at the California DMV, whose site is: www.dmv.ca.gov.

Cutts notes that at one point the California Department of Motor Vehicles had a robots.txt that blocked all search engines. "Now these days pretty much every site is savvy enough, you know, at one point the New York Times and eBay and a whole bunch of different sites would use robots.txt," he says.

If someone searches for "California DMV" in Google, there’s pretty much only one answer, he says. So that is the answer that Google wants to return. Luckily for Google a lot of people were linking to that page with the anchor text "California DMV". That helps Google be able to return the result without having to crawl the page.

Cutts also says that they can get descriptions from a directory like the Open Directory Project (DMOZ). He cites Nissan and Metallica.com as examples of sites that used to block Google with robots.txt. They had been listed in the Open Directory Project, however, and Google went and got the information from there to include as the snippet.

When this type of thing happens, it looks like the page was crawled, when in fact it wasn’t. "So we are able to return something that can be very helpful to users without violating robots.txt by not crawling that page," says Cutts.

He also notes that when you don’t want pages to show up, you can use the "noindex" meta tag at the top of the page. When Google sees this tag, it drops the page from its search results completely. Another option is the URL removal tool.

On Monday, Whitehouse.gov was still blocking search engine access to a tremendous amount of website information. In all, the robots.txt file used the “Disallow” command 2,400 times, blocking search engine access to information on earmarks, African American history, photo essays from various places and events, first lady initiatives, the budget, defense, on and on.

Obviously, if posted on the White House website, none of this information would be considered classified, or even sensitive, so it’s unclear why Bush’s web crew felt the need to prevent the site from being searchable.

Regardless, all search crawler barriers were removed with the Bushes’ furniture, the “Disallow” command lines reduced from 2,400 to basically none.

Requests for comment and/or explanations from prior and current administrations were not returned. Meanwhile, it appears President Obama will be able to keep his Blackberry after all—with some super-encryption functionality added to it. 
 

"A Google Profile is simply how you represent yourself on Google products — it lets you tell others a bit more about who you are and what you’re all about. You control what goes into your Google Profile, sharing as much (or as little) as you’d like." – Google

The news is that Google Profiles are now being indexed by search engines!

It is because of the fact that Google has added a new line into their robots.txt file and has lifted the nonindex style command for these files.

Garett Rogers writes :

Just about a half hour ago, Google added a new line into their robots.txt file which makes all those profiles (or at least 50,000 of them) crawlable by search engines. The new entry tells search engines to use “http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xml” as a sitemap. The sitemap looks something like this:

http://www.gstatic.com/s2/sitemaps/sitemap-000.txt
2008-10-15

http://www.gstatic.com/s2/sitemaps/sitemap-001.txt
2008-10-15

According to Garett, Google would soon launch a “People Onebox” at one point. Google has earlier performed this task for local and books as well.

Comments

I had a lot of comments about whether or not Adwords was necessary, so I thought I’d try it again without running Adwords this time. Here’s how it all played out:

1) I created 5 pages of content (Home, FAQ, About Us, etc.).

2) I put them in a simple template with site-wide links. I also linked to it from one of my other sites (it’s very relevant so it makes sense).

3) I added tagged the site on only 2 social bookmarking sites.

4) Commented in 1 forum, put the URL in one directory (niche specific), and submitted it to Digg.

5) Installed Google Analytics

6) Created a sitemap, pinged Google, and put the sitemap in my Robots.txt. Logged into Google Webmaster Central and submitted my sitemap there.

When I checked exactly 24 hours later I was sitting in the index and had already begun to get a few visitors from Google.

I had previously done Google adwords out of both necessity (get quick traffic) but also because of the trust factor I believe it gives to Google, and the fact that Google integrates a quality factor into their quality score (so they come to your site and look at it). Obviously this is just one test compared to several others I’ve done with Adwords, but it seems its very possible without running some ads.

Anyone else seeing 24 hour indexing for new sites?

Comments