Unvalidated Robots.Txt Risks Google Banishment

Webmasters welcome being dropped out of Google about as much as they enjoy flossing with barbed wire. Making it easier for Google to do that would be anathema to being a webmaster. Why willingly exclude one’s site from Google?

That could happen with an unvalidated robots.txt file. Robots.txt allows webmasters to provide standing instructions to visiting spiders, which contributes to having a site indexed faster and more accurately.

Google has been considering new syntax to recognize within robots.txt. The Sebastians-Pamphlets blog said Google confirmed recognizing experimental syntax like Noindex in the robots.txt file.

This poses a danger to webmasters who have not validated their robots.txt. A line reading Noindex: / could lead to one’s site being completely de-indexed.

The surname-less Sebastian recommended Google’s robots.txt analyzer, part of Google’s Webmaster Tools, and only using the Disallow, Allow, and Sitemaps crawler directives in the Googlebot section of robots.txt.

follow me on Twitter

From what I have read on the pre-release materials, it sends trackback notifications to create one-way links to your website.

That means:-

• You are not being referenced on the site sending the trackback
• You are not being linked to by the site sending the trackback

This is Blackhat, lots of tools have been available for this kind of thing in the past, and this kind of software is actually fairly unpopular even among most shady web masters.

Fortunately it is not very effective on active blogs for a number of reasons.

• It is becoming much harder to monetise sites which are using trackback spam. Bloggers tend to be quite vocal and keen to report such sites to Google.
• Blogspot uses linkbacks – even splogs have to reciprocate a link – Google should really provide a way to moderate linkbacks.
• WordPress has a number of good solutions to handle trackback spam – my favorite is Spam Karma which always checks for a reciprocal link, and my installation also has quite a growing blacklist.
• Most bloggers who receive a trackback actually check them

There might be some minor success on old abandoned WordPress blogs, and maybe abandoned blogs on other formats, but those generally are low quality, and fairly low traffic.

So even if you want to wear a black hat, this type of tool is becoming more and more useless daily.

Disclosure: I do have some past history with the developer, as I purchased Domain Dashboard his previous product, and whilst many people were satisfied, I have never managed to get Domain Dashboard working consistently for all my domains, the product hasn’t been updated for months, and support was unresponsive.

I didn’t ask for a refund – if the product had worked with my various hosting plans, it would have saved me a huge amount of time daily. Maybe in the future it will, because I am thinking about moving a lot of my hosting.

I don’t think there is much of a legitimate market for this new product. Most people I know who dabble on the dark side already have similar solutions, and those who I would look on as "grey hat" don’t dabble with this kind of trackback spam.

* Originally published at AndyBeard.eu
Comments

Tag:

A little while back, we explored the idea that RSS, as an automatic distribution agent, could imply permission to republish. But that involves two human parties, essentially, with a technical agent in between.

A court battle in Colorado, however, focuses on claims brought by Suzanne Shell against the Internet Archive’s Wayback Machine, which holds in searchable perpetuity pages that appear on the Web, for future historical reference.

Shell owns the website www.profane-justice.org, devoted to providing information and support for people who feel they’ve been unlawfully targeted by state agents (like police or child services organizations) or unfairly accused of child abuse.

A notice appears on the site stating that users copying or distributing the content on the site automatically agree to the terms of a contract. Failure to abide carries a fee of $5,000 per page copied; $250,000 per occurrence of unauthorized use, and a charge of $50,000 for each occurrence of failure to pay, plus costs and triple damages.

There was no mechanism in place on the site (such as in the robots.txt file) to prevent Internet Archive’s robot from scanning, copying, and storing the pages on Shell’s website. She discovered that the Wayback Machine had reproduced the contents of her website about 87 times in five years, "and displayed her entire website to the public daily during that period." 

Shell sued the company for conversion, civil theft, breach of contract, and violations of the Racketeering Influence and Corrupt Organizations Act (RICO) and the Colorado Organized Crime Control Act (COCCA).

As might be guessed, most of these claims were dismissed. But the breach of contract claim is still under consideration, awaiting more information. The question that will be decided, ultimately, is whether a web crawler that is not blocked by a website can legally be bound by a contract posted on the site.

The outcome of that question could also have important impact on Web-crawling and Internet copyright law itself. Search engines like Google have leaned on Fair Use principles when scouring the Web (and off-line libraries) for information. The Google Book Search project defenders have claimed that publishers can "opt out" of their network by letting Google know their desire to do so.

Of a similar vein, then, is Web crawling. Webmasters must opt-out of indexing via the robots.txt file, preventing the spider from crawling the site. If the courts continue to back search engines’ and other Internet companies’ right to copy and index at will, then the whole copyright system, by default, it would seem, goes opt-out, too.

Copyright law blogger John Ottaviani, on Eric Goldman‘s blog, goes into the Internet Archive v. Shell case and its implications in greater detail.  

Wikipedia defines cloaking as:

Cloaking is a black hat search engine optimization (SEO) technique in which the content presented to the search engine spider is different from that presented to the users’ browser. This is done by delivering content based on the IP addresses or the User-Agent HTTP header of the user requesting the page. When a user is identified as a search engine spider, a server-side script delivers a different version of the web page, one that contains content not present on the visible page. The purpose of cloaking is to deceive search engines so they display the page when it would not otherwise be displayed.

Basically, you are presenting search engine bots with a certain kind of content while delivering different content to the site visitor. Normally, the cloaked pages are created to fool search engines in order to get better result rankings. However, what if you are using cloaking procedures for legitimate reasons like protecting paid content or serving different content based on the visitor’s IP address? Should sites doing this be subject to the same penalties? It depends on whom you ask.

On the Graywolf SEO blog, readers are asked how they save articles from the New York Times because they are only available to the public for a limited amount of time. Once an article gets to a certain age (2 weeks), the NYT hides it unless the Google crawler (or other search engine bot) requests it – fitting the definition of cloaking, something Graywolf takes the search engines (and the NYT) to task over.

Philipp Lenssen of Google Blogoscoped also has some issues with Google seemingly allowing WebmasterWorld to cloak their pages, which goes against the search engine’s webmaster guidelines. For his post, Philipp conducted a search related to CMS and PHP and a WebmasterWorld post held the first position. However, when Lenssen tried to access the page from the search results, he was taken to a login page – another example of cloaking in action (unfortunately, when I try to duplicate the search, I am taken directly to the content).

Both Lenssen and Graywolf wonder how these otherwise legitimate sites get away with these cloaking exercises when Google and the rest are explicitly against the act. However, the examples given by both bloggers represent the “white-hat” side of cloaking in the sense they are not trying to game the search engines. These sites and companies are merely trying to protect their content.

However, this does not matter to either Lenssen or Graywolf. Because Google has actually addressed this issue in their guidelines, both believe there should be no quarter when it comes to punishing the guilty parties, whether the sites have a legitimate reason for cloaking or not. They also feel Google’s Matt Cutts should address the situation so there will be no more confusion.

At the Chicago SES, while it was never explicitly stated (at least in the sessions I attended), there seems to be a growing sentiment that as long as the webmaster isn’t trying to be deceptive, search engines will tolerate some cloaking. The Wikipedia page discusses delivering content based on a visitor’s IP location (IP Delivery) as one of the instances where cloaking is indeed accepted. Although, the explanation also points out IP delivery isn’t the best example of cloaking because the content in question is not being hidden from search engines or users; it’s just being manipulated based on the visitor’s location.

The question remains, however – should the search engines punish pages being cloaked for content protection reasons? If you follow the two bloggers cited in this article, then yes, all sites doing so should be punished. If they are not going to punish these sites, then the search engine spokesmen and women should speak up and address the confusion.

Add to Del.icio.us | Digg | Reddit | Furl

Chris Richardson is a search engine writer and editor for WebProNews. Visit WebProNews for the latest search news.

If they can’t “see” you, you might as well not exist–and if they can’t understand your code, no amount of keywords can get you in the Golden Top 20.

The problem that many website developers used to encounter was that search engines worked differently; so you could end up with a high ranking in Lycos but languish at the bottom of Google. How exactly should you optimize your site so you perform well in all search engines?

Enter ROR (short for Resources for a Resource), an independent XML format that translates your content in a way that all search engines can understand.

Think of it as a web spider’s Cliff’s Notes. it describes all the objects, services, discounts, images, podcasts, etc. If it’s on the site, it’s on the ROR feed, but in a format that’s easy to process and removes all risks of skipping or ignoring a link.

ROR calls its “magic file” structured feeds, which guide search engines as they scan the text. Unlike Google Sitemaps, it’s universally understood–and very easy to process. It’s also more detailed. It doesn’t just give a map or “table of contents”, it actually summarizes what’s inside. It’s also been in existence far longer than Google, so its reliability has been proven by time.

Though it’s been around for a long time, ROR is by no means outdated. The majority of the file formats are already available in ROR, although it is currently being updated to keep up with the growing number of website innovations. But to avoid being too unwieldy, the ROR system tries to re-use existing data structures. It boasts of being very streamlined, a strength that makes it one of the more efficient ways of indexing a site.

Usually the ROR feed is located in the directory, and is named by default ror.xml. It is possible to rename the file, and the search engines will still find it. The only thing it needs to have is a <link> tag in your main page (between the <head> and </head> tags). Another alternative is to create a smaller ror.xml file which will direct the search engines to the ROR feed.

You can create this file in the ROR sitemap generator.

XML-Sitemaps.com has an online sitemap generator that creates XML, HTML, text and ROR sitemaps and also provides some useful SEO tools.

Search engines find sites mostly by following links from sites that are already known to find new sites and pages. The sofware programs that search engines use to perform this task are often called “Bots” or “Spiders”. You get the analogy right? “Web”, “Spiders”, “Crawl”.

If you don’t make sure your site is crawable and indexed, then you’re putting your web site at a gross disadvantage. For example, if you have a 1,500 page web site and only 700 pages are getting indexed, that’s like showing up to a baseball game with only 5 of your players. You need the whole team to win, so make sure your site is crawlable and getting indexed properly.

As search engine spiders crawl the links of your site, they make copies of the pages and then peform other functions that strip away the code, interpret the remaining text as well as other analysis that ultimately leads to a score for the page and association of the page to certain words. All of this along with links into your site from other web sites, influence your rankings. On the PPT slides from the recent Google Press Day, it says there are over 200 “signals” used to rank web pages on Google.

Here’s an animation of how Yahoo’s SLURP crawls a network of pages.

If a search engine has difficulty “crawling” the links within your site, then the pages either won’t get indexed at all or will only get partially indexed – neither of which will help your site’s rankings.

OK, now I know why, but what about the how? Search engine friendly URLs are simple. As in, short and simple. For example, the url of this web page is: http://www.toprankblog.com/2006/05/seo-tips-let-the-spider-crawl/

It could be something like http://www.toprankblog.com/?pageid=234234&articleid=5tips&postid=435345 or something similar. The second url is still crawlable, but if you got to pick, which one would you prefer to index? Which one would you be more likely to remember as a user?

Most problems with links and the URLs they point to getting crawled involve shopping cart software or content management systems that place a lot of extra information in the web page URL. If references to “?sid=” or a large number of variables are included in the URL it can cause issues. Search engine bots are leary of “spider traps” or situations with calendars or where an infinite number of url versions display the exact same web page. This often occurs with the use of session ids.
Simple and short urls are typically the easiest to crawl so try to use a content management system that produces short, clean URLs.

You can also use programs like Google Sitemaps to submit your site URLs for inclusion. There is no guarantee it will work, but it’s been pretty effective for many web sites. Google Sitemaps works in conjunction with a normal “crawl” of your web site. Plus there are many useful troubleshooting features and information available with Google Sitemaps. You can also submit an RSS feed or plain text file of your site’s URLs to Yahoo.

There’s actually quite a bit more involved with making your site crawlable, but I’ll leave it at this for now.

Resources on crawler friendly web sites:

• Google Webmaster Guidelines
• Yahoo! Search Content Quality Guidelines
• MSN guidelines
• Ask/Teoma Crawler Information
• Smart IT Consulting Weblog
• Speed Bumps and Walls – SEOMoz
• “Meet the Crawlers” SES NYC 2006

Add to Del.icio.us | DiggThis | Yahoo! My Web | Furl

Lee Odden is President and Founder of
TopRank Online Marketing, specializing in organic SEO, blog
marketing and online public relations. He’s been cited as a search
marketing expert by publications including U.S. News & World Report and
The Economist and has implemented successful search marketing programs
with top BtoB companies of all sizes. Odden shares his marketing
expertise at Online Marketing Blog offering
daily news, interviews and best practices.

As the story goes, a web development firm was given a contract to rebuild an existing site using a CMS. As the client already had a site with a significant amount of content, they took it slow and fully populated the site with all the content from the previous site. When they had finally uploaded everything, they took the site live.

“Things went pretty well for a few days after going live. But, on day six, things went not-so-well: all of the content on the website had completely vanished and all pages led to the default “please enter content” page. Whoops.”

After painstaking investigation, Googlebot, the spider Google uses to find information on the web, was found to be the cause.

When one of the users entered information to the CMS (using copy and paste), he or she included an EDIT hyperlink that was left in a multi-user document. As a human error, this wouldn’t normally be a problem because users are required to log-in with a password before they can make changes.

“But, the CMS authentication subsystem didn’t take into account the sophisticated hacking techniques of Google’s spider. As it turns out, Google’s spider doesn’t use cookies, which means that it can easily bypass a check for the “isLoggedOn” cookie to be “false”. It also doesn’t pay attention to Javascript, which would normally prompt and redirect users who are not logged on. It does, however, follow every hyperlink on every page it finds, including those with “Delete Page” in the title.”

In short, Googlebot muscled its way into the CMS and followed the edit link. The rest was history, or at least that’s what became of months of work. Fortunately, a recent backup of the full site was available for uploading.

Add to | Digg | Yahoo! My Web

Technorati:

Jim Hedger is the SEO Manager of StepForth Search Engine Placement Inc. Based in Victoria, BC, Canada, StepForth is the result of the consolidation of BraveArt Website Management, Promotion Experts, and Phoenix Creative Works, and has provided professional search engine placement and management services since 1997. http://www.stepforth.com/ Tel – 250-385-1190 Toll Free – 877-385-5526 Fax – 250-385-1198

I started digging into this and you’d be surprised at what I found out.

First, lets look at the timeline of events:

In Late September some astute spider watchers over at Webmasterworld spotted unique Googlebot activity. In fact, it was in this thread that the bot was first reported on. It concerned some posters who thought that perhaps this could be regular users masquerading as the famous bot.

Early on it also appeared that the new bot wasn’t obeying the Robots.txt file. This is the protocol which allows or denies crawling to parts of a website.

Speculation grew on what the new crawler was until Matt Cutts mentioned a new Google test data center. For those that don’t know, Matt Cutts is a senior engineer with Google and one of the few Google employees talking to us “regular folk.” This mention happened in November.

There wasn’t much mention of Big Daddy until early January of this year when Matt again blogged about it asking for feedback.

Much feedback was given on the accuracy of the results. There were also those that asked if the Mozilla Googlebot (known as “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)” in your visitor logs) and Big Daddy were related, but no response was made.

Now I’m going to begin some of my own speculation:

I do in fact believe the two are related. In fact, I think this new crawler will eventually replace the old crawlers just as Big Daddy will replace the current data infrastructure.

Why is this important?

Based on my observations, this crawler may be able to do so much more than the old crawler.

For one, it emulates a newer browser. The old bot was based on the Lynx text based browser. While I’m sure Google added features as time went on, the basic Lynx browser is just that – basic.

Which explains why Google couldn’t deal with things like JavaScript, CSS and Flash.

However, with the new spider, built on the Mozilla engine, there are so many possibilities.

Just look at what your Mozilla or Firefox browser can do itself – render CSS, read and execute JavaScript and other scripting languages, even emulate other browsers.

But that’s not all.

I’ve talked to a few of my clients and their sites are getting hammered by this new spider. It has gotten so bad that some of their servers have gone down because of the volume of traffic from this one spider!

On the plus side, I have clients who went from a few hundred thousand indexed pages to over 10 million in just a few weeks! Literally since December, 2005 there’s been a 3500% increase in indexed pages over an 8 week period! Just so you know, this is also the client’s site that went down because of the huge volume of crawling happening.

But that’s still not all.

I have another client which uses IP recognition to serve content based on a person’s geographic location. If you live in the US you get American content and pricing; if you live in the UK you get UK content and pricing. As you may imagine, the UK, US, Canadian and Australian content is all very similar. In fact about the only thing noticeably different is the pricing aspect.

This is my concern – if the duplicate content gets indexed by Google what will they do? There’s a good chance that the site would be penalized or even banned for violation of the webmaster quality guidelines set forth by Google.

This is why we implemented IP recognition – so that Googlebot, which crawls from US IP addresses only sees one version of the site.

However, a review of the server logs shows that this new Googlebot has been visiting not only the US content but also the content of the other sections of the site. Naturally, I wanted to verify that the IP recognition was working. It is. This leads me to wonder then; can this browser spoof its location and/or use a proxy?

Imagine that – the browser is smart enough to do some of its own testing by viewing the site from multiple IP addresses. If that’s the case then those who cloak sites are going to have problems.

In any case, from the limited observations I’ve made, this new Google – both the data center and the spider – are going to change the way we do things.

If you have experienced anything similar in the past few months to do with Google, be sure to add it to our comments section below.

Add to | DiggThis | Yahoo! My Web

Technorati:

Rob Sullivan is a SEO Consultant and Writer for http://www.textlinkbrokers.com. Textlinkbrokers is the trusted leader in building long term rankings through safe and effective link building. Please provide a link directly to Textlinkbrokers when syndicating this article.

Initial Research

On Saturday, March 11, 2006, I received a somewhat urgent telephone call from a client of mine, Hibiscus Florals. The owner, Mark Morkowski, was concerned because he had been reviewing his website traffic statistics and had noticed that at numerous points throughout the day, a user or spider from “ASSERTIVENET” (IP 66.154.103.125) had visited the Hibiscus website.

Since this was rather unusual, Mark elected to investigate further by searching for more information “Assertivenet” via the Google search engine. The first three results that he found appear below:

• http://www.webmasterworld.com/forum10/11080.htm
• http://www.powerbasic.com/support/forums/Forum12/HTML/002979.html
• http://www.completewhois.com/cgi-bin/whois.cgi?query=66.154.111.40

From this information, Mark and I gathered that the owner of the spider in question appears to be a company called Assertive Networks, and hosted through a company called “BC Hosting.” More information wass not immediately available.

It is this lack of information that likely led some of the members of the PowerBASIC forums to block the IP range 66.154.* from accessing their various websites, and justifiably so. But this same lack of information led to additional questions:

1. What files was the Assertivenet spider accessing/trying to access? Was the spider crawling pages or, like some bots, was it looking for specific files that could be used for malicious purposes (e.g. files and scripts that could be manipulated for website attacks?)

2. Why is the apparent owner of the Assertivenet spider a web hosting company (BC Hosting)?

3. What is the intended purpose of the Assertivenet spider?

Additional Research – All Is Not As It Appears

At this point, I decided to look beyond what the website traffic statistics revealed, as well as the information that Mark’s initial search revealed. I needed to start by answering the questions I posed earlier, and in
order to do so, I needed to access the raw log files for the Hibiscus website.

I opened up the log files, searched for the particular IPs in question, and found a series of entries such as these:

2006-03-11 03:47:34 66.154.103.125 - 216.89.218.168 80 GET /robots.txt - 200 0 400 285 78 HTTP/1.0 www.hibiscusflorals.com Gigabot/2.0/gigablast.com/spider.html -
2006-03-11 03:47:34 66.154.103.119 - 216.89.218.168 80 GET /larger_image.asp PID=215 200 0 0 299 125 HTTP/1.0 www.hibiscusflorals.com Gigabot/2.0/gigablast.com/spider.html -
2006-03-11 03:50:37 66.154.103.119 - 216.89.218.168 80 GET /larger_image.asp PID=195 200 0 0 299 31 HTTP/1.0 www.hibiscusflorals.com Gigabot/2.0/gigablast.com/spider.html -
2006-03-11 07:47:05 66.154.103.125 - 216.89.218.168 80 GET /robots.txt - 200 0 400 285 78 HTTP/1.0 www.hibiscusflorals.com Gigabot/2.0/gigablast.com/spider.html -
2006-03-11 07:47:05 66.154.103.119 - 216.89.218.168 80 GET /larger_image.asp PID=219 200 0 0 299 109 HTTP/1.0 www.hibiscusflorals.com Gigabot/2.0/gigablast.com/spider.html -

The spider in this case actually belongs to a search engine called Gigablast, and is appropriately named the Gigabot. The Gigabot only crawled pages and files as other search engines have, and made no attempts whatsoever to access files and scripts of a known malicious nature.

Gigablast is a “Tier 2″ search engine that has over 1,000,000,000 pages indexed as of the date of this article (March 13, 2006.) While it is not on the same level in terms of popularity as the Big 3 of Yahoo!, MSN, and Google, it has indexed a significantly large portion of the web, and can be useful for some searches. In particular, Gigablast has implemented a “Giga bits” feature whereby alternate searches are suggested based on the user’s original query in order to help narrow the query down and provide greater relevancy.

I conducted additional research and discovered that some IP addresses from the 66.154.* IP block do resolve to gigablast.com e.g.:

Conclusion – The Gigabot is Safe

As you may well have gathered by now, the Gigabot is a perfectly safe spider that acts and operates in the same manner as other search engine spiders operate. There is no reason at this time to block the 66.154.* IP range that the bot uses; if anything, webmasters would gain from the potential free traffic that Gigablast would generate for their websites as the result of the Gigabot’s efforts.

*Previously published at www.searchenginefriendlylayouts.com

Add to | DiggThis | Yahoo! My Web

Technorati:

Adam Senour is the owner of ADAM Web Design, a leading web design and development company in Toronto, Ontario, Canada. Visit http://www.adamwebdesign.ca for more information on ADAM Web Design products and services.

Makes sense. It’s the type of on-topic, informative material that the original investors in the original highly granular directory thought would find an audience in the first place.

The next logical step would be to partner with Yahoo, Google, Quigo, etc. to actually serve (at least some of) the ads on those pages. LookSmart built a pretty robust platform for paid search, but the leaders in the field may well have larger advertiser networks and more importantly, advertisers willing and able to take the trouble to fund and maintain their contextual ad accounts.

I’d rather tweak my Google AdWords account to take account of (eg.) LookSmart’s new inventory than to have to open up a LookSmart account to get at it.

LookSmart Searches for Vertical Comeback

Andrew Goodman is Principal of Page Zero Media, a marketing consultancy which focuses on maximizing clients’ paid search marketing campaigns.

In 1999 Andrew co-founded Traffick.com, an acclaimed “guide to portals” which foresaw the rise of trends such as paid search and semantic analysis.