There is a little confusion about Delicious and its handling of visiting robots. Search Engine Journal cited a blogger who claimed Delicious blocks bots from the big search engines:

Colin Cochrane found this out the other day, saying that ‘This isn’t a simple robots.txt exclusion, but rather a 404 response that is now being served based on the requesting User-Agent.’

Look a little closer at the robots.txt file, and you see something different happening. The bots from the four big search sites have been disallowed from certain subdirectories at Delicious, and not the bookmarks. The top line of the robots.txt file is a broad go-away to all bots, but from what we can tell from the Robots.txt standard, the lines aimed as the four specific bots allow them to go anywhere on Delicious that has not been expressly disallowed to them.

Nothing to see here folks, move along, move along.

Regardless, Freakonomics blogger Ian Ayres reports that untraceable poker bots are on the rise and may run many online gamblers offline faster than the DOJ.

[T]he rise of gambling bots may soon depress online poker participation for a very different reason. In the very near future, online poker may become a suckers’ game that humans won’t have a chance to win. Bots are quite scale-able and it will be virtually impossible to prohibit computer or computer-assisted online playing.

Yikes. That’s bad news for poker-lovers. Worse news for online casinos. Looks like it’s back to catching cheaters the old-fashioned way – by checking their sleeves

In fact, most advertisers are still in the dark about quality score and how they can work with it rather than fight it or work around it. One aspect of the quality score algorithm that has had me frustrated from the get go is how can Google determine the landing page quality score of a page that you tell it it cannot access (by means of robots.txt exclusion) ? After speaking with our Google rep, we finally got our answer.

Are You Still Credible?

While Michael Gray gave a great explanation on working with the AdWords quality robot there were still some unanswered questions. We all know that some of the factors required to get a great quality score are creating credibility indicators like a privacy policy or a terms of service document. We also know that these documents have little value in the main search index and many people like to exclude these documents rather than wasting link authority and to be honest, we are among those people. However, our biggest concern was that if we excluded these documents from the main index to preserve our link equity, would the quality score bot remove whatever quality points it gave for these documents because it could not see them? Getting no answers from anyone, we decided to simply risk it for ourselves. We created a brand new P3P privacy policy for this very site and excluded it using our robots.txt file. Then we waited.

The Results ?

After over a month, our main index rankings have all increased (Although this is more than likely due to other factors). The point is that there was no noticeable decrease in any areas that we focus on in the organic results. It might still be a little early to tell (and I may eat these words) but it appears that this has more of a positive effect than negative.

What About AdWords?

This is where we expected to see the biggest change. We expected the Quality Score for the campaigns that we run for this site to suffer. We might as well use ourselves as guinea pigs to know for sure. After over a month our quality score has not changed at all. Our quality score for all campaigns is “great” across the board. It is important to note at this point that the privacy page was denied access by all robots. However, we noticed that despite this exclusion the AdWords bot (AdsBot-Google) was still visiting. After contacting our Google rep about this, we got the following reply:

In order to avoid increasing CPCs for advertisers who do not intend
to restrict AdWords visits to their pages, the system will ignore blanket
exclusions (User-agent: *) in ‘robots.txt’ files.

So basically, the AdWords bot does not obey the rules set out in your robots.txt unless you explicitly state that you didn’t want that particular bot snooping around. Fair enough, I actually think this is a good thing and I am sure it has already saved a lot of advertisers a lot of heartache. So in our small example, we have saved ourselves some link equity and still had the benefit of having the AdWords bot be (im)polite and reviewing our privacy page.
So The Quality Score Robot Still See You?

So far, this is the perfect setup and plays nice with the organic side of things. But that got us thinking. Is there really a need for a privacy policy? While the use of Google Analytics actually requires you to display a privacy policy, we thought we would tempt fate. We decided to see how powerful a credibility indicator the privacy policy actually was. Last week we added to following lines to our robots.txt file:

User-agent: AdsBot-Google
Disallow: /privacy-policy/
Disallow: /w3c/policy.html

Two days later, most of our campaigns keywords went to either “OK” or “POOR”. Immediately after discovering this we removed the lines from the robots.txt. The next day things returned to normal. While it is possible that there were other factors at play here (the test was far from lab tested) I think it is fair to assume that a privacy policy carries a significant weight when it comes to quality score calculation. We also performed this “test” by excluding ALL PPC landing pages with similar or worse results. Our Google rep explained:

We feel that a non-participating advertiser does detract from the user’s
search experience and from the overall quality of the AdWords programme.
While you can exclude your pages from review, this tells us little about
the quality and relevance of your page. Therefore, if you restrict AdWords
from visiting your pages, you will experience a drop in Quality Scores for
your related keywords. (This will cause higher minimum bid requirements
for any landing page for which you have restricted access.)

In Conclusion.

We got exactly what we wanted from this little test. We wanted to know if excluding pages from the main index would effect our quality scores. The answer is no. Everything is now back to normal, our quality scores are “Great” across the board and our organic rankings have far from suffered. I hope that has answered that question for all that have called and mailed in to ask.

Comments

Search engines discover content using their bots (or ‘crawlers’), and determine relevancy (and by extension ranking) using advanced algorithms.

Discovery

The golden rule of SEO is that search engines cant rank a page they don’t know about. This is what makes discovery is so important. The most natural way for a search engine to discover a new resource is by crawling a link pointing at that content. So to get any new resource crawled quickly you should get a few links from other sites that are crawled regularly. (The major search engines have a sitemap initiative, but remember that without a solitary link Google will not index your content regardless of sitemap).

Relevancy

Getting your page crawled is less than half the battle. Now comes the hard part – ranking well. The second factor that determines whether your page ranks well is relevancy. Relevancy is determined by search engine algorithms which decide the order to display results to searchers. A number of on-site and off-site factors are incorporated into the relevancy determination which I’ll look at in a moment. (Trust could be also be dropped into the mix here, but I’m assuming that away for the moment).

How can you guide the search engines?

Webmasters actually have the greatest say in signalling for both discovery and relevancy. I use the term signalling because that’s really what SEO is all about – sending the right signal to the search engines.

To explain more about signals I’m going to have a look at another of the Irish Blog Award nominee sites which availed of the free site review offer.

First Partners

I met Paul Browne at the Irish Blog Awards a few weeks back. Paul writes regularly on his technology-themed First Partners blog:

Back to relevancy

The page title is probably one of the most important on-page elements used by search engines to determine the relevancy of your web pages. By and large you should target 1-3 keyword phrases, and bear in mind that most searches are around 3 words in length.

In the case of Paul’s blog homepage I notice that he is using dynamic titles which include the title of the most recent post. This in my view is a mistake – the homepage page title is about as sacred as it gets, and you don’t want it changing every day or so. I think Paul should concentrate on the main focus of his blog, whatever niche that might be, and use that in his blog homepage title.

The canonical URL problem (again)

I’m probably beginning to sound like a broken record. The canonical URL problem is a condition where your site or page is accessible by typing either of the following into your browser:

www.mysite.com

or

mysite.com

(notice this second case drops the www)

If you can reach your page via either URL AND the URL in the address bar does not change your site is suffering from the canonical URL problem.

In Paul’s case his site is accessible via both the www and non-www URLs. To fix this problem you need to redirect one URL to the other with a 301 redirect.

Don’t use 302 redirects for your homepage

When checking Paul’s blog I noticed that the FirstPartners.net homepage had a Toolbar PR0. This is odd given that the blog has PageRank 5. Then I noticed that the root page is redirecting to firstpartners.ie/rp/:

http://www.firstpartners.net/

GET / HTTP/1.1
Host: www.firstpartners.net
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3
Accept: text/xml,application/xml,application/xhtml+xml,[... ]png,*/*;q=0.5
Accept-Language: en,en-us;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Cookie: __utma=67859462.28111[... ]__utmc=67859462

HTTP/1.x 302 Found
Date: Mon, 02 Apr 2007 16:35:35 GMT
Server: Apache/2.0.52 (Red Hat)
Location: http://www.firstpartners.net/rp/
Content-Length: 304
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=iso-8859-1
X-Pad: avoid browser bug

If the homepage is going to stay there then I suggest changing that to a 301 redirect. The most probable reason why the temporary homepage is currently PageRank 0 is that it has few if any backlinks. The backlinks Paul has accumulated point at www.firstpartners.net rather than www.firstpartners.net/rp/, and Google doesn’t realise that /rp/ is now the homepage. No 301 = No transferral of links and trust

And some advice for the blog?

I had a few ideas when I looked at Paul’s blog. I found that the page weight was a little too beefy, with the blog homepage weighing in at 800KB+ on one occasion last week. I also thought that Paul could cut the number of posts published per page to a more manageable number. And I even considered whether NOFOLLOWing some of the internal links (e.g. the cloud) might help.

But I can safely scrap all that advice for one simple suggestion: give each and every blog post a unique META description.

When I looked at all the pages in the supplemental index it was instantly apparent that Paul wasn’t using META descriptions:

You can see that Google is picking up boilerplate content for every snippet. I’d be willing to bet that at least some of the 265 pages in supplemental will pop out if they have a unique description META.

I did spend a short amount of time looking at the backlink profile for the blog and the majority of links use the anchor “Paul Browne – Technology in plain English”. I reckon Paul probably ranks well for his name (he had a thread about his on-line doppleganger but I couldn’t find it). I think some diversification of the link anchor could pay off – non-diverse backlink anchors may actually raise a flag that could damage your site.

So find that niche and push it in your titles and anchors. In Paul’s case that niche should be highly relevant to his company’s products and services. I’ll leave the idea generation to Paul.

So to recap my advice

1. Fix the blog homepage title
2. Sort the canonical URL
3. Change the root page 302 redirect
4. Assign unique META descriptions to each blog post

Comments

Tag:

I’ve never believed this old adage – I think it’s an arrogant dismissal by incompetent marketers who can’t manage their public relations properly. Until recently it has never really been a big concern in the online world, but with an explosion of online communities over the past few years and the more recent increase in link baiting and brand protection services, perhaps online marketing professionals should be taking their branding more seriously?

If I could be bothered hunting down “how to” link bait articles, you’d see that most recommend undertaking some kind of negative efforts in order to spark off some link love to your site. A war with another blogger. A campaign against a well known brand. Posting inaccurate information to tempt righteous bloggers to correct you. Etc. I’m sure most people who read this blog have already seen such articles, so no need to rehash them here.

Let’s consider some examples.

A while back, Brett Tabke banned all bots from WebmasterWorld. While this wasn’t intended link bait as such (it had a technical justification), it did serve as link bait for WmW. Did they get more links and coverage? Yes. Did they suffer any negative impact? No.

They didn’t lose any traffic (that they didn’t want to get rid of). They didn’t lose any users over the experiment. The site and business hasn’t suffered from it – in fact, a lot of people had a little more respect for Brett & WmW at the time for having the balls to ditch all search traffic as an experiment.

While this isn’t a great example (it wasn’t an intentional link bait exercise), it does show how positive link bait can lead to positive results.

On the other hand, 2006 saw countless cases of companies screwing up their branding online by handling their online PR badly. While not all of these are because of stupid link baiting, the end results are the same. Perhaps the companies have received more exposure because of it, but what is the long term impact of this?

Zipatoni launched a splog for the Sony PSP and ended up getting slated by pretty much everyone over it. What’s worse is that their response was grossly inadequate for the scale of their screw up and ended up compounding their failure. Now search for “Zipatoni” and you find a great deal of negative results in the top 20.

Another example from 2006 is Big Mouth Media’s ban from Google. While this is an embarrassing slip up in itself (which was rectified pretty quickly), their response to critics in the SEO community was arrogant and badly presented and made the whole affair much worse for them when Matt Cutts told the SEO world that they were in fact penalised from Google (and it wasn’t a mistake as BMM claimed). Again, search for “Big Mouth Media” and you find negative results (this from a company that offers online brand protection services).

In both these examples it was entirely within the control of the companies to manage these negative situations properly, but in both cases they failed to do so and it resulted in much more negative publicity.

Was any publicity, good publicity in these cases?

Well, there is the SEO argument that says “yes”. After all, they did get a lot of link love from various sources so perhaps it will help them in the long term in terms of improved rankings and traffic. Notice I haven’t linked out to them just now – I’m not a link scrooge – I just do see any point in rewarding stupidity. If I thought for a second that these were well planned link bait campaigns and not simple screw ups I’d probably have more respect for them!

However, what happens when clients carry out due diligence before they sign up for services provided by these marketing agencies? While a Google search might not be the first thing they do, it is fairly likely it will be part of the research. Will a client want the services of an agency that has caused an international uproar for a major brand client or an agency that have themselves been unable to assure the service they are offering?

The overall impact is debatable and obviously there are many other factors in the client decision making process, but the point is do we really want to throw some negative factors into the mix?

Link baiting is a short term exercise that can potentially have long term positive results, but I think that all too often people neglect to consider the long term negative results as well.

Credit to those who do link bait well – it can be a useful tool, just as SEO can be. But as with SEO, many take the basic concepts and run with them without looking too deeply into the strategic implications they can have on their site, business or brand.

Comments

Tag:

Add to Del.icio.us | Digg | Reddit | Furl

Bookmark WebProNews:

Scott Boyd (aka Marketing Guy) is an Edinburgh based online marketing consultant with over 6 years experience in the industry. He is the founder of SEO agency eFlaunt, where he mixes a
blend of traditional marketing and SEO.

Scott’s musings relating to the marketing and SEO industries can be found on his blog – Fused Nation.

This will enable marketers to create and distribute their own versions of AOL Instant Messenger. I am really excited about this marketing opportunity. The reason I am enthusiastic here is that the marketer gets to use AIM to solidify their relationship with the consumer on the desktop. However, I wouldn’t just slap your brand on AIM. I would layer in some value-added services, such as AIM bots and feed alerts.

Add to | DiggThis | Yahoo! My Web

Steve Rubel is a PR strategist with nearly 16 years of public relations, marketing, journalism and communications experience. He currently serves as a Senior Vice President with Edelman, the largest independent global PR firm.

He authors the Micro Persuasion weblog, which tracks how blogs and participatory journalism are changing the public relations practice.

Anthony Scott Clark is looking at 10 years in jail and a $250,000 fine should a judge impose the maximum penalties in his case. Clark pleaded guilty in federal court in San Jose to participating in a distributed denial of service (DDoS) attack against eBay according to a Department of Justice statement.

The attacks took place in July and August 2003. Clark and his associates used a worm to exploit a Windows RPC-DCOM vulnerability and gain control of some 20,000 PCs. This bot network was then used to attack eBay computers and caused a denial of service.

Federal authorities including the Secret Service’s Electronic Crimes Task Force investigated the attacks. “It’s not that frequently that you see people successfully prosecuted for participating in these attacks,” Christopher Sonderby, chief of the Computer Hacking and Intellectual Property unit in San Jose, told Mercury News.

A status hearing regarding sentencing has been set for April 3, 2006. That sentencing will depend on how much damage they determine was caused by Clark’s attacks.

—
Email the author here.

Add to | DiggThis | Yahoo My Web

David Utter is a staff writer for WebProNews covering technology and business.

It’s a touchy subject for many bloggers who see some automated content creation systems (particularly those that aggregate RSS feeds) as a threat to their own livelihood.

Richard looks at a number of Content Creation systems and ends up taking some middle ground, liking some systems and being very wary of others. It’s worth a read as this is a topic we’re going to have to grapple with more and more in future. In fact in the past two days I’ve had emails from 4 different software makers announcing these sorts of packages. In all cases I promptly deleted them as I could see that they were being pretty blatantly marketed to spambloggers.

As Richard notes in his post – systems like these can actually be used in positive and helpful ways (he gives examples) – the problem is that they can also be used to rip off the content of others and to clutter the web. I guess it comes down to the individuals using them to make a call on how they’ll approach their content creation.

What do you think about these systems?

Bots and Content Creation / Aggregation report

Reader Comments…

Darren Rowse is the founder of ProBlogger.net, a blog about the many ways of adding an income stream to blogs.

Darren owns and writes a variety of blogs including Digital Photography Blog and Camera Phone
Zone. He is also a co-founder of the Breaking News Blog Collective.

None of them work, of course, because my MT install isn’t quite what they think it is (they don’t know to type “jeremy” in the extra field). But it still takes up a bit of CPU effort to ask cgiwrap to fire up mt-comments.cgi and whatnot.

So I finally did what I’ve been meaning to do for a while now:

RewriteCond %{HTTP_REFERER} ^$

RewriteCond %{REQUEST_METHOD} ^POST$

RewriteRule ^/mt/mt-comments.cgi – [F]

And 20 seconds after I did an apachectl graceful it blocked an attempt.

Morons, I tell you.
See Also

Jeremy Zawodny is the author of the popular Jeremy Zawodny’s blog. Jeremy is part of the Yahoo search team and frequently posts in the Yahoo! Search blog as well.

Visit Jeremy’s blog: Jeremy Zawodny’s blog.

Discuss risky SEO techniques in WebProWorld.

Walking on Ice: Dangerous SEO Tatics…

EGOL in an SEOChat forum discussion said, “I am getting new pages in the index by making them one of the first links on one of my already indexed pages.”

He goes on to say that he’s not putting links to hundreds of pages up there – just the ones he considers the most important at the time. And he’s doing this only in similarly-themed sites so that the links don’t seem out of place.

“I have rotated them in and out of my top-of-page navigtation – and I don’t think that it stunk up my design or content – these are all similar-theme sites.”

The rotation allows him to get more than a few pages crawled from the already existing page.

Another poster in the thread suggested that those concerned with hiding these links to maintain the sites current layout could try hiding your links in a noscript tag.

Scott Harris, our head designer, suggested a more efficient, and definitely risky, method of hiding your links using CSS. Don’t try this on your main site – you could kill your business. If you’re going to use high risk techniques try them on a test site to see what happens.

His first example was simply to set the visibility of your links to “hidden.”

Here’s the code as he sent it to me:
Risky Technique

That link would only show up in the html and not on the site itself.

As Yahoo could possibly check links for the phrase “visibility: hidden,” you might want to make an external style sheet and name your tags so that your intentions aren’t evident to the spider.

Your hidden link would appear to the spider this way:
<a href=”http://www.link.com” class=”risky”>Risky Technique</a>

The search engine won’t know that “risky” means:

.risky {
position: absolute;
visibility: hidden;
z-index: -1;
}

If this technique works in Yahoo it would, in all likelihood, work in Google as well.

While this is probably an ancient high risk concept to some SEOs, I’ve not read much of this technique for hiding links thus far, and therefore don’t know if it’s currently an effective high risk technique or not. If you’re considering using this or any high risk optimization method be sure to test it out on a sandbox site first. And remember – you’re optimizing at your own risk.

If you notice that your competitors have links and text in their source code that doesn’t appear on their site you could look for the .css link (do a ctrl + f and type in “.css”). In that style sheet do a ctrl + f and type in “visibility: hidden” to see what stands for hidden text in their source code. And brush up on your CSS here because there’s probably more ways than that to hide text and links.

This may help you uncover some hard evidence of high-risk techniques you can take to the search engine officiators and get your competitor’s banned.

Garrett French is the editor of iEntry’s eBusiness channel. You can talk to him directly at WebProWorld, the eBusiness Community Forum.