Limited Run’s story is a reminder that bot traffic is growing across the Web. In fact, they may one day pose a threat to Internet commerce and advertising that relies of real human interaction. A new infographic from the folks at Solve Media sheds some light on the bot problem.

It’s revealed that the majority of bots come from the United States, but its Southeast Asia and the Middle East that have the highest number of bot traffic. Solve Media also found that bot traffic grew 26 percent across the Web since 2011. Now bot traffic takes up 10 percent of all traffic on the Web.

If the number of bots, including those that click in ads, it could lead to marketers and advertisers wasting anywhere from $612 million to $4.7 billion on ads that won’t reach a human consumer. So what’s a marketer to do to keep themselves from wasting money?

Solve Media suggests that marketers seek out publishers that take a hard stance against bots. Facebook has recently done this by deleting millions of fake accounts, which led to many brands have their total number of likes decreased by a sizable margin. They also suggest that firms implement tracking technology that will tell them if click traffic is coming from human or bot sources.

So, with thoughts of bots and inactive human users rattling around your brain, it looks like Twitter has finally crossed the 500 million user mark.

That’s the news according to analyst group Semiocast, who says that Twitter crossed the half a billion user threshold back in June. They also say that 140 million of those users are from the United States.

By comparison, Facebook just reported in their Q2 earnings that they now have 955 million “active” users. That’s a monthly figure, as the daily active users tally is around 552 million.

Back in September of 2011, Twitter CEO Dick Costolo said that there were 100 million active Twitter users. And in March of this year, as Twitter celebrated their 6th birthday, they announced that there were 140 million active users. If we do a little math and assume that the growth rate stayed roughly the same, we can assume that there would be about 160 million active Twitter users by now.

So, the discrepancy between this figure and the 500 million figure is all about “users” vs. “accounts.” If Semiocast found over 500 million accounts and Twitter’s growth rate in active users hasn’t skyrocketed since March of 2012, that means that there are a whole hell of a lot of worthless and bot Twitter accounts out there. A pretty hefty majority, actually.

Other fun facts from the Semiocast study:

• Japan, while growth is slowing, still remains the second most Twitter-happy countries.  They post 10.6% of all public tweets, worldwide.
• People only tag tweets with a location 0.77% of the time
• Arabic usage is growing on Twitter, as it’s now the 6th most popular language on the site.  I’m sure this has something to do with Twitter launching in right-to-left languages (including Arabic) back in March.

But sometimes you’re up in the air about whether an account is automated or being operated by a real life human being. Some journalism students at The New School in New York have developed a test that will tell you who’s really behind that Twitter handle.

The Bot or Not test is the culmination of a semester-long project that had students analyzing thousands of articles and how they were linked and retweeted to the Twitterverse. They started out by tracking articles from three sites – Mashable, Read Write Web, and Tech Crunch.

In order to determine the botfestation of the web, we focused on three popular tech media sites: Mashable, Read Write Web, and TechCrunch. Once a story was published by one of these sites, we tracked the number of times people shared it on Twitter within the first two and half hours after being published.

We tracked 159 stories, which together were tweeted over 79,000 times by more than 18,000 distinct twitter users. We then built a multi-stage test to determine whether a user is a bot, or not. Next, we applied the test to our data to find the percentage of users that are bots.

What they determined was that 15% of the accounts that they tracked were bots and an additional 16% were labeled “probably bots.” Only 33% passed the test and were labeled human (36% were “probably human”).

To create the test, we gathered various criteria that demonstrated either very human or very bot-like behavior on a high-low scale. Tweeting only links to Mashable stories, for instance, is considered very bot-like behavior

And now, you can enter any Twitter handle into the bot test and it will give you its best determination based on a few signifying factors. For instance, here’s what it said about me:

It looks like it penalized me for posting a lot of links – something I admit to doing. But since I do reply to other users and because I don’t follow a suspiciously large number of people – I got the probably human nod.

Here’s how another account was treated:

Twitter bots are tough to battle, and they seem to pop up despite efforts to keep them at bay. But this nifty new tool will at least give you a little bit of information on any suspicions you may have,

[Via Poynter]

There is a little confusion about Delicious and its handling of visiting robots. Search Engine Journal cited a blogger who claimed Delicious blocks bots from the big search engines:

Colin Cochrane found this out the other day, saying that ‘This isn’t a simple robots.txt exclusion, but rather a 404 response that is now being served based on the requesting User-Agent.’

Look a little closer at the robots.txt file, and you see something different happening. The bots from the four big search sites have been disallowed from certain subdirectories at Delicious, and not the bookmarks. The top line of the robots.txt file is a broad go-away to all bots, but from what we can tell from the Robots.txt standard, the lines aimed as the four specific bots allow them to go anywhere on Delicious that has not been expressly disallowed to them.

Nothing to see here folks, move along, move along.

Regardless, Freakonomics blogger Ian Ayres reports that untraceable poker bots are on the rise and may run many online gamblers offline faster than the DOJ.

[T]he rise of gambling bots may soon depress online poker participation for a very different reason. In the very near future, online poker may become a suckers’ game that humans won’t have a chance to win. Bots are quite scale-able and it will be virtually impossible to prohibit computer or computer-assisted online playing.

Yikes. That’s bad news for poker-lovers. Worse news for online casinos. Looks like it’s back to catching cheaters the old-fashioned way – by checking their sleeves

In fact, most advertisers are still in the dark about quality score and how they can work with it rather than fight it or work around it. One aspect of the quality score algorithm that has had me frustrated from the get go is how can Google determine the landing page quality score of a page that you tell it it cannot access (by means of robots.txt exclusion) ? After speaking with our Google rep, we finally got our answer.

Are You Still Credible?

While Michael Gray gave a great explanation on working with the AdWords quality robot there were still some unanswered questions. We all know that some of the factors required to get a great quality score are creating credibility indicators like a privacy policy or a terms of service document. We also know that these documents have little value in the main search index and many people like to exclude these documents rather than wasting link authority and to be honest, we are among those people. However, our biggest concern was that if we excluded these documents from the main index to preserve our link equity, would the quality score bot remove whatever quality points it gave for these documents because it could not see them? Getting no answers from anyone, we decided to simply risk it for ourselves. We created a brand new P3P privacy policy for this very site and excluded it using our robots.txt file. Then we waited.

The Results ?

After over a month, our main index rankings have all increased (Although this is more than likely due to other factors). The point is that there was no noticeable decrease in any areas that we focus on in the organic results. It might still be a little early to tell (and I may eat these words) but it appears that this has more of a positive effect than negative.

What About AdWords?

This is where we expected to see the biggest change. We expected the Quality Score for the campaigns that we run for this site to suffer. We might as well use ourselves as guinea pigs to know for sure. After over a month our quality score has not changed at all. Our quality score for all campaigns is “great” across the board. It is important to note at this point that the privacy page was denied access by all robots. However, we noticed that despite this exclusion the AdWords bot (AdsBot-Google) was still visiting. After contacting our Google rep about this, we got the following reply:

In order to avoid increasing CPCs for advertisers who do not intend
to restrict AdWords visits to their pages, the system will ignore blanket
exclusions (User-agent: *) in ‘robots.txt’ files.

So basically, the AdWords bot does not obey the rules set out in your robots.txt unless you explicitly state that you didn’t want that particular bot snooping around. Fair enough, I actually think this is a good thing and I am sure it has already saved a lot of advertisers a lot of heartache. So in our small example, we have saved ourselves some link equity and still had the benefit of having the AdWords bot be (im)polite and reviewing our privacy page.
So The Quality Score Robot Still See You?

So far, this is the perfect setup and plays nice with the organic side of things. But that got us thinking. Is there really a need for a privacy policy? While the use of Google Analytics actually requires you to display a privacy policy, we thought we would tempt fate. We decided to see how powerful a credibility indicator the privacy policy actually was. Last week we added to following lines to our robots.txt file:

User-agent: AdsBot-Google
Disallow: /privacy-policy/
Disallow: /w3c/policy.html

Two days later, most of our campaigns keywords went to either “OK” or “POOR”. Immediately after discovering this we removed the lines from the robots.txt. The next day things returned to normal. While it is possible that there were other factors at play here (the test was far from lab tested) I think it is fair to assume that a privacy policy carries a significant weight when it comes to quality score calculation. We also performed this “test” by excluding ALL PPC landing pages with similar or worse results. Our Google rep explained:

We feel that a non-participating advertiser does detract from the user’s
search experience and from the overall quality of the AdWords programme.
While you can exclude your pages from review, this tells us little about
the quality and relevance of your page. Therefore, if you restrict AdWords
from visiting your pages, you will experience a drop in Quality Scores for
your related keywords. (This will cause higher minimum bid requirements
for any landing page for which you have restricted access.)

In Conclusion.

We got exactly what we wanted from this little test. We wanted to know if excluding pages from the main index would effect our quality scores. The answer is no. Everything is now back to normal, our quality scores are “Great” across the board and our organic rankings have far from suffered. I hope that has answered that question for all that have called and mailed in to ask.

Comments

Search engines discover content using their bots (or ‘crawlers’), and determine relevancy (and by extension ranking) using advanced algorithms.

Discovery

The golden rule of SEO is that search engines cant rank a page they don’t know about. This is what makes discovery is so important. The most natural way for a search engine to discover a new resource is by crawling a link pointing at that content. So to get any new resource crawled quickly you should get a few links from other sites that are crawled regularly. (The major search engines have a sitemap initiative, but remember that without a solitary link Google will not index your content regardless of sitemap).

Relevancy

Getting your page crawled is less than half the battle. Now comes the hard part – ranking well. The second factor that determines whether your page ranks well is relevancy. Relevancy is determined by search engine algorithms which decide the order to display results to searchers. A number of on-site and off-site factors are incorporated into the relevancy determination which I’ll look at in a moment. (Trust could be also be dropped into the mix here, but I’m assuming that away for the moment).

How can you guide the search engines?

Webmasters actually have the greatest say in signalling for both discovery and relevancy. I use the term signalling because that’s really what SEO is all about – sending the right signal to the search engines.

To explain more about signals I’m going to have a look at another of the Irish Blog Award nominee sites which availed of the free site review offer.

First Partners

I met Paul Browne at the Irish Blog Awards a few weeks back. Paul writes regularly on his technology-themed First Partners blog:

Back to relevancy

The page title is probably one of the most important on-page elements used by search engines to determine the relevancy of your web pages. By and large you should target 1-3 keyword phrases, and bear in mind that most searches are around 3 words in length.

In the case of Paul’s blog homepage I notice that he is using dynamic titles which include the title of the most recent post. This in my view is a mistake – the homepage page title is about as sacred as it gets, and you don’t want it changing every day or so. I think Paul should concentrate on the main focus of his blog, whatever niche that might be, and use that in his blog homepage title.

The canonical URL problem (again)

I’m probably beginning to sound like a broken record. The canonical URL problem is a condition where your site or page is accessible by typing either of the following into your browser:

www.mysite.com

or

mysite.com

(notice this second case drops the www)

If you can reach your page via either URL AND the URL in the address bar does not change your site is suffering from the canonical URL problem.

In Paul’s case his site is accessible via both the www and non-www URLs. To fix this problem you need to redirect one URL to the other with a 301 redirect.

Don’t use 302 redirects for your homepage

When checking Paul’s blog I noticed that the FirstPartners.net homepage had a Toolbar PR0. This is odd given that the blog has PageRank 5. Then I noticed that the root page is redirecting to firstpartners.ie/rp/:

http://www.firstpartners.net/

GET / HTTP/1.1
Host: www.firstpartners.net
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3
Accept: text/xml,application/xml,application/xhtml+xml,[... ]png,*/*;q=0.5
Accept-Language: en,en-us;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Cookie: __utma=67859462.28111[... ]__utmc=67859462

HTTP/1.x 302 Found
Date: Mon, 02 Apr 2007 16:35:35 GMT
Server: Apache/2.0.52 (Red Hat)
Location: http://www.firstpartners.net/rp/
Content-Length: 304
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=iso-8859-1
X-Pad: avoid browser bug

If the homepage is going to stay there then I suggest changing that to a 301 redirect. The most probable reason why the temporary homepage is currently PageRank 0 is that it has few if any backlinks. The backlinks Paul has accumulated point at www.firstpartners.net rather than www.firstpartners.net/rp/, and Google doesn’t realise that /rp/ is now the homepage. No 301 = No transferral of links and trust

And some advice for the blog?

I had a few ideas when I looked at Paul’s blog. I found that the page weight was a little too beefy, with the blog homepage weighing in at 800KB+ on one occasion last week. I also thought that Paul could cut the number of posts published per page to a more manageable number. And I even considered whether NOFOLLOWing some of the internal links (e.g. the cloud) might help.

But I can safely scrap all that advice for one simple suggestion: give each and every blog post a unique META description.

When I looked at all the pages in the supplemental index it was instantly apparent that Paul wasn’t using META descriptions:

You can see that Google is picking up boilerplate content for every snippet. I’d be willing to bet that at least some of the 265 pages in supplemental will pop out if they have a unique description META.

I did spend a short amount of time looking at the backlink profile for the blog and the majority of links use the anchor “Paul Browne – Technology in plain English”. I reckon Paul probably ranks well for his name (he had a thread about his on-line doppleganger but I couldn’t find it). I think some diversification of the link anchor could pay off – non-diverse backlink anchors may actually raise a flag that could damage your site.

So find that niche and push it in your titles and anchors. In Paul’s case that niche should be highly relevant to his company’s products and services. I’ll leave the idea generation to Paul.

So to recap my advice

1. Fix the blog homepage title
2. Sort the canonical URL
3. Change the root page 302 redirect
4. Assign unique META descriptions to each blog post

Comments

Tag:

I’ve never believed this old adage – I think it’s an arrogant dismissal by incompetent marketers who can’t manage their public relations properly. Until recently it has never really been a big concern in the online world, but with an explosion of online communities over the past few years and the more recent increase in link baiting and brand protection services, perhaps online marketing professionals should be taking their branding more seriously?

If I could be bothered hunting down “how to” link bait articles, you’d see that most recommend undertaking some kind of negative efforts in order to spark off some link love to your site. A war with another blogger. A campaign against a well known brand. Posting inaccurate information to tempt righteous bloggers to correct you. Etc. I’m sure most people who read this blog have already seen such articles, so no need to rehash them here.

Let’s consider some examples.

A while back, Brett Tabke banned all bots from WebmasterWorld. While this wasn’t intended link bait as such (it had a technical justification), it did serve as link bait for WmW. Did they get more links and coverage? Yes. Did they suffer any negative impact? No.

They didn’t lose any traffic (that they didn’t want to get rid of). They didn’t lose any users over the experiment. The site and business hasn’t suffered from it – in fact, a lot of people had a little more respect for Brett & WmW at the time for having the balls to ditch all search traffic as an experiment.

While this isn’t a great example (it wasn’t an intentional link bait exercise), it does show how positive link bait can lead to positive results.

On the other hand, 2006 saw countless cases of companies screwing up their branding online by handling their online PR badly. While not all of these are because of stupid link baiting, the end results are the same. Perhaps the companies have received more exposure because of it, but what is the long term impact of this?

Zipatoni launched a splog for the Sony PSP and ended up getting slated by pretty much everyone over it. What’s worse is that their response was grossly inadequate for the scale of their screw up and ended up compounding their failure. Now search for “Zipatoni” and you find a great deal of negative results in the top 20.

Another example from 2006 is Big Mouth Media’s ban from Google. While this is an embarrassing slip up in itself (which was rectified pretty quickly), their response to critics in the SEO community was arrogant and badly presented and made the whole affair much worse for them when Matt Cutts told the SEO world that they were in fact penalised from Google (and it wasn’t a mistake as BMM claimed). Again, search for “Big Mouth Media” and you find negative results (this from a company that offers online brand protection services).

In both these examples it was entirely within the control of the companies to manage these negative situations properly, but in both cases they failed to do so and it resulted in much more negative publicity.

Was any publicity, good publicity in these cases?

Well, there is the SEO argument that says “yes”. After all, they did get a lot of link love from various sources so perhaps it will help them in the long term in terms of improved rankings and traffic. Notice I haven’t linked out to them just now – I’m not a link scrooge – I just do see any point in rewarding stupidity. If I thought for a second that these were well planned link bait campaigns and not simple screw ups I’d probably have more respect for them!

However, what happens when clients carry out due diligence before they sign up for services provided by these marketing agencies? While a Google search might not be the first thing they do, it is fairly likely it will be part of the research. Will a client want the services of an agency that has caused an international uproar for a major brand client or an agency that have themselves been unable to assure the service they are offering?

The overall impact is debatable and obviously there are many other factors in the client decision making process, but the point is do we really want to throw some negative factors into the mix?

Link baiting is a short term exercise that can potentially have long term positive results, but I think that all too often people neglect to consider the long term negative results as well.

Credit to those who do link bait well – it can be a useful tool, just as SEO can be. But as with SEO, many take the basic concepts and run with them without looking too deeply into the strategic implications they can have on their site, business or brand.

Comments

Tag:

Add to Del.icio.us | Digg | Reddit | Furl

Bookmark WebProNews:

Scott Boyd (aka Marketing Guy) is an Edinburgh based online marketing consultant with over 6 years experience in the industry. He is the founder of SEO agency eFlaunt, where he mixes a
blend of traditional marketing and SEO.

Scott’s musings relating to the marketing and SEO industries can be found on his blog – Fused Nation.

This will enable marketers to create and distribute their own versions of AOL Instant Messenger. I am really excited about this marketing opportunity. The reason I am enthusiastic here is that the marketer gets to use AIM to solidify their relationship with the consumer on the desktop. However, I wouldn’t just slap your brand on AIM. I would layer in some value-added services, such as AIM bots and feed alerts.

Add to | DiggThis | Yahoo! My Web

Steve Rubel is a PR strategist with nearly 16 years of public relations, marketing, journalism and communications experience. He currently serves as a Senior Vice President with Edelman, the largest independent global PR firm.

He authors the Micro Persuasion weblog, which tracks how blogs and participatory journalism are changing the public relations practice.

Anthony Scott Clark is looking at 10 years in jail and a $250,000 fine should a judge impose the maximum penalties in his case. Clark pleaded guilty in federal court in San Jose to participating in a distributed denial of service (DDoS) attack against eBay according to a Department of Justice statement.

The attacks took place in July and August 2003. Clark and his associates used a worm to exploit a Windows RPC-DCOM vulnerability and gain control of some 20,000 PCs. This bot network was then used to attack eBay computers and caused a denial of service.

Federal authorities including the Secret Service’s Electronic Crimes Task Force investigated the attacks. “It’s not that frequently that you see people successfully prosecuted for participating in these attacks,” Christopher Sonderby, chief of the Computer Hacking and Intellectual Property unit in San Jose, told Mercury News.

A status hearing regarding sentencing has been set for April 3, 2006. That sentencing will depend on how much damage they determine was caused by Clark’s attacks.

—
Email the author here.

Add to | DiggThis | Yahoo My Web

David Utter is a staff writer for WebProNews covering technology and business.