Known as MediaBot, this spider only travels across sites involved displaying Google AdWords (through the AdSense program).

The SEO community is discussing the implications of MediaBot with the common assumption being MediaBot brings increased spider activity and that increased activity will have an effect on the general search engine rankings.

While we are not yet prepared to suggest all our clients sign up for AdSense, we do see a fair degree of logic in the assumption.

The idea that the more often a site is spidered the better it will (or should) rank in the general SERPs is sound.

Frequent spidering of documents is noted in the Historic Profile Google keeps on all indexed documents and is thought to be judged as a positive indicator of relevant site content, thus increasing the ranking potential of a site.

We know that the use of Google Analytics promotes a slightly higher degree of spider activity. Those running Google Analytics on a site displaying AdSense advertising can expect even more spider activity.

Matt Cutts acknowledges the existence of Mediabot but claims it is only for load balancing. From that, I extrapolate that Google is separating and specifying the roles each of its spiders play in the document indexing cycle.

Previously, there were different versions of Googlebot agents associated with Mozilla. They performed a series of similar roles and appear to have behaved generally the same. This one does not have a Mozilla associated name and appears only to target sites partnered with Google via AdSense.

Several people in the SEO industry are working to verify if Google is giving more attention to sites running AdSense. More information should be available by next week.

In the meantime, if you are running AdSense, check your logs going back a few months. Count the number of times a Google agent (aside from MediaBot) showed up at your site and try to figure an average per week or per month.

Now, start tracking new log files, keeping a running record of how often MediaBot shows up, along with the usual Google/Mozilla agents.

If you perceive any discrepancies between average number of visits, there are a number of people who would be interested in your stats, myself included.

Jon Revill from MarketSmart Interactive (publisher of Search Engine Lowdown) is also conducting a detailed study of MediaBot and its effect on Google spider cycles.

Add to | DiggThis | Yahoo! My Web

Technorati:

Jim Hedger is the SEO Manager of StepForth Search Engine Placement Inc. Based in Victoria, BC, Canada, StepForth is the result of the consolidation of BraveArt Website Management, Promotion Experts, and Phoenix Creative Works, and has provided professional search engine placement and management services since 1997. http://www.stepforth.com/ Tel – 250-385-1190 Toll Free – 877-385-5526 Fax – 250-385-1198

WebGuerilla’s Greg Boser learned this lesson the hard way when setting up 301 redirects for the Googlebot crawler. Boser says he had neglected to redirect the mediabot, however. The end result: Google indexed and cached both the old and the new URL’s with identical content.

“The interesting thing to note about this page,” writes Boser, “is that the post was originally made in January. And for quite sometime it had a cached page that was a representation of what Googlebot was given. But then the Mediapartner bot visited on April 7th. And the page it was served on that date ended up replacing the Googlebot version in the cache.”

The ShoeMoney blog relays that Cutts confirmed the news:

“Matt said it is a bandwidth saving feature to have GoogleBot and MediaBot both contributing to big daddy. Matt also stated that you will gain zero advantage in search listings however if you are serving different content to MediaBot then [sic] to Googlebot then you could be in trouble.”

Besides duplicate content issues and past speculative issues that AdSense clients could enjoy a certain favorable treatment if indexed by the mediabot, it is unclear the full implications of this change.

“It will be interesting to see if other consequences arise for webmasters, such as excluding pages for googlebot via robots.txt that end up being indexed via the mediabot,” writes Jennifer Slegg.

But until now, it has been Google’s position that the mediabot and the Googlebot were entirely separate technologies with separate purposes, citing a desire to support freedom of expression without bias towards AdSense clients.

From the AdSense Help Center:

“Adding the Google AdSense ad code or AdSense for search code to your site will not queue your pages for crawling by our main index bots. While our bot (starting with ‘Mediapartners-Google’) does crawl content pages for the purpose of targeting ads, participation in AdSense does not increase the number of pages from a site in our main index.”

Tag: |

Drag this to your Bookmarks.

Add to DiggThis Yahoo My Web