From mid-winter until this week, StepForth has strongly advised our clients to be conservative with any changes to their sites until enough time has passed for us, along with many others in the SEO community to observe, analyze and articulate our impressions of the upgrade. About ten days ago, the light at the end of the intellectual tunnel became eminently visible and SEO discussion forums are abuzz with productive and proactive conversations regarding how to deal with a post-Bigdaddy Google environment.

To make the long back-story short, in September 2005, Google began implementation of a three-part algorithm update that became known as the Jagger Update. Shortly after completing the algo update in late November, Google began an upgrading of their server and data storage network that was dubbed the Bigdaddy Infrastructure Upgrade. The Bigdaddy upgrade took several months to completely roll out across all of Google’s data centers, which are rumoured to number in the hundreds.

In other words, the world’s most popular search engine has, in one way or another, been in a constant state of flux since September. The only solid information SEOs had to pass on to curious clients amounted to time tested truisms about good content, links and site structure. Being the responsible sort we are, no good SEO wanted to say anything definite for fear of being downright wrong and misdirecting others.

Starting in the middle of May and increasing towards the end of the month, ideas and theories that had been thrown around SEO related forums and discussion groups started to solidify into the functional knowledge that makes up the intellectual inventory of good SEO firms.

Guided by the timely information leaks from Google’s quality control czar Matt Cutts , discussion in the SEO community surrounding Bigdaddy related issues has been led by members of forums WebMasterWorld , SEW ( 1 ) ( 2 ) ( 3 ), Threadwatch , SERoundtable ( 1 ) ( 2 ) ( 3 ), and Cre8asite ( 1 ) ( 2 ). SEO writers Aaron Wall , Jarrod Hunt , and Mark Daoust have also added their observations to the conversation in a number of separate articles.

By now, most good SEOs should be able to put their fingers on issues related to Bigdaddy fairly quickly and help work out a strategy for sites that were adversely affected by the upgrade. The first thing to note about the cumulative effects of Jagger and Bigdaddy is the intent of Google engineers to remove much of the poor quality or outright spammy commercial content that was clogging up their search results.

The intended targets of the Bigdaddy update go beyond sites that commit simple violations against Google’s webmaster guidelines to include affiliate sites, results gleaned from other search tools, duplicate content, poor quality sites and sites with obviously gamey link networks. In some cases, Google was targeting sites designed primarily to attract users to click on paid-search advertisements.

After the implementation of the algo update and infrastructure upgrades, SEOs have seen changes in the following areas: Site/Document Quality Scoring, Duplicate Content Filtering and Link Intention Analysis.

The first area noted is site or document quality scoring. Did you know there are now more web documents online than there are people on the planet? Many if not most of those documents are highly professional and some are sort of scrappy. While Google is not looking for perfection, it is trying to assess which pages are more useful than others and attention to quality design and content is one of the criteria.

Quality design simply means giving Google full access to all areas of the site webmasters want spidered. Smart site and directory structures tend to place spiderable information as high in the directory tree as possible. While Google is capable of spidering deeply into database sites, it appears to prefer to visit higher level directories much more frequently. We have noted that Google agent visits do tend to correspond with update times set via Google Sitemaps.

Accessibility and usability issues are thought to make up elements of the Jagger algorithm update, marking the way visitors use a site or document and the amount of time they spend engaged in a user-session associated with a site important factors in ranking and placement outcomes. Internal and outbound links should be placed with care in order to make navigating through and away from a site as easy as possible for site visitors.

Another quality design issue involves letting Google know which “version” of your site is the correct one. For most, a website can be access with or without typing the “www” part of the URL. (ie: http://www.stepforth.com , http://stepforth.com , http://www.stepforth.com/index.shtml ) This presents a rather funny problem for Google. Because links directed into a site might vary in the way they are written, sometimes it doesn’t know which “version” of a site is the correct one to keep in its cache. To Google, each of the variations of the URL above could be perceived as unique websites, an issue known as “canonicalization”, a subject Matt Cutts addressed on his blog in early January.

‘Suppose you want your default URL to be http://www.example.com . You can make your webserver so that if someone requests http://example.com/ , it does a 301 (permanent) redirect to http://www.example.com/ . That helps Google know which url you prefer to be canonical. Adding a 301 redirect can be an especially good idea if your site changes often (e.g. dynamic content, a blog, etc.).”

Quality content is a bit harder to manage and a lot harder to define. Content is a word used to describe the stuff in or on a web document and could include everything from text and images, Flash files, audio or video and links.

There are two basic rules in regards to content. It should be there to inform and assist the site user’s experience and it should be, (in as much as possible), original.

Making an easy to use site that provides visitors with the information they are looking for is the responsibility of webmasters but there are a few simple ways to show you are serious about its presentation.

Focus on your topic and stick to it. Many of the sites and documents that have found themselves demoted or de-listed during the Bigdaddy upgrade were sites that delved across several topics at the same time without presenting a clear theme. Given the option between documents with clear themes and documents without clear themes, Google’s choice is obvious.

Google is working to weed out duplicate content. Google appears to be looking for incidents of duplicate content in order to try to promote the originator of that content over the replications. This has hit sites in the vertical search sector; affiliate marketing sector, real estate sites, and even retail sites that carry brand name products, especially hard. Several shopping or product focused database sites have seen hundreds or even thousands of pages falling out of Google’s main index.

In many cases, there is little or nothing to do for this except to start writing unique content for products listed in the databases of such sites. Many real estate sites, for example, use the same local information sources as their competitors do and all tend to draw content from the same selection of MLS listings. It’s not that Google thinks this content is “useless”, it’s that Google already has several other examples of the same content and is not interested in displaying duplicate listings.

Many of the listings previously enjoyed by large database driven sites have fallen into a Google index known as the supplemental listings database. Supplemental listings are introduced to the general listings shown to Google users when there are no better examples to choose from to meet the users search query. This is the same index that is often referred to as the “Google Sandbox”.

The last major element noted in the discussions surrounding Bigdaddy is how much more robust Google’s link analysis has become. Aside from site quality and duplicate content issues, most webmasters will find answers to riddles posed by Bigdaddy in their network of links.

In order to ferret out the intent of webmasters, Google has increased the importance of links, both inbound and outbound. Before the updates, an overused tactic for strong placement at Google saw webmasters trying to bulk up on incoming links from where ever they could. This practice saw the rise of link farms, link exchanges and poorly planned reciprocal link networks.

One of the ways Google tries to judge the intent of webmasters is by mapping the network of incoming and outgoing links associated with a domain. Links, according to the gospel of Google, should exist only as navigation or information references that benefit the site visitor. Google examines the outbound links from a page or document and compares them against its list of inbound links, checking to see how many match up and how many are directed towards, and/or coming from pages featuring unrelated or irrelevant content. Links pointing to or from irrelevant content or reciprocal links between topically unrelated sites are easily spotted and their value to the overall site ranking downgraded or even eliminated.

The subject of links brings up an uncomfortable flaw in Google’s inbound link analysis that is being referred to as Google Bowling. As part of its scan of the network of links associated with a document or URL, Google keeps a detailed record of who links to who, how long the link has been established, if there is a recip link back, along with several other items.

One of those items appears to be an examination of how and why webmasters might purchase links from another site. While bought-links are not technically a rankings killer, a bulk of such links purchased from un-relevant sites in a short period of time, can effectively destroy a site or document’s current or potential rankings. In an article published at WebProNews last autumn, Michael Perdone from e-TrafficJams speculates on the issue.

Google has tried to deal with the predatory practice of “Google Bowling” by considering the behaviour of webmasters whose site have seen a number of inbound links from “bad neighbourhoods” suddenly appear. If a site that has incoming links from bad places also has or creates links directed out-bound to bad places, the incoming links are judged more harshly. If, on the other hand, a website has a sudden influx of bad-neighbourhood links but does not contain outbound-links directed to bad places, the inbound ones might not be judged as harshly.

The combination of the Bigdaddy upgrade and the Jagger algorithm update have made Google a better search engine and are precursors to the integration of video content and other information pulled from other Google services such as Google Base, Google Maps and Google Groups in the general search results.

Before the completion of both, Google’s search results were increasingly displaying a number of poor quality results stemming from a legion of scraped content “splog” sites and phoney directories that had sprung up in efforts to exploit the AdSense payment system. Bigdaddy and Jagger is a combined effort to offer improved, more accurate rankings while at the same time, expanding Google’s ability to draw and distribute content across its multiple arms and networks.

Moving forward, that is what users should expect Google to do. Google is no longer a set of static results updated on a timed schedule. It is constantly updating and rethinking its rankings, especially in light of the number of people trying to use those rankings for their own commercial gain.

The effect of the duo upgrades seems to be settling out. Credible, informative sites should have nothing to worry about in the post-Bigdaddy environment. As Google is trying to move into the most mainstream areas of modern marketing, credibility is its chief concern. The greatest threat to Google’s dominance does not come from other tech firms. It comes from the users themselves who, if displeased with results being shown by Google, could migrate en masse to another search engine.

|

Add to | DiggThis | Yahoo My Web

Jim Hedger is the SEO Manager of StepForth Search Engine Placement Inc. Based in Victoria, BC, Canada, StepForth is the result of the consolidation of BraveArt Website Management, Promotion Experts, and Phoenix Creative Works, and has provided professional search engine placement and management services since 1997. http://www.stepforth.com/ Tel – 250-385-1190 Toll Free – 877-385-5526 Fax – 250-385-1198

Reference to their sites, generally to the index pages and a seemingly random selection of internal pages existed in Google listings but pages that once drove sizable amounts of traffic appeared to vanish into the ether. As February rolled into March, more reports were posted to blogs and forums by frustrated webmasters who started to notice the number of pages from their sites had declined, significantly, in Google’s index.

Many SEO firms, including StepForth, received information requests and research projects from clients who wanted to know what had happened to their sites. In all cases, we did the best we could but, given the obvious complexity of the update and the lack of fresh information from Google, recommendations given during this period have more resembled shotgun style SEO advice than the finer laser focus most of us would normally prefer to offer our clients. As is the case with most major updates, investigation as often as not leads to more questions.

Matt Cutts, Google’s Search Quality Officer and #1 communicator, answered many of those questions yesterday in an open and wide ranging post titled, ” Indexing Timeline “.

The post outlines how Google staff have examined and responded to webmasters’ queries and complaints stemming from the Bigdaddy update. It also addresses a number of issues webmasters who have seen sections of their pages disappear from the SERPs including the quality of both in-bound and out-bound links, irrelevant reciprocal linking schemes, and duplicate text found on vertical reference and affiliate sites.

According to his timeline, on March 13, Googleguy asked webmasters to offer example sites for Google’s analysis in a post at WebmasterWorld . Commenting on the sites offered up for examination Cutts wrote,

“After looking at the example sites, I could tell the issue in a few minutes. The sites that fit “no pages in Bigdaddy” criteria were sites where our algorithms had very low trust in the inlinks or the outlinks of that site. Examples that might cause that include excessive reciprocal links, linking to spammy neighborhoods on the web, or link buying/selling. The Bigdaddy update is independent of our supplemental results, so when Bigdaddy didn’t select pages from a site, that would expose more supplemental results for a site.”

That quote covers a lot of ground but it explains a great deal of Google’s post-Bigdaddy behaviour.

Google bases its ranking algorithm on trust. That might sound nave to the uninformed, but we are discussing one of the most informed electronic entities that has ever existed. Google also keeps historic records on every item contained in its index. Though it bases its opinions on a baseline of trust, those opinions are extremely well informed.

In order to remain continually informed, it spiders everything it can and sorts the data later. Google maintains a massive number of indexes including one known as the supplemental index . The supplemental index is a much larger representation of documents found on the web than those included in the main Google index.

” We’re able to place fewer restraints on sites that we crawl for this supplemental index than we do on sites that are crawled for our main index. For example, the number of parameters in a URL might exclude a site from being crawled for inclusion in our main index; however, it could still be crawled and added to our supplemental index.” (source: Google Help Center )

Many of the results that appeared to have disappeared are assumed to have been drawn from the supplemental results before the update. ” A supplemental result is just like a regular web result, except that it’s pulled from our supplemental index”.

As Cutts is quoted saying above, Bigdaddy results are separate from supplemental results. When a reference to a site is found in the main (Bigdaddy) results, Google does not necessarily dip into supplemental results as often as it might have previously.

Quality On, Quality In and Quality Out

Google has gotten better at judging the quality of content found on a document and within a site. Content includes text, images, titles, tags and both inbound and outbound links. Consistently said that well-built sites offering quality information and a positive user experience should perform well throughout its search indexes, Google provides a wealth of information via the Google Help Center and through its webmaster focused spokespersons, Cutts and Googleguy.

As Google has gotten better at determining the origin and history of content found in its various indexes, it tries to snip away at duplicate forms of on-site content, with the goal of listing the most trust worthy sites under any given user query in the main index.

Having been inundated over the years with multiple replications of what was already considered duplicate content. Google (and other search engines) has gotten very good at knowing if it has already indexed similar or duplicate content. Google is capable of examining text (including individual paragraphs), images and link networks (in and outbound links), looking for telltale signs of duplicate content.

If, for example, it perceives a site displaying product information pulled from the same product database that 25,000 other sites pull duplicate product information from, Google is not likely to rank that site well. Similarly, if it finds duplicate networks of reciprocal links shared among several pages in its index, it is not likely to assign a high trust value to that document.

Reciprocal linking strategies

“As these indexing changes have rolled out, we’ve improving how we handle reciprocal link exchanges and link buying/selling.”

Though Cutts points at reciprocal linking as an indicator to Google that there might be issues with a website’s credibility, that doesn’t automatically mean that all reciprocal links are going to cause problems for webmasters. Common sense and the value of delivering a quality user experience should dictate decisions around link strategies.

For example, if a professional landscaper provided links to plant nurseries in his or her region, and those nurseries in turn provided links to that landscaper, Google would likely consider those to be quality links. There is a direct relevance between the two sources of information. A network of links between local landscaping businesses, nurseries, horticultural institutes, permaculture initiatives, non-profit volunteer groups and a number of gardening centers, shared amongst a relevant set of websites would also likely be judged beneficial to Google users and not subject to supplemental penalization.

On the other hand, a network of obviously purchased links between anyone who will exchange links with each other, regardless of relevancy or direct user benefit is likely to trip any number of filters present in the Bigdaddy/Jagger upgrades.

Cutts provided an example of a simple error made by a real estate site. Along with a number of internal reference links to exotic properties displayed as a footer-style site-map, Cutts found several out-bound links with anchor text reading;
1-exersize-equiptment.com, Credit Cards, Quit Smoking Forum, Hair Care, and GoSearchFor.com. When he reset, he saw a similar set of links, only this time, the out-bound links were directed towards, mortgages sites, credit card sites, and exercise equipment. Cutts commented, “…if you were getting crawled more before and you’re trading a bunch of reciprocal links, don’t be surprised if the new crawler has different crawl priorities and doesn’t crawl as much. ”

Affiliate Text and Content

Cutts devoted a long paragraph covering affiliate text, mentioning a T-shirt site that once had about 100 pages indexed, a number recently reduced to only 5.

” The person said that every page has original content, but every link that I clicked was an affiliate link that went to the site that actually sold the T-shirts. And the snippet of text that I happened to grab was also taken from the site that actually sold the T-shirts. The site has a blog, which I’d normally recommend as a good way to get links, but every link on the blog is just an affiliate link. The first several posts didn’t even have any text, and when I found an entry that did, it was copied from somewhere else. So I don’t think that the drop in indexed pages for this domain necessarily points to an issue on Google’s side. The question I’d be asking is why anyone would choose your “favourites” site instead of going directly to the site that sells T-shirts?”

The Ghosts of minutes past

We live in the present. Our websites live in the past as well as the present. Google keeps tabs on all documents in its index and even if it has, “… spidered content that was posted only moments before,” it has an elephant’s memory for previous details and a computer’s ability to pull lots of information together to get a bigger picture of how all those details fit together.

Google works by following links. Google ranks by examining the quality of content found on a site and also on the sites that link into, or are linked to from, sites in its indexes. If you have seen a great deal of page content fall away from Google’s index, or if you are just generally interested in how Google is working, read Cutts’ Bigdaddy “ Indexing Timeline “.

Add to | DiggThis | Yahoo! My Web

Technorati:

Jim Hedger is the SEO Manager of StepForth Search Engine Placement Inc. Based in Victoria, BC, Canada, StepForth is the result of the consolidation of BraveArt Website Management, Promotion Experts, and Phoenix Creative Works, and has provided professional search engine placement and management services since 1997. http://www.stepforth.com/ Tel – 250-385-1190 Toll Free – 877-385-5526 Fax – 250-385-1198

Matt Cutts Discusses Cache Crawling

Cutts’ latest blog post reviewed Bigdaddy’s crawl-caching proxy in greater depth. He even provided helpful charts to illustrate the process.

As a webmaster, one may see numerous fetches from multiple Googlebots, each of them using some bandwidth while accomplishing their appointed rounds. It makes for a more accurate Google index, but the site impact has given some webmasters fits over the bandwidth usage.

The proxy used in the Bigdaddy infrastructure works like other proxies. It handles the effort of retrieving pages from websites, and fulfills requests from the various Google crawlers. Instead of multiple spiders hitting a website, they hit the cache instead.

Cutts breaks down the crawl caching in a summary during his post (spacing added; we like Matt, but we’d really like him to enjoy the Return key a bit more often :

The essential goal of the proxy, to reduce bandwidth, seems to have worked to Google’s satisfaction. Cutts wrote that “it was working so smoothly that I didn’t know it was live.”

—

Add to | DiggThis | Yahoo! My Web | PreFound.com

Bookmark WebProNews:

David Utter is a staff writer for WebProNews covering technology and business.

Matt Cutts Answers Reader Questions

…as Cutts responded to a question about his likelihood of traveling to the UK to visit or speak, and regretfully a trip across the Atlantic isn’t in his plans.

That means the Clan of Cutts will have to attend events like Boston Pubcon and SES San Jose to catch a glimpse of him. However, he does make himself available on his well-regarded blog, most recently to answer a passel of questions from commenters.

He reiterated that link sellers should use the “nofollow” attribute to mark links they sell. “Not doing so can affect your reputation in Google,” he wrote. One can imagine the sonorous intonation of James Earl Jones sounding out that response.

The Bigdaddy software infrastructure update has been completed and fully deployed. This means webmaster should see different visits from Google. Said Cutts, “You will probably see less crawling by the older Googlebot, which has a User-Agent of “Googlebot/2.1 (+http://www.google.com/bot.html)”. I believe crawling from the Bigdaddy infrastructure has a new User-Agent, which is “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”.”

But forget about seeing the RK parameter ever again. Cutts noted that the parameter used to be visible in a Google toolbar query. “I wouldn’t expect to see the RK parameter have a non-zero value again,” he wrote.

One commenter questioned Cutts about the 64.233.185.104 datacenter, saying it seemed to work differently from the others and asked if it mainly consisted of newly spidered data. “That wouldn’t surprise me,” said Cutts. “As Bigdaddy cools down, that frees us up to do new/other things.”

Although many follow Google’s most publicly known engineer and his writings, few may realized Cutts has an Emmy. Given that the Emmy in question is of the feline persuasion, it’s more likely Emmy has a Matt Cutts:

The question and answer thread offers some interesting insights into Google and webmaster issues. It is definitely worth a few minutes of the reader’s time today.

—

Add to | DiggThis | Yahoo! My Web

Drag this to your Bookmarks.

David Utter is a staff writer for WebProNews covering technology and business.

Rand has posted some insightful take aways including Matt’s thoughts on Sandbox, BigDaddy, linkbait and that v7ndotcom malarkey.

If you need more Matt Cutts, then listen to this previous interview, also at WebmasterRadio.fm and of course, you would do well to visit his blog, where he provided an update on Bigdaddy recently.

Bigdaddy is the rollout of some signigicant infrastructure changes at Google and to see Bigdaddy search results compared to current results, try this. Although Matt says, “I’d expect a new data center to be converted to Bigdaddy roughly every 10 days”, so a normal search on Google.com will likely return Bigdaddy style results.

Lee Odden is President and Founder of
TopRank Online Marketing, specializing in organic SEO, blog
marketing and online public relations. He’s been cited as a search
marketing expert by publications including U.S. News & World Report and
The Economist and has implemented successful search marketing programs
with top BtoB companies of all sizes. Odden shares his marketing
expertise at Online Marketing Blog offering
daily news, interviews and best practices.