“We are asking for examples, and may use data you submit to test and improve our algorithms,” the company says on a “Report Scraper Pages” form, found here.

Google’s head of web spam, Matt Cutts, tweeted about the new initiative:

This testing comes after months of iterations of Google’s Panda Update, designed to improve the quality of search results, though there has been no shortage of complaints about scrapers ranking over original content in that time.

The testing also follows a recent, big refresh of Google’s spam submission process, discussed here.

This past week, Google shared an interesting video, providing an inside look at the search algorithm tweaking process. While no earth shattering information was necessarily contained, it did provide a rare visual glimpse into the process. Watch it below.

I don’t think everyone would agree that they’ve all been improvements, as we see complaints about this every day, but Google makes a lot of changes aimed at improving search results, nevertheless.

“There are almost always a set of motivating searches, and these searches are not performing as well as we’d like,” says Engineering Director Scott Huffman. “Ranking engineers then come up with a hypothesis about what signal, what data could we integrate into our algorithm.”

The video does provide some unique behind the scenes footage of Google engineers plugging away on their computers, presumably working on the algorithms.

Google briefly talks about the process of raters. “These are external people that have been trained to judge whether one ranking is more relevant and higher quality than another,” says software engineer Mark Paskin.

“We show these raters a side-by-side for queries that the engineer’s experiment might be affecting,” explains Google Search Scientist Rajan Patel. “We also confirm these changes with live experiments on real users.”

“We do this in something called a sandbox. We send a very small fraction of actual Google traffic to the sandbox. We compute lots of different metrics,” says Paskin.

“In 2010, we ran over 20,000 different experiments. All the data from the human evaluation and the live experiment are then rolled out by a search analyst,” says Huffman.

Sangeeta Das, a quantitative analyst says, “For each project, it’s usually one analyst assigned from the moment that we’re talking to the engineers, trying to learn about their change.”

“We then have a launch decision meeting where the leadership of the search team then looks at that data and makes a decision,” says Huffman.

“Ultimately, the goal of the search eval analyst team is to provide informed, data-driven decision, and present an unbiased view” says Das.

“If our scientific testing says this is a good idea for Google users, we will launch it on Google,” says Google Fellow Amit Signhal.

The video then looks at the “did you mean” and “showing results for” features as an example.

Clearly there’s still room for humans on the web. In search, that’s good news for Blekko, which brings the old human-edited approach back into the mix of an industry that has largely been dominated by the algorithm for the last decade, though the jury’s still out on whether it will ever be as effective as Google.

In terms of content creation, we’ve already seen the beginnings of what the algorithm can do. Look at Demand Media’s business model (at least for the content portion) – it’s largely algorithm based, though it still uses humans to write and edit the content.

The future content farm may be a different story though. We’ve also seen the absence of human intervention in content creation. Look at what Narrative Science is doing. The company, run by a former DoubleClick executive, describes itself in the following manner:

“We tell the story behind the data. Our technology identifies trends and angles within large data sources and automatically creates compelling copy. We can build upon stories, providing deeper context around particular subjects over time. Every story is generated entirely from scratch and is always unique. Our technology can be applied to a broad range of content categories and we’re branching into new areas every day.” 

Look at what IBM has been able to accomplish through machine learning with its robot Watson. How long until a bunch of Watsons are creating content for the web (and creating other Watsons, for that matter)?

The good news is it might still be a while before robots replace us all. Back to the points made in the Harvard Business Review, by Eli Pariser, he notes that algorithms aren’t yet good at predicting future news, to the extent that humans are.

As far as risk-taking, “Chris Dixon, the co-founder of personalization site Hunch, calls this “‘he Chipotle problem,’ he writes. “As it turns out, if you are designing a where-to-eat recommendation algorithm, it’s hard to avoid sending most people to Chipotle most of the time. People like Chipotle, there are lots of them around, and while it never blows anyone’s mind, it’s a consistent three-to-four-star experience. Because of the way many personalization and recommendation algorithms are designed, they’ll tend to be conservative in this way — those five-star experiences are harder to predict, and they sometimes end up ones. Yet, of course, they’re the experiences we remember.”

Would you trust content created by algorithms or do you put your trust in humans?

Have you noticed recent changes in your ranking? Tell us about it.

Algorithm Change

Google makes changes to its algorithm all the time, but when a change comes with an announcement, you know people are going to talk. On Friday, Google announced a tweak designed to surface multiple pages from a single site for relevant queries.

"For queries that indicate a strong user interest in a particular domain, like [exhibitions at amnh], we’ll now show more results from the relevant site," says Google software engineer Samarth Keshava. "Prior to today’s change, only two results from www.amnh.org would have appeared for this query. Now, we determine that the user is likely interested in the Museum of Natural History’s website, so seven results from the amnh.org domain appear. Since the user is looking for exhibitions at the museum, it’s far more likely that they’ll find what they’re looking for, faster. The last few results for this query are from other sites, preserving some diversity in the results."

Not all webmasters have been thrilled with this. "Brace yourselves! Another Mayday disaster coming," one person commented on our story about it.

What do you think of this algorithm change? Comment here.

Experimenting

Just as the company frequently changes its algorithm, it also frequently experiments with different features, showing them to small sets of users before either turning them into full-fledged features or throwing them away. The jury’s still out on this one, but a new experiment has been spotted, which alters search results as you type your query.

Think of this like autosuggest taking over the entire SERP. The video demonstrates:

Again, this is only an experiment at this stage, and it may never make its way to the mainstream Google experience, but people are already expressing a great deal of concern about it (particularly with regards to queries that begin with words that could yield undesired NSFW results).

My guess is that Google would have ways around that issue, but it remains to be seen if users/webmaters will have to deal with it. If the feature does come to fruition, this is something SEOs are going to have to consider, as it could have a big impact on the habits of searchers. You may, for example, want to optimize more for the earlier words in a longer key phrase, in addition to the key phrase itself. But, we’ll see.

Should Google change search results as you type? Comment here.

Google Crawling Sites From Numerous IPs

Barry Schwartz at Search Engine Roundtable points to some discussion from SEOs in Webmasterworld, who have found for the first time that Googlebot is now crawling from several different IP addresses at the same time. One webmaster said, ". their fast activity notified me so I took a peek to see who was scraping the site… I’ve never seen Google spider so fast and from so many IP addresses, they were all valid Google ip’s but there was like 10-20 of them running at once."

Acquisition

The other day, it was officially announced that Like.com has been acquired by Google. Like.com is a shopping search company offering visual search technology and an automated cross-matching system for clothing and other merchandise.

At this point, it’s unclear what Google has planned for this technology, but it could very well affect search results for shopping queries, which means it could affect small business visibility for better or for worse. Shopping search is going to be an area of Google to keep an eye on.

Have you noticed anything else interesting happening with Google search within the last week or so? Are you seeing things that are impacting your rankings? Let us know.

"For queries that indicate a strong user interest in a particular domain, like [exhibitions at amnh], we’ll now show more results from the relevant site," says Google software engineer Samarth Keshava. "Prior to today’s change, only two results from www.amnh.org would have appeared for this query. Now, we determine that the user is likely interested in the Museum of Natural History’s website, so seven results from the amnh.org domain appear. Since the user is looking for exhibitions at the museum, it’s far more likely that they’ll find what they’re looking for, faster. The last few results for this query are from other sites, preserving some diversity in the results."

"We’re always reassessing our ranking and user interface, making hundreds of changes each year," adds Keshava. "We expect today’s improvement will help users find deeper results from a single site, while still providing diversity on the results page."

This change may prove to be helpful for a lot of searches, but they could still do more in this area if you ask me. For exmaple, they could rank tag pages (commonly used on news sites and blogs) as top results for appropriate searches.

For instance, if I search for "WebProNews SEO articles" or "SEO articles on WebProNews", it couldn’t get any more relvant than the WebProNews SEO tag page. That points to every SEO article we have. Yet, this result is nowhere to be found for either query, and it’s certainly not unique to us (ironically, another article in which I made a similar complaint ranks at the top). These pages aren’t often linked to, so they don’t get the PageRank, but does that make them less relevant for queries like this?

Bill Slawski at SEO By the Sea has some background on the patent, as well as a patent now filed by Demand Media. Peter Berger, CEO of Suite101, another company targeting the long tail with content, reached out to WebProNews to contribute his thoughts on the subject.

"The Google patent seems to be specific enough to leave enough room for a variety of approaches to identify underserved content areas on the Web, so even if Demand Media’s current algorithm was at risk, it should not be too hard for them to evolve it," he says. "Demand Media’s counter patent applications seem to be very broad – at superficial reading, a lot (if not all) of this will likely have to be classified as prior art as it describes methods that were in use well prior to the filing."

"The Google patent constitutes a very interesting opportunity if Google has learned its lesson that it works best as an algorithmically-driven enabler and provider, not a content publisher itself: if exploited with the web ecosystem in mind, e.g. as a feed to content publishers, it would open up the opportunity to retrieve better, actionable data from the leading search engine for our entire industry and serve the web much more useful information," he adds. "Looking at the level of investment several companies have made in this field, I wouldn’t be surprised to see this kind of data becoming a commodity soon. Google, as the owner of the most precise data, has an obvious advantage and would be virtually impossible to challenge. And, in this case, sites like Suite101.com can focus on where they have their edge: creating superior, editorial-grade content on any topic people want to read about."

We’ll leave the judgment of quality up to the readers, but Berger’s comments are interesting, and it will be interesting to see what happens in this growing industry of content.  What do you think?

One recent change in particular, however, has gotten some webmasters riled up. It’s being referred to as "Mayday," and some claim it is costing them money.

Do you think the update has affected your revenue? Comment here.

Ex-Googler Vanessa Fox, who spoke on a panel with current Googler Matt Cutts at Google I/O last week, quotes Cutts as saying, "this is an algorithmic change in Google, looking for higher quality sites to surface for long tail queries. It went through vigorous testing and isn’t going to be rolled back."

She also says Google told her that it was a rankings change, as opposed to a crawling/indexing change. This, she says, "seems to imply that sites getting less traffic still have their pages indexed, but some of those pages are no longer ranking as highly as before."

"This change seems to have primarily impacted very large sites with ‘item’ pages that don’t have many individual links into them, might be several clicks from the home page, and may not have substantial unique and value-added content on them," says Fox. "For instance, ecommerce sites often have this structure. The individual product pages are unlikely to attract external links and the majority of the content may be imported from a manufacturer database. Of course, as with any change that results in a traffic hit for some sites, other sites experience the opposite."

She has more to say about it at Search Engine Land, offering some of her own speculation. One reader accuses the change of delivering "a real blow" to his revenue.

This is not the first we’ve heard about "Mayday". There’s been discussion about it around the SEO community all month, but this is the first we’ve seen it really addressed by Google.

Most savvy webmasters have learned by now that they can’t rely on Google rankings alone to drive traffic. This is why social media opportunities presented by networks like Facebook and Twitter have become so attractive. The way people search and obtain information is becoming more and more diversified, not only spread out around different applications (largely due to increased mobile usage), but also within search engines themselves.

For example, Google recently rolled out its big SERP redesign, which gives users a great deal more options for filtering their results (or at least puts these options in the spotlight). The importance of ranking in a completely natural, organic search has become greatly diluted over time. Don’t get me wrong, it’s still nice, but it’s getting harder to rely on as well as less critical for discovery.

Stay tuned to WebProNews for our exclusive interviews with both Cutts and Fox from Google I/O, in which they talk with our own Abby Johnson about a variety of search-related topics. They should be posted soon.

Do you think your rankings have been affected by the "Mayday" update? Let us know.

Original Article: It appears that the effects of Google’s Caffeine update may be felt sooner rather than later. That is if you really do feel the effects at all.

What are your thoughts on Google’s Caffeine update? Discuss here.

If you are unfamiliar with Caffeine, it is an algorithm update that Google announced in the summer. Upon the announcement, Google’s Matt Cutts said, "The Caffeine update isn’t about making some UI changes here or there.  Currently, even power users won’t notice much of a difference at all.  This update is primarily under the hood: we’re rewriting the foundation of some of our infrastructure." Cutts told WebProNews about Caffeine in the following interview:

When Caffeine was introduced, so was a sandbox, where people could play around with Caffeine based search results, and get a look at how rankings were altered (if at all), and try to get a feel for how it was going to go. Now that sandbox has closed up shop, it looks like the Caffeine update will be live in Google search before too long. It will start after the holidays at least though.

"I know that webmasters can get anxious around this time of year, so I wanted to reassure site owners that the full Caffeine roll out will happen after the holidays," says Cutts on his blog. "Caffeine will go live at one data center so that we can continue to collect data and improve the technology, but I don’t expect Caffeine to go live at additional data centers until after the holidays are over. Most searchers wouldn’t immediately notice any changes with Caffeine, but going slowly not only gives us time to collect feedback and improve, but will also minimize the stress on webmasters during the holidays."

The announcement at what used to be the Caffeine sandbox reads:

We appreciate all the feedback from people who searched on our Caffeine sandbox.

Based on the success we’ve seen, we believe Caffeine is ready for a larger audience. Soon we will activate Caffeine more widely, beginning with one data center. This sandbox is no longer necessary and has been retired, but we appreciate the testing and positive input that webmasters and publishers have given.

Of course as people tested Caffeine via the sandbox, many of them blogged about their results and findings. The general consensus seemed to be that Caffeine is fast and utilizes real-time search a great deal. Given Google’s frequent announcements related to speed, and a recently announced deal with Twitter, speed and real-time search seem like logical updates to Google search results.

When SEOBook’s Aaron Wall tested Caffeine, he said he thought there was:

- an increased weighting on domain authority & some authoritative tag type pages ranking (like Technorati tag pages + Facebook tag pages), as well as pages on sites like Scribd ranking for some long tail queries based mostly on domain authority and sorta spammy on page text

- perhaps slightly more weight on exact match domain names

- perhaps a bit better understanding of related words / synonyms

- tuning down some of the exposure for video & some universal search results

This stuff should not necessarily be taken as gospel. These are just the results and speculations of individuals from tests of a product that was only introduced (for testing purposes), let alone finalized. It is what it is.

As the Caffeine update rolls out, there will no doubt be more and more mystery unraveled as search industry professionals scramble to stay ahead of the game, and Google drops subtle hints from time to time. It’s going to be interesting to see where Caffeine takes the world’s most popular search engine.

Have you tested Caffeine? What do you think about the update? Share your findings here.

Related Articles:

> Matt Cutts Talks Google Caffeine Update

> Where’s Bing’s Real-Time Search?

> Facebook/Twitter Use May Now Mean More for Google/Bing Rankings

We have launched a ranking change that reduces the number of results that are returned because of blogroll matches.  There are still problems to work out, but this change appears to be a big improvement over our earlier fix.  We had originally planned to launch an experiment for link: queries, but decide more recently to release this change first.  We are still working on the link: change and expect to have that ready in a few more weeks.

The change can improve the convenience of using Google Blog Search as part of your online reputation management strategy. Anyone with such a strategy in place is probably used to wading through all kinds of mentions of their company that are simply just mentions from blogrolls.

"Well, lets say you are like me and you track who links or mentions you via Google Blog Search," says Schwartz. "If someone has the Search Engine Roundtable in the blogroll, and the do a daily blog post, even if that blog post doesn’t mention the Search Engine Roundtable, blog search would show that new blog post as a match. Why? Because it is in the blogroll and Google thinks it is part of the content of the blog post."

The update should eliminate this problem and make tracking your brand through Google Blog Search a little easier. On the other hand, if you really like finding who has you in their blog roll, you might consider the update a loss.

"We’re constantly tweaking the Digg algorithm to ensure that a unique and diverse set of Diggs drives the content on the homepage," says Anton Kast on Digg the Blog. "We’ve made a few notable enhancements to our promotional algorithm recently, to ensure that all Diggers have a fair chance at getting their submitted stories promoted to the homepage. We felt it was important to call these out."

Power users have been gaming Digg for quite a while. A common complaint is that the average Digg user barely has a chance to get their submissions as much exposure as the power users.

"In addition to ongoing tweaks, we’ve taken steps to prevent abusive Digging behavior in an effort to improve the quality of content that is promoted to the homepage," says Kast. "Only a very small portion of the Digg community will experience these limits, well under 1%, and the vast majority of folks won’t be affected by them at all."

So how does the algorithm work? According to this video from Digg Community Manager Jen Burton, it has something to do with MC Hammer… Why not?

I have to wonder if this list of most-dugg sites from Hitwise will change. The blog post announcing the updates has less than a thousand diggs at the time of this writing. As one commenter points out, it wasn’t submitted by a power user.