Earlier this year, following the initial roll-out of the Panda update, Google’s Matt Cutts and Amit Singhal did an interview with Wired, in which they talked about people they had rating search result quality.
Last month, the raters handbook was leaked. Morris Rosenthal had some interesting things to say about it here.
The raters are discussed a bit more in this video from Google:
Cutts and Singhal spoke together at PubCon this week. Here some takeaways from their keynote discussion (which includes talk about algorithms in testing, which would focus more on sites with too many ads above the fold).
Matt Cutts was reportedly talking about the raters guide at a PubCon networking event. In the WebmasterWorld forums (via Barry Schwartz), user Tedster says the following clarifications came from Cutts:
Webmasters tend to put a slightly skewed angle on this. The quality raters are actually rating a SERP (that is, a particular algo configuration) as a quality control measure for the algo team. Their ratings do not directly change rankings- but they hep the algo team see if the algo worked as planned or not.
Also, note that this document is not for the spam team. They also have a training document and use human quality raters – but that document has never been leaked.
While it may not hurt to pay attention to this handbook, I would consistently refer back to that set of questions Google put out earlier this year. That “above the fold” stuff is right in line with some of the stuff on that list.