WebProNews

Tons of data can't beat a skillful human's judgment

Despite massive server farms with processing power to burn through mountains of user data, Google relies on a quaint approach to tweaking its rankings.

Many data points impact how a site appears in Google's listings. For many in the SEO industry, the challenge is straightforward: get a site into the top five listings on the first search results page for a given query, or forget about receiving any traffic from searchers.

But the factors that need to be influenced may harken back to an older time of data research, contended Kosmix co-founder Anand Rajaraman, who blogged about a discussion with Google's Peter Norvig at Datawocky. Surprisingly enough, Norvig revealed to Rajaraman the impact human raters still have on Google's page rankings:

Peter confirmed that Google does collect such (user) data, and has scads of it stashed away on their clusters. However - and here's the shocker - these metrics are not very sensitive to new ranking models!

When Google tries new ranking models, these metrics sometimes move, sometimes not, and never by much. In fact Google does not use such real usage data to tune their search ranking algorithm.

What they really use is a blast from the past. They employ armies of "raters" who rate search results for randomly selected "panels" of queries using different ranking algorithms. These manual ratings form the gold-standard against which ranking algorithms are measured - and eventually released into service.

We think there is an interesting side note regarding Google and its past discussions of artificial intelligence in its operations. They have hinted obliquely at times about machine, rather than man, being the ultimate arbiter of information passing through its voracious network.

But humans may not be as disposable and fallible as AI advocates would like us to believe. If the engine that drives Google's billions in search ads begins with a small number of human raters, maybe there are some things a computer can't do.

Gary Kasparov, the chess grand master who suffered defeat at the hands of a computer, should take heart. For all of the machine's stone-cold processing, the human brain still wins out in matters where billions of dollars are at stake.