Google Runs 20,000 Search Experiments A Year, Here’s The Process Diagramed

    April 13, 2012
    Chris Crum
    Comments are off for this post.

We posted a video Google uploaded this week of a Matt Cutts presentation from late January at a webmaster conference in Korea. Here it is again, in case you missed it:

There’s a interesting section of his talk, in which he notes that Google runs over 20,000 search experiments a year. We know Google makes hundreds of changes to its algorithm each year, but Cutts sheds a little more light on the process Google goes when making a change. Here’s one of the slides he showed demonstrating the process a change goes through from the idea stage to the launch stage:

Google Idea

“So these are numbers from 2009, but the proportions, the rough percentages, are about the same,” says Cutts. “We would try out anywhere from 10,000 to 20,000 ideas. Of that, many more thousand, 8,549, we would send to these blind side-by-sides, and then a smaller fraction of that actually get sent out to real users and to see whether users tend to click on the newer results or tend to click on the older results. And the final number changes that we launched last year was 585.”

If you have the time, check out the full 45-minute presentation.

  • http://ramcharanmarriagephotos.com/ bhanava

    Super .. They are running experiments are wasting their times

  • http://www.tipsinablog.com Daniel

    And I complained about the 500 Algorithm updates in the past 12 months!

    20’000 search experiments annually is quite incredible. Given that many of these would be smaller parts of much larger tests being conducted..though, it is still a large number.

    The number of tests surrounding each Algorithm update would be many, especially for the more stringent(large scale–influence) changes to the search(quality)system…

  • http://www.dallascommercialrealestateco.com Anthony

    20,000 tests a year seems a little excessive but an interesting article none the less.