We posted a video Google uploaded this week of a Matt Cutts presentation from late January at a webmaster conference in Korea. Here it is again, in case you missed it:
There’s a interesting section of his talk, in which he notes that Google runs over 20,000 search experiments a year. We know Google makes hundreds of changes to its algorithm each year, but Cutts sheds a little more light on the process Google goes when making a change. Here’s one of the slides he showed demonstrating the process a change goes through from the idea stage to the launch stage:
“So these are numbers from 2009, but the proportions, the rough percentages, are about the same,” says Cutts. “We would try out anywhere from 10,000 to 20,000 ideas. Of that, many more thousand, 8,549, we would send to these blind side-by-sides, and then a smaller fraction of that actually get sent out to real users and to see whether users tend to click on the newer results or tend to click on the older results. And the final number changes that we launched last year was 585.”
If you have the time, check out the full 45-minute presentation.