Bing has no idea if people are more likely to select Bing results when using its Bing It On challenge, and has admitted as much in a blog post defending its campaign.
Despite being over a year old, the Bing It On challenge has been in the news this week, due to a study from Freakonomics’ Ian Ayres and a a group of Yale law students.
When it launched the campaign last year, Bing said that people prefer its search engine 2:1 over Google. Earlier this year, they updated the wording to say that people simply prefer Bing to Google for “the web’s top searches”. The Bing It On site currently says, “Wherever we go, people prefer Bing over Google for the web’s top searches.”
And they are indeed still going around offering the challenge. They were recently spotted presenting it at a Seattle Seahawks game.
Ayres wrote this week, “When I looked into the claim a bit more, I was slightly annoyed to learn that the ‘nearly 2:1′ claim is based on a study of just 1,000 participants. To be sure, I’ve often published studies with similarly small data sets, but it’s a little cheeky for Microsoft to base what might be a multi-million dollar advertising campaign on what I’m guessing is a low-six-figure study. To make matters worse, Microsoft has refused to release the results of its comparison website, BingItOn.com.”
He said, “So together with four Yale Law students, I set up a similar-sized experiment using Microsoft’s own BingItOn.com site to see which search engine users prefer. We found that, to the contrary of Microsoft’s claim, 53 percent of subjects preferred Google and 41 percent Bing (6 percent of results were ‘ties’). This is not even close to the advertised claim that people prefer Bing “nearly two-to-one.” It is misleading to have advertisements that say people prefer Bing 2:1 and also say join the millions of people who’ve taken the Bing-It-On challenge, if, as in our study, the millions of people haven’t preferred Bing at a nearly a 2:1 rate. Microsoft might have realized this and has more recently altered its advertising to back off their original claim to just say that people ‘prefer’ Bing.”
Bing’s Matt Wallaert defended the challenge in a blog post, criticizing Ayres’ claims and study. You can read the entire post to see all of his jabs, but here’s what he said about Bing not tracking the results of Bing It On:
Ayres is bothered that we don’t release the data from the Bing It On site on how many times people choose Bing over Google. The answer here is pretty simple: we don’t release it because we don’t track it. Microsoft takes a pretty strong stance on privacy and unlike in an experiment, where people give informed consent to having their results tracked and used, people who come to BingItOn.com are not agreeing to participate in research; they’re coming for a fun challenge. It isn’t conducted in a controlled environment, people are free to try and game it one way or another, and it has Bing branding all over it.
So we simply don’t track their results, because the tracking itself would be incredibly unethical. And we aren’t basing the claim on the results of a wildly uncontrolled website, because that would also be incredibly unethical (and entirely unscientific).
Ayres’ final issue is the fact that the Bing It On site suggests queries you can use to take the challenge. He contends that these queries inappropriately bias visitors towards queries that are likely to result in Bing favorability.
First, I think it is important to note: I have no idea if he is right. Because as noted in the previous answer, we don’t track the results from the Bing It On challenge. So I have no idea if people are more likely to select Bing when they use the suggested queries or not. Emphasis added.
He goes on to explain some differences between Bing’s controlled studies and the Bing It On site.
More from the Bing It On saga here.
Image: BingItOn.com
