Nothing, if you use online panels properly.

My worry, perhaps unwarranted, is that old-time marketing folks will mistake an online panel’s ability to answer the "why" question for an ability to answer the "what" question. In other words, I mistrust people to tell you what they will really do in all situations, but I do trust them to tell you why they would do something.

We know that people don’t always do in real life what they say they will do when questioned in an artificial situation. They may be embarrassed to say they don’t know what they’d do, or feel uncomfortable saying what they would really do, or perhaps they don’t know. As an example, no one surveyed would ever tell you that they prefer to click on more pages to get the right answer, but in reality they sometimes do. We know from tracking real users that people prefer several "easy" clicks—ones that require little thought—to one very painful click where they must read a lot and then take a guess at the right thing to do next. But they would always tell you on a survey that they prefer fewer clicks; we know what they really do only by watching clicks from real users on live sites.

So use online panels for what they are good for—brainstorming ideas or getting to the "why" behind what you know people do. But use real activity tracking to see "what" people do. Let your metrics system prove what people do in real life and then use online panels to find out why they do—between them, you’ll have the information you need to identify areas for change and change them. Because in the old days we had no mechanism to find out "what" customers were doing, I don’t want us to sieze upon online panels as the only information we need—let’s use all the information available: both activity and survey data.