WebProNews » cookie deletion

More on the Great comScore Debate

Gary Angel — Wed, 25 Apr 2007 15:12:12 +0000

I’ve gotten several comments on my comScore post – and because the issue is so topical I wanted to post directly about my thoughts regarding them.

The longest, most detailed and, frankly, most baffling post was by Anil Batra. In my original take on this issue, I disagreed sharply with Anil about several aspects of the comScore findings. First, I argued that knowing the "real" number of visitors to your site isn’t useless and that not all analytics is simply an exercise in trend following and comparison. In the real-world, the actual numbers often matter and matter greatly. But that being said, the main thrust of my argument was that Anil was misunderstanding the potential impact of the comScore findings by treating them as something relevant only to reporting on total site traffic. Admittedly, this is the arena comScore is concerned with. But for a web analyst, data that suggests severe problems with cookie persistence have an impact far beyond mere traffic. Lack of cookie persistence will deeply color almost any analysis that persists across session: campaign attribution, "new" visitor analysis, "repeat" visitor analysis, "customer" analysis, attrition, sales cycle analysis, etc., etc. etc.

Let’s start with Anil’s arguments concerning traffic:

"The point I was trying to make is that you have to take everything in context. Going to Gary’s example of a conference, let’s say conference A tell you they attract 5,000 visitors and the other conference B says they get 4,000 visitors. Next day a third party comes out and says that all the conferences numbers reported by any conference are inflated and actual number is 75% of what they state then what’s the net result? Well Conference A is still better than conference B. Only thing is that they each now have 3750 and 3000 visitors respectively. Every conference in the world will have the same issue, their rank is still the same. I don’t think based on this information conferences will start charging less for the booth. However the rate per visitor has gone up for you but you can’t do much, that’s the market rate. Same argument goes for sites that sell advertising based on how many users they reach."

Unfortunately, however, advertisers don’t work in a world where there only option is the web. If I tell an advertiser that my reach is 1 million visitors, that’s going to be compared to other sites with a reach of 1 million visitors but also to radio, print, TV and more. So unless Reach just flat out doesn’t matter to advertisers, I fail to see how massive and consistent mis-reporting of web site traffic isn’t an issue.

Nor is it at all the case that we should expect cookie persistence issues to be the same for every type of site. I already pointed out two significant reasons why traffic estimates would be different for different types and different sites (% of Firefox users and Percent of Heavy Repeat Visitors). So it simply isn’t the case that every sites’ numbers will be equally inflated. Which pretty much seems to demolish Anil’s case, since every site won’t be effected equally.

So let’s go on to point number two – the impact of cookie deletion on many critical web analyses. Here’s Anil’s take:

"I understand Gary’s issue about repeat users and new users. But again, if you use two different systems they will report different numbers so which one is correct?
As Jacques Warren pointed out as a response to Gary’s post, the right solution (at this time) is to provide a reason for users to not delete their cookie (or give a reason to login). If Gary care’s about repeat users then I am sure he has strategies to get them engaged and give them a reason to login (or not delete cookies). Give users a reason to be loyal and they will be. Then you won’t have to worry about cookie deletion and hence your numbers will be accurate. Till you get to that level any number is a close estimate weather it is panel based or cookie based; and is not worth loosing sleep over."

This is the part I really don’t get. This isn’t an issue about comScore vs web analytics. When you use two different systems then one of them is more correct or both are correct or both are wrong. There is a real world we are trying to measure. And the problem is that when you do an analysis of "New" visitors and a significant percentage of your "New" visitors aren’t new, then your analysis sucks. It’s that simple. You aren’t looking at the right data and you have no reason to draw any conclusions from the data. And if you do forge conclusions from the data, they are probably wrong. For an analyst to suggest that very large non-random errors in the data don’t matter is, to say the least, perplexing. It’s as if someone told me that though I meant to poll Democrats but got half Republicans it won’t impact my survey findings on the Democratic Primary!

Jacques Warren may well be right – the only solution we may be left with is asking the measurement Tail to wag the web site dog. But how can anyone think this is something we shouldn’t be worried about (Jacques certainly seems to be!). First, we have to know whether this is true and then we have to let clients (or bosses) know that if they want measurement they have to fundamentally change their measurement approach to one based on opt-in principles. If you don’t think this is a big deal, go talk to industries that live in that world.

Nor is it reasonable to say that a site should be able to engage visitors enough to get them not to delete their cookies. If cookie deletion is a function of mass deletion or browser exit settings, that simply isn’t an option. And we all know what great success the measurement community had convincing everyone that 3rd Party cookies aren’t a privacy issue. I look forward to telling my clients that they have to become evangelists for cookie persistence. I’m sure they will love that!

And not all sites are Amazon.Com – many sites aren’t looking to marry visitors – they just want to date them occasionally. It’s unreasonable to think that many sites can get users to opt-in for measurement purposes when all they are doing is reading an article.

I’m not taking the comScore study for granted. As I pointed out in my original post, there are several good reasons to doubt the results and it may be that we don’t have to fundamentally change our ways of doing business. But as far as I’m concerned, if you’re a web analyst and you weren’t worried about the comScore results then you aren’t getting it.

Which brings me Clint’s concerns about the study being so limited. I share his concern – and it’s probably the main reason I think we shouldn’t jump to conclusions about the quality of our data. There are just too many studies based on single points of reference that turn out to be seriously flawed when applied to much larger industry. And, as I pointed out in my original post, no site is likely to be more vulnerable to serial deleters than a major portal.

I also agree with much of what Jacques says – assuming that we really do find our numbers are this flawed. It will certainly mean that sites need to focus measurement much more on opt-in users – and find many new ways to drive that level of commitment. I won’t pretend I think this is easy or that it won’t have a big impact on our business – and I’d much prefer a truly workable technical solution. And there are, as I discussed as well, technical methods for screening off the effects of cookie deletion from many kinds of analysis. If the community as a whole ends up buying into the comScore numbers I think that all of these directions will emerge as very important.

Comments

comScore – The Great Debate

Gary Angel — Tue, 24 Apr 2007 14:31:46 +0000

By now, nearly everyone in the web analytics community is abuzz over the recent release by comScore of a study on cookie deletion rates. comScore tracked a specific web site (advertised as a portal) and one 3^rd Party Ad Serving network against a panel of 400,000 users. During the study, comScore measured how often the 1st Party cookies (issued by the portal) and the 3rd Party cookies (issued by an Ad Serving Network) were deleted and replaced. The results can fairly be described as startling in several respects.

First, comScore pegged 1st Party Cookie deletion rates during an average month at 31%. Second, comScore measured the number of cookies and found that there were a significant percentage (about 7%) of serial deleters – who ended up having more than 12 1st Party cookies from the Portal site during a month. Taken together, these numbers imply a dramatic error rate in the total unique visitors a site measured even during a month timeframe. Perhaps as, or even more, surprising, comScore tracked relatively little difference between 1st & 3rd Party cookie deletion. 3rd Party cookie deletion was higher (as almost everyone would have expected) but by a margin best described as small.

Some of the reaction to this is predictable. First, there is considerable skepticism about the findings themselves. Perhaps that’s justified. Lord knows there are plenty of flawed studies done, not least when the studying party has a strong self-interested stake in the outcome. In addition, the frequency of deletion coupled with the lack of disparity between 1st & 3rd Party cookies (where automated Spyware tools might provide an explanation) makes it hard for many people to understand. Are internet users really this consistent about deleting cookies when they have to do it manually? It does seem hard to believe. Keep in mind, as well, that we are talking cookie deletion – not rejection. So though 3rd Party cookies may share a similar lifespan to 1st Part cookies, they are considerably less likely to find their way onto a computer in the first place. And while the comScore study wasn’t especially clear about this, it doesn’t seem to have focused on cookie rejection at all.

There is a second common reaction summed up in the idea that this level of errors in reporting doesn’t much matter because analysts are concerned with trends not absolute numbers. Anil Batra of Zaaz, for instance, argues (here) that it would make no (or little) difference whether your site got 5k visitor or 3K visitors – the important thing is how your site is trending in response to your actions.

I’m afraid I can’t agree with this thinking. At first glance it’s not necessarily wrong-headed though I’d argue that in fact it’s important to know your actual audience size on the web – that real numbers do matter and not just trends. Imagine a conference organizer who wouldn’t tell sponsors how many visitors the conference draws – only that it is trending upwards. I doubt I’d sponsor a booth!

But this is really just the tip of the iceberg when it comes to analysis. After all, total visitors IS pretty far down on the list of statistics of interest to the analyst. Unfortunately, almost every single statistic that matters is going to be effected – and sometimes devastatingly – by this level of error in visitor tracking. Perhaps Zaaz doesn’t track "New" visitors to the site vs. returning ones. But I do. And when half of my "new" visitors are really heavy repeat visitors I can hardly hope that my analysis will be crisp. Then too, I’ve always believed that tracking repeat customers on a site is central to most eCommerce analysis. But with this level of error I’m misclassifying a big chunk of that behavior.

Or again, perhaps people aren’t worried about what tools or pages drove repeat visits or cross-session sales behaviors. But it seems important to me. For a portal site like the one tested by comScore, the bellwether analysis is content impact on engagement. I’d like to say that this analysis may still be possible with this level of error, but I’m not sure that it would be.

Not concerned about these little things? Well how about the fact that every one of your campaigns might be significantly mis-counted in terms of conversion? That seems pretty consequential.

In short, just because comScore isn’t focused on analytics but only on traffic reporting doesn’t mean the impact of their findings is limited to traffic. And the impact for analysis of this level of error in 1st Party cookies would be very bad indeed. That’s not an admission you’d probably expect to hear from a web analytics consultancy and I’m sorry if this constitutes breaking ranks, but our first commitment ought to be to the truth of the numbers.

On the other hand, I too have some caveats about the general applicability of these results to every site. First, it’s important to understand the difference between each statistic. The 31% 1st Party cookie rate is, in my view, probably the most damning statistic in the study. Why? Because unless the behavior is somehow related to the specific site in question, you’d expect it to hold up for every site. Or then again, maybe you wouldn’t. If we accept that this statistic represents manual deletion of cookies, then there are two alternatives. The first is that people simply erase all cookies. That’s certainly possible – maybe even likely since it’s so easy to do. But a user may also scan the cookies on the target system and since cookies are often quite identifiable it’s possible that some cookies are much more likely to be deleted than others. It may be that an Amazon or Charles Schwab cookie is much less likely to be deleted than an AOL one. That’s one of the reasons it would be very interesting to understand the qualitative part of the comScore study – namely, what deleters (especially "serial" deleters) were actually doing. In addition, I’d like to understand if this problem is especially severe for certain browser populations (Firefox). And are users simply setting their browser to delete all cookies every time they close? This might explain the behavior of deleters – they aren’t doing any manual work at all. It would also imply that for this segment even same day uniques are overstated.

As comScore positions the study, however, the most damning finding is probably the one about the potential traffic impact of serial cookie "deleters." And for their particular chosen site, the impact would indeed be considerable. But for many of our sites, this group would probably have much less of an impact. Why? Portal sites are unusual in attracting a very high volume of repeat visits. comScore reported an average of nearly 13 1st Party cookies during a month for the 7% of visitors with 4 or more cookies detected. For most of our client sites (that aren’t portals), the overwhelming majority of visitors will come much less often than this. And if only a very small percentage of visitors are multiple repeaters and only a small percentage (7%) of those are multiple deleters, then the impact is likely to be considerably less (in terms of percentage of total visitors) than for a portal site. It’s likely true that no set of sites has a higher percentage of frequent return visitors than a true portal.

I’ll be honest – I find the study worrisome. It’s not that I haven’t realized that cookie deletion is a significant issue or that the potential impact of frequent return visitors on visitor traffic eluded me. But every analyst knows there is a line between data that is fuzzy but useful and data that is too messy to analyze. And I’ll be honest as well in saying that we see many sites where the number of visitors flagged as "New" would be better explained by the comScore view of the world than by the one prevalent (including by us) in the web analytics world. Depending on how the comScore study shakes out, web analysts may be facing a significant rethinking of where we are relative to that "fuzzy" line for almost every kind of interesting analysis. For some types of sites, at least, this might drive our short-term tool kit back to the pathetic same session analysis we were stuck with in tools a couple of years back.

comScore has their own agenda, here, obviously. But to me, the issue is ultimately about much more than unique visitor counts. Perhaps Anil is right to discount the importance of that statistic. But in doing so, he’s surely missing the larger point. In my view, the basic analytic toolkit of visitor segmentation and cross-session tracking is at risk in light of these findings. What’s the point of visitor segmentation when your visitors are lost every couple of days? This is an issue the measurement community just can’t ignore. I remember when 3rd Party cookie rejection rates began to climb and the ostrich like mentality that insisted this wasn’t an issue. Till suddenly the vendors made it possible to use tagged 1st Party cookies and everyone began implementing first party cookies. Sadly, we may not have solved our problem.

Are there short-term solutions? Possibly. And they begin with getting a handle on the scope of the problem for your site and understanding how to protect your analysis from these problems. You can begin to get a handle on the issue for your site by examining the trends in "New" visitors – especially if you can measure from significant events like a measurement start, a new cookie, or major traffic spikes. You might also want to look at the percentage of New/Returning visitors by Browser type. If you have strong behavioral cues on your site (like Login), you can measure the degree to which these behaviors are used by "NEW" visitors. This percentage compared to total visitor usage is an excellent way to get a reasonable read on how much or little of a problem you have. Handling bias is trickier, but if you want to analyze, for example, whether types of tools or content drive repeat visits, then you’ll be wise to start with a universe of visitors that you can prove has remained cookie constant. How might you do this? By insuring that you segment for behavior in both the 1st and last month of your study. This will narrow your analysis, but insure that you’ve ruled out cookie issues.

In the end, every problem in data quality simple complicates the life of the analyst more and makes doing good analysis just that much harder. I don’t much care who wins the great traffic reporting war. But I care very much that web analytics have a mechanism for tracking with some degree reliability the over time behavior of visitors. Where there is a will, I’m confident our truly fine tool vendors will find a way. It may be incumbent on analysts to provide that will.

Comments