Fair Isaac shook up the online advertising world with its early determination of a much higher rate of click fraud affecting advertisers than has been previously determined. The initial research by Fair Isaac chief scientist Joseph Milana found rates of 10 to 15 percent of “pathological” traffic hitting advertisers.

The need for more data from a broader pool of advertisers became apparent when it was revealed Fair Isaac only had a sample of less than a dozen for its research. They may be on their way to broadening that pool now.

Boris contacted us to say AdWatcher and Fair Isaac were now working together. AdWatcher will contact large volume advertising clients about participating in Fair Isaac’s ongoing study of click fraud. Those clients would be anonymous in the reports Fair Isaac produces.

Google derided the small sample used by Fair Isaac when they first publicized their findings. If Fair Isaac can grow its pool and demonstrate click fraud rates run in the same range as their initial report, Google and others will be hard-pressed to discount such a discovery.

It’s news that has grabbed the attention of the search industry. Even though Fair Isaac has been careful to note it only discussed early results of its study at InterACT 2007, the figures they have found for advertisers being charged for illicit clicks corroborates some third party research into the matter.

We talked with Dr. Joseph Milana, who worked on the Fair Isaac click fraud study. He stressed the expertise Fair Isaac possesses after years of fraud detection in other industries, and that it could be applied to illicit clicks.

A follow-up message we received from Google bears out Dr. Milana’s point. Google and Fair Isaac both use statistical anomaly detection as part of their click fraud detection process. Here’s Google’s response:

•  We have post-click data from thousands of advertisers through our conversion tracking tool. They are claiming post-click data from a “handful” of advertisers.

•  It sounds like FI is doing the same kind of analysis we’re doing – statistical anomaly detection – only with considerable less data. That is, if you assume an advertiser has a 1% CTR, that means they are receiving 100 ad impressions (no one but Google gets ad impression data) for every one click. As a result, Google has 101 pieces of data to analyze vs. their 1.

•  Most clicks don’t result in conversions, and in fact many/most (depending on the site) clicks don’t result in much “advertiser side data”. Imagine legitimate users who click an ad and then immediately leave if they don’t like their site. That’s obviously not click fraud.

One point Dr. Milana made at InterACT will be of interest to the statisticians among our readership. He said Google’s automated effectiveness claims at detecting click fraud would be roughly equal to a Kolmogorov-Smirnov Test score of 95.

He called that figure “not believable.” Without some more openness from Google into their fraud detection methods, which they guard closely for the protection of their advertisers, or more results from the Fair Isaac test, it’s hard to tell who to believe right now.

Click Fraud Unbeatable Without Advertiser Data

Pathological traffic that impacted a small sample of less than a dozen websites resulted in advertisers being billed for illicit clicks that the search engines did not catch. The early results of a Fair Isaac study of click fraud showed some channels could hit ad campaign budgets at a rate of 10 to 15 percent.

Also, Fair Isaac contends that Google’s specific estimates of what their unsupervised detection efforts find in click fraud is "not believable," based on their experience detecting fraud in other industries. Without advertiser data, search engines cannot defeat click fraud.

We spoke with the newly popular Dr. Milana about his presentation at InterACT 2007 regarding click fraud. Fair Isaac did share the 35-slide presentation with us, and Dr. Milana spoke at length about how Fair Isaac uses technology to detect fraud.

Part of his research and development work involves looking for places to apply his firm’s technology. Most people know of Fair Isaac for their work on credit scores, but there is a lot of high-tech they can and have brought to other fields.

"It caught our attention," Dr. Milana said of click fraud, which arrived on their radar about two years ago. The issue for them would be to employ their signature Falcon technology to assessing click fraud.

Fair Isaac turned to SEMPO for assistance with the project. SEMPO encouraged advertisers to provide data to Dr. Milana so they could profile transactions, i.e., clicks, look for anomalies, and determine a pathology score for that visit.

(We contacted current SEMPO president Gord Hotchkiss about the Fair Isaac study. Though he could not comment on Fair Isaac’s findings due to the small size of the dataset, he did say, "At this point, our primary concern is increasing the size of the data set (getting more advertisers to volunteer their data) to allow Fair Isaac to extend their methodology to a more representative data set.")

"We see some nuances, and want more data," Dr. Milana said of their early results. They broke pathological traffic into two categories: malicious competitors, a threat they see as benign; and clicks for profit, where affiliate traffic in particular was a problem.

He noted how their technology has been proven in the payment card industry. Applying it to click fraud assessment, where search engines do not have the advertiser data, meant doing anomaly detection to find illicit traffic.

"Advertisers were charged" for this traffic, Dr. Milana said. His preliminary conclusion contends that search engines cannot solve the click fraud problem alone without having advertiser data available.

A request to Shuman Ghosemajumder at Google for his company’s feedback on Fair Isaac’s early results brought us a statement from Google on click fraud:

We take the protection of our advertisers very seriously. The amount we filter proactively is consistently less than 10% of all clicks but individual invalid click rates can vary by advertiser, campaign and even keyword.

The relatively rare cases we find of advertisers being affected by undetected click fraud constitute less than 0.02% of all clicks.

Without knowing more about Fair Isaac’s data gathering and methodology it is difficult to comment on their study except to say that a handful is not a representative sample size for our hundreds of thousands of advertisers.

Here is what we do know: Fair Isaac had access to advertiser data from a small sample of websites. Their modeling methods as applied to determining click fraud found some high instances of those advertisers being charged for illicit clicks, at a much higher rate than search engines claim gets through to those ad campaigns.

Considering Fair Isaac’s interest in finding new markets for their products, we wanted to know if Dr. Milana entered this study with a formed hypothesis about click fraud. He said he had "an open mind going in." Since stats on click fraud were so varied, Fair Isaac had no preconceived notions about what they might find.

Dr. Milana’s main point holds that click fraud can’t be beaten with search data alone. Unlike the payment card industry, there are no objective click reviewers out there. Advertiser data needs to be part of the equation because search engines can’t see the true impact of a click.

(We asked Google about the need for advertiser data to provide a better picture of click fraud, but we did not receive a response by press time.)

The percentage of click fraud Fair Isaac found was in line with numbers reported regularly by Click Forensics. Tom Cuthbert, Click Forensics’ CEO, told WebProNews that web marketers need to take the numbers seriously. "14.8 percent of pay-per-click advertising dollars is a huge amount of money," he said.

For Dr. Milana, the early research will proceed into more in-depth work, as they gain more advertiser data to use in their assessments. The search advertising industry can probably get away with being somewhat dismissive of the early Fair Isaac report due to the small number of sites being studied.

The real test comes when Fair Isaac has a larger body of data from advertisers to calculate. If a wider sample proves the accuracy of the first report, advertisers will really escalate their demand for greater transparency into search engine efforts to stop click fraud.

Fair Isaac Enters Click Fraud Debate

The whiff of click fraud has the potential to erode the multi-billion dollar revenue streams that big online ad providers like Google and Yahoo generate. It’s a big part of the reason why they have looked at third-party claims from firms like Click Forensics so negatively.

Tom Cuthbert, Click Forensics president, has a little more ammunition to play with, now that Fair Isaac has found in its research since August 2006 that the figures cited in their study mirror those from Click Forensics.

A report in the Globe and Mail noted how Fair Isaac found a significant percentage of questionable traffic in its study. That “pathological” traffic as cited by Fair Isaac’s Joseph Milana likely means click fraud is taking place.

Both Google and Yahoo have taken issue with third party click fraud claims before, as they claim their systems throw out illicit clicks before they ever impact an advertiser’s account. Google has even said they detect all but about 0.02 percent of the click fraud affecting their AdWords clients.