Two weeks ago, a few IT journalists quoted a report that painted challenge/response (C/R) spam filtering in glowing colors. But as we’ve said before, C/R is a bad idea, which causes false positives, poor deliverability, and turns its users into spammers.

C/R is by no means the final, ultimate solution to the spam problem. So how come this report is so much at odds with the views at Ferris Research?

The press was quoting a survey, conducted by a new analyst company on the scene. Users of spam control were asked to rate the spam filtering products they used. It seems the users rated C/R more highly than any other type. However, after studying the survey methodology, we conclude that it’s heavily biased toward C/R — although we don’t know whether this bias is by accident or design.

The survey questions calculate an aggregate score for a user’s spam filter, based on measures such as the number of spam messages that make it through to the inbox — false negatives — and the number of legitimate messages that get filtered out — false positives. The problems with the measures and the way they are aggregated into a final score include:

1. The number of false positives are weighted similarly to the number of false negatives — but in the real world, a single false positive is far more significant than a false negative.
2. It’s practically impossible to get an accurate count of false positives by simply asking users — and from our research, we conclude that users of C/R technology are subject to more false positives than users of conventional spam filters, either because many challenges never reach their intended recipient, or because the challenges appear untrustworthy.
3. The measures take no account of the fact that C/R causes innocent third parties to receive misdirected challenges, because spammers usually forge the envelope sender of their messages — not only is sending of such backscatter poor net citizenship, it can cause a C/R user’s outbound email to be filtered as spam when their challenges hit spamtrap addresses.

Again, for more on our views on this subject, see Why C/R is Bad, and More About the Backscatter Problem.

In summary: Don’t believe all that you read in the press. Or, as Mark Twain quoted Benjamin Disraeli:

There are three kinds of lies: lies, damned lies, and statistics.

… Richi Jennings