Full-Text Search Should Be Fast

Interesting Statistics on Mobile Phone Use

We recently wrote that statistical filters such as POPfile are extremely accurate when used correctly, but too hard for most people to use. That’s why we’re excited by the potential of Clearswift’s new product line. It takes the underlying technology from POPfile — in the form of the commercial polymail library — and adds the capability for automatic learning. Clearswift uses the library to implement statistical spam control without the need for end users to train the filter. Here, the filter is trained not by users flagging errors, but by calculating how confident the filter is that a message is spam or not.

It decides its confidence by using a cocktail of techniques, such as IP blacklists, call-to-action blacklists, and automatic whitelisting. If this cocktail indicates that a message is probably spam and the statistics indicate that as well, then the filter will train itself that messages that look similar are also likely to be spam. Similarly, it will auto-learn what legitimate messages look like.

… Richi Jennings

Notes on Message’05, London, Oct 26-27

Statistical Spam Filters Are Too Hard To Use

Just because you install collaborative technology doesn’t mean it gets widely adopted. Luddites still exist.

Here are what we believe are the key things to do to ensure widespread adoption in an organization:

• Think through the training program. Substantial resources will probably be needed, in a variety of forms; for example, classroom training, helpful little cheat sheets, help desk, etc.
• Get top management involved. When they use it, others have to follow.
• Get champions involved, who will actively promote the system.
• Identify mission-critical applications for the technology that are likely to benefit significantly from the technology. Get the technology used in such cases, and promote the successes.
• Make sure that office administrators and executive assistants are given good training and encouragement; they have great influence.
• Set deployment goals and track them. When things aren’t working as planned, take adjusting action.

… David Ferris, with thanks to panelists at the recent Collaborative Communications Summit in San Francisco

Full-Text Search Can Be Serendipitous; Should Be Global

John Graham-Cumming (or jgc to the anti-spam cognoscenti) is best known as the author of the open source tool POPfile. He has now created a commercial product to build this functionality into other commercial products.

POPfile is an email classifier — it’s best known for it’s ability to classify email as spam, using Bayesian analysis and other statistical methods. The new product is a library called polymail. The first commercial product to use it is the new range of MIMEsweeper email security products from Clearswift.

POPfile can be a very accurate spam filter, so we have high expectations for the accuracy of Clearswift’s new products. More details at jgc’s commercial site: www.extravalent.com.

… Richi Jennings