Testing The Spam Filters
from the so-so dept
One of the columnists at the San Jose Mercury News ran some short unscientific tests on the email he gets to see how well the filters from Brightmail and Spamcop work in stopping the spam he gets. He found that both help a little bit. Brightmail stopped about half the spam with no “false positives”, while Spamcop caught 79% – but did have a few false positives (though, not very important ones). I’ve still been thinking about signing up for Spamcop, but people keep telling me good things about SpamAssassin. I wonder how well it would work on a similar test. In the meantime, I still think it sucks that people need to do anything at all to prevent spam. We shouldn’t have spam. People should simply know not to send out spam. I have a friend who has suggested that we hold public beatings of people convicted of spamming. It keeps sounding like a better idea.
Comments on “Testing The Spam Filters”
What to do with spammers
>I have a friend who has suggested that we hold public beatings of people convicted of spamming. It keeps sounding like a better idea
How about giving the spammer’s home phone number to telemarketers?
Junk postal mail helps subsidize 1st class postage, but spam puts most of the cost on ISPs and the recipients. I wonder if these spammers are making any money. I sure has hell don’t do business with them (telemarketers either) and I don’t anyone who does.
We need anti-spam laws with some teeth, complain loudly to your elected officials and never do business with a spammer.
As butt ugly as most pop-up, banner and other forms of web page adverts are, I can live with them. Spam is another story
Spam fighting
I quite like Spamassassin ( http://spamassassin.taint.org ) , I have it mark emails it thinks are spam, so that I can filter them with my email reader. It does erroneously mark some mails as spam, so I have to write specific filters on my reader to correctly redirect these addresses (which tend to be mailing list and hotmail friends) – I probably ought to just change spamassassin’s whitelist. It misses relatively few spam (17 out of 141), those it lets through I report to Spamcop which hopefully makes them less likely to filter through next time.
Currently Running A SpamAssassian Test
I admit it’s not highly scientific but I do get A LOT of spam so I’ve been running a 1 week test of SpamAssassin. So far, I have had only one false positive (a marketing email from United Airlines which I signed up for) and on average it has been missing between 7% – 12% per day. With an average of 60 – 80 spam messages a day, that’s usually under 10 false positives.
I would also add that I do not use the Razor or RBL modules or modules to support those features so perhaps that 7% – 9% could be cut down even further. All I know is that I was getting sick and tired of coming up with ever more complex rules in my mail client trying to stop this crap. I highly recommend it at this point and will be posting the full results of my test at http://www.rini.org sometime early next week.
Privacy Manager
Paying for spam filtering is no different than paying AT&T (who cleverly works both sides of the telemarketing fence) for the Privacy Manager service. No, I shouldn’t have to pay for it, but I shouldn’t have to pay what I pay in taxes either. I just consider it a quality of life cost.
Re: False Positives vs False Negatives
Don’t get these two mixed up — they are different types of errors and you should be sensitive to the rates of error attributable to each type.
False Positives (type I error) — a spam filter stops a message it shouldn’t have stopped (for me, about 1% of all messages reviewed by Spamcop)
False Negative (type II error) — a spam filter doesn’t stop a message it should have stopped (for me, about 10% of all messages reviewed by spamcop)
Going back a long time to my statistics days (can a practicing scientist add clarification here?), you can typically reduce one kind of error rate, but it tends to result in an increase in the other error rate.
The way around that problem is to have two tests (or in this case, screens) running in serial — one with a really low false positive rate, but a higher false negative rate, followed by a test with the opposite characteristics. Best of both worlds. This process is used all the time in diagnostic testing.