Spam is an issue that has been discussed and discussed. Laws have even been passed about it. The reason I decided to write something now about it is because I've been using a spam filter for awhile and I think it is working. Usually. I also think it represents the proper way of thinking about Spam.

Sen and I have talked about spam a lot and we often talk about how it is yet another basic mistake in the way that the Internet was designed. If only smtp could let you authenticate the user before you received mail, we could make much better mail filter. Alas, this can not be changed now. (Or it would be very difficult.) So we have to come up with some solutions.

The best solution we have found so far is whitelisting. It is a way to make a filter that only lets mail from certain addresses through. Originally Sen had made a script for me where I had a separate mailbox for mail from people who were in my address book. Now we have moved over to TMDA which lets you create white lists, black lists and a variety of other things. I have mine set up so that I can create and maintain my own list of addresses and domains that I want to receive mail from. We also have it set up so that if someone sends mail that is from someone not on the list, they get a message asking to reply and confirm that they are a human being. Once we receive the confirmation, the mail comes through. To filter for humans I don't want to get mail from or for that intelligent spam robot, I can make black lists.

There are still various problems with the system, but it works quite well. I still have it set up so that I have a mailbox for all of the mail that is rejected and I go through this periodically to make sure I did't miss something important.

In Japan, spam has become a huge issue because the recipient has to pay for the mail on i-mode phones. NTT Docomo is trying very hard to deal with this issue with filters of their own, but there are still major problems.

I do think that spam should be solved by certificates, authentication, keys, etc. on the user side and not by some huge central server...

15 Comments

Do you know anything about Vipul's Razor? The idea of combining collaborative filtering and spam filtering on a p2p basis is appealing:

http://razor.sourceforge.net/

What is Vipul's Razor?

Vipul's Razor is a distributed, collaborative, spam detection and filtering network. Through
user contribution, Razor establishes a distributed and constantly updating catalogue of
spam in propagation that is consulted by email clients to filter out known spam. Detection is
done with statistical and randomized signatures that efficiently spot mutating spam content.
User input is validated through reputation assignments based on consensus on report and
revoke assertions which in turn is used for computing confidence values associated with
individual signatures.

This sounds interesting Howard. Thanks. The main problem I have with centralized spam filters is that I am against widespread use of blacklists in general. I think it is important that it can not be used for censorship. I think that making sure something like Vipul doesn't get implemented and then fall under the control of twisted interests is important. That's why I like enabling individuals. Having said that, if there were a way to design it so that it can't be controlled, it might work. I know you are into http://www.rheingold.com/smartmobs smart mobs these days. I wonder if there is a way for smart mobs to be coded to prevent corruption. I guess that is the p2p theme...

FWIW, there's also SpamAssassin:

http://www.spamassassin.org/

which can be configured to use Razor -- I imagine something similar can be done for TMDA as well.

Recently, I've started to see other commercial offerrings as well.

For a more radical solution, there's the idea of doing mail delivery where each user fetches their mail directly from a sender's "outgoing mailbox".
Since you only retrieve mail from people which you want to get mail from, there's no way to get spam.

Over the last 4 years, I've seen various different people independently come up w/ this idea. One good write up is by DJB:

http://cr.yp.to/im2000.html

I think about spam like I think about the Death Penalty - better 100 guilty email should get through than 1 innocent email is trashed. I'd hate to see important/personal correspondence lost to a blacklist. I guess that argues in favor of whitelists.

One friend required an application for permission to send mail to him - I sent him a casual message and got a computer reply asking me to reply again in a certain way to get on his whitelist.

At the time I thought it was impersonal. But perhaps whitelists are the future. I'm holding off; for the time being I endure 60% of my hundreds of daily emails being spam. When I am on a modem or slow PHS wireless connection I get more irritated.

So between legislation, black lists and white lists Joi, you favor White Lists? I still dream of legislating responsibility for email. Perhaps that's silly and irresponsible. Perhaps I'll eventually take up the means of my filtration in my own hands.

The death penalty, while arguably a social evil (I'm agin it), is not an attack on the commons. Spam threatens to make email useless. I am wary of blacklists. I corresponded with the Well's sysadmin and he noted that:

1. None of my email goes away, but anything that gets above five points (and I can alter those criteria) is sent to my Spambox. I've looked through hundreds of filtered emails, and they are ALL Spam.

2. I can also ask Spam Assassin to forward the email to my pop, and I can filter anything marked Spam to a Eudora mailbox.

3. Spam Assassin uses recommendations from services that use blacklists, but these are only one of many criteria. Blacklisting of the originating ISP alone is not sufficient to mark email Spam.

I am ALL FOR an arms race against spammers. Yes, it will make the smarter ones come up with new tricks. But most of them are stupid and will go away. I'm hoping that this shoes that the Internet as a common pool resource can defend itself.

Howard, I agree with you short term, but I think that there should be a better way to solve this technically. I think the idea of whitelists or some sort of digital signature might better. For instance, if you wanted to accept mail from people you knew or people introduced by people who you accept introductions from... or something like that.

Having said that, if Spam Assassin gets it right 99.9% of the time, maybe that's OK. It just seems... messy. ;-)

Perhaps the following post from spamtools is also of interest:

http://archive.iecc.com/article/spamtools/20020612002

Below is a link to another anti-spam project:

http://www.paganini.net/ask/

ASK -> Active Spam Killer

Link to page listing various TMDA-like systems:

http://software.libertine.org/tmda/faq.cgi?req=show&file=faq01.009.htp

Howard just sent me a link to a BoingBoing entry about Ed Felten who got blacklisted on a SpamCop and got cut off by his ISP. Horrible story and EXACTLY why I think centralized spam filters aren't the answer.

http://boingboing.net/2002_08_01_archive.html#85361424

Joi, you noted that Docomo is trying very hard to stop its users becoming victims of spammers.

Unfortunately, the way they seem to be doing this is completely arbitrary. They appear have a rule on their mail server which says "if more than x number of mails arrive from a specific domain within y time period, this is spam, so block all mail from this IP range for a period of time"

I have worked for 2 ISPs within the last year, and both have had problems with this docomo policy. Many customers forward their email from one of their IDs to their cell phone - Docomo`s rule stops them getting their own mail.

Docomo`s press release (below)suggests that the user can change a setting to allow mail they wish to receive, but in reality mail from our domains is being blocked from reaching the customer - customer settings are irrelevant.

DoCoMo Phones to Take Advantage of New Anti-spam Legislation
TOKYO, JAPAN, September 3, 2002 ---

...As a convenience to its customers, DoCoMo will begin automatically blocking all such e-mails starting October 1, 2002. If users wish to receive such e-mails, however, they can simply go to iMenu, the official
i-mode portal site, and change the setting free of charge beginning September 24.

DoCoMo also announced today that from September 24 it will begin enabling PDC-based i-mode users to block, or exclusively receive, e-mail from up to 20 (currently 10) user-specified addresses or domains...

There is no way for ISPs to be put on a docomo whitelist, as no such white list exists.

Some of the other comments (including Justin`s) noted that one piece of lost mail was too much.

What about if that one piece of mail you cannot receive is being legitimately forwarding by yourself?

The word `draconian` springs to mind.

Gary Garner
JENS Corporation
http://www.spinnet.jp/

SPAM is a huge problem for me. We've got a lot of websites running and got started on the Internet very early, long before spam was much of a problem.

Therefore there were lots of email addresses used and available on public pages. I'm on lots of lists and spam accounts for about 99% of my email so it's a HUGE problem for me.

With extensive filters, on a daily basis I only look at about 1% of the email that gets to me on certain accounts and still 80% of that is spam. The rest of it I do check and delete every couple of months...

I was thinking of trying this TDMA, but then realized that the reason I don't just chuck all my accounts and use only one email address, (especially getting rid of my yahoo email address so I don't have to see another obnoxious Bernie Mac ad on Yahoo), is because I have some things I don't want to lose.

A single lost email from whatever still hasn't been moved from Verisign can spell disaster for a particular domain.

And I certainly can't remember all those little things I signed up for under different email addresses over the years...

In retrospect, had we known what problems spam causes, we would have never had anyemailaddress@domain.com forward to the main address on domain.com. Since then we've changed it to only accept specific emails for individual users. But I still have a bunch of accounts. And missing that one important email means keeping all these accounts and having to deal with this problem that is growing by leaps and bounds daily.

I feel that the only solution is some serious efforts by government to make it very illegal with very severe penalties. Sure, some spam will still come about from certain countries. But if everyone in the United States, everyone in Europe, everyone in Japan etc.. were scared to spend their lives in jail for this, I bet 90 - 95% of spam would be gone overnight.

The fact is, on a legal basis this is not being taken seriously. As I blogged about recently on the minor change in policy by the DMA (Direct Marketing Association), in America at least, congress is still allowing the DMA PAC (political action committee) to rule them despite the fact that spam is one of the most hated things by their constituents. The technical solution is clearly not easy and not working and our governments are failing us in a big way.

I wish the whole blogging community got together and put pressure on our local governments to make this all illegal w/ stiff penalties. I really feel that until we the people don't put enough pressure on, nothing will change much...

And the irony is, that the DMA's self-serving policies, don't really help themselves, because their own members emails get lost in the spam...

Here is a site that talks about spam filtering using Bayesian statistics.

Leave a comment

About this Archive

This page is an archive of recent entries in the Business and the Economy category.

Books is the previous category.

Computer and Network Risks is the next category.

Find recent content on the main index.

Monthly Archives