Joi Ito's Web

Joi Ito's conversation with the living web.

I'm at a dinner where we're talking about spam. There are high level execs from many of the companies involved in email. One person said that he thought we've seen the worst of spam and that it's getting better. It's too bad I can't quote people with attribution, because I think this is a totally unreasonable position.

We've now moved on to Internet governance and as usual, I haven't heard a single opinion that convinces me that email isn't broken and that it isn't just getting worse. We talked about pay to send, better filters, re-inventing smtp, regulations... all of the usual. Yet another fruitless discussion about spam. (yafudas). 17% of legitimate email is not delivered. 81% of people in a recent survey are afraid of false positives.



Where did you get your stats from?

Yes, spam is definitely frustrating. But it’s surprising (and equally, if not more, frustrating) that the computer industry isn't open to novel ways of dealing with the problem. I've emailed a few VCs about a solution that I've been working on and no one has gotten back to me. I guess you have to be a "high level exec" to warrant a response. Of course, I'm assuming that they received my email in the first place, which may not be a valid assumption ...

"Seen the worst of spam" The weekly amount of spam I get has now crossed the 1000 emails barrier and it's still on the rise.

I have a sad feeling that we won't have seen the worst of spam until there is a MAJOR melt-down of e-mail. We may be leading up to it, with all the talk of virus-infected machines used as spam clients. It's scary to think of what could happen if things get even worse than they are now.

I blogged about a solution a little time back that Microsoft, of all companies, is working on and I think it is a great solution. Basically it is a new protocol. Challenge-Response type deal, but built into the protocol. The way it would work is that the receiving mail server would send a mathematical problem to the sending email server. Something that would "cost" processing power. I'm not sure if the math "problem" is something configured by the mail server or the client, although it sounded like the client. Let me try to dig up the reference:

OK, found it:

I hate to sound like an apologist here but I can empathize with the execs who think the problem is solved. E-mail spam (as opposed to blog comment/cell phone spam) has moved for many people to being a supreme daily annoyance to something that's still annoying but very manageable.

Lots of companies have installed filters on their corporate mail systems. Commercial offerings like yahoo mail or hotmail added their own filters. Consumers went out and bought filters that help a little. Personally, I think the MS Outlook 2003 filters are great (full disclosure, I work for the company). My 8 year old hotmail account with a 4 letter username and my other accounts dating back 12 years (with old newsgroup posts and the such in google) bother me with up to 5 pieces of junk mail a day, the other 230 or so are filtered.

False positives are a problem but the reality is that if a message/contact is that important you would still call/IM/physically meet/blog with that person. Spam filter settings can be tuned and you can look at your filtered stuff before deleting it to see if people/mailing lists need to be whitelisted.

I think we see the biggest problems when companies or individuals are inflexible about how they deal with spam by using silly blacklists or not allowing whitelists.

There's no question that we'd rather not have any of it, but the reality is that as long as e-mail is almost completely free (both as in speech and as in beer) there will be attempts at commercial exploitation. I find it strange how luminaries like Lessig or other influential and intelligent individuals who run in the tech elite circles love to talk about how wonderful it is having an unintelligent, unregulated end to end network but at the same time seem to forget that spam is a result of the unauthenticated network. Spam is the tragedy of the commons on the Internet and the solutions lie in smarter clients and not a smarter network.

Well, how about a real anti spam law? Like making it illegal and suable to send spam unsolicited? The "Anti Spam" law just made the problem worse. It legalized spam in the states where you used to be able to sue a spammer. And now if someone sends a child a nude picture that child has to "opt-out"? What a joke. That law is a tool of the Marketing Lobby and congress has been their tool all along.

I have an idea that will cut down worldwide spam by 50 to 80% without the need to introduce any new fancy SMTP protocol or challenge-response or certificate-based malarkey:

Simply reject all the incoming SMTP connections from the IP address ranges of all the broadband ISPs who allow their cable/DSL subscribers to open SMTP connections to arbitrary hosts.

Spam is an issue that can also be fought at the spammer's ISP level. One way to facilitate this would be to force the spammer to use his/her ISP's own mail servers, thus leaving an actionable audit trail.

The cost caused by spam, incurred by the non-spamming Internet community, far outweighs the loss of convenience to the minority of users who do not want to use their ISP's SMTP gateways.
People who *really* need to use e.g. their office's SMTP gateway instead of their ISP's should install VPN software -- somewhat inconvenient, I realize, but *their* inconvenience is not worth that of the community at large.

There *are* already ISPs in the world who prevent their DSL/cable customers from using any other SMTP gateway but theirs. The scheme is *workable*.

I agree it will help a lot. But they'll just use other tricks (I think).

I really think the protocol needs a serious update, be it Microsoft's or some other scheme.

Hmmm... i think vowel's arguments will never work... when all ISPs block specific protocols at port 25/26, why should I not sell SMTP gateways from Armenia that can receive their data via port 80666 ?.

The general problem is SMTP and other sucky protocols itself... the whole full-commercial internet of 2004 is based on protocols that were invented in the 70ies or so - by really genious (for 30 times ago's time) university dudes ...

what happened the years after? nothing ... nobody really introduced a better, more traceable and reliable protocol since then... why? because everybody has smtp and it works "fine" ...

I believe that a total smtp e-mail collapse will happen some day and force us to use better protocols... without the pressure of not being able to send "real" mail nothing will change...


Christoph misses the point -- spam is not an issue of SMTP itself, it's an issue with how flexible SMTP is.

Allowing random dialup addresses to open an SMTP connection to your mailserver, and then claim to be you, is the big hole that the spammers get to drive through.

MostlyVowels is correct; closing up these massive, gaping holes in the wall is what'll help make spam a more manageable problem (note: the other aspect is that it's a social problem, and it'll never go away completely. But right now it's waaay out of control.)

BTW note that we don't have to throw out SMTP to do this. This is incredibly valuable, given the massive installed base out there.

whoops -- forgot half of my point! I said:

'Christoph misses the point -- spam is not an issue of SMTP itself, it's an issue with how flexible SMTP is.

Allowing random dialup addresses to open an SMTP connection to your mailserver, and then claim to be you, is the big hole that the spammers get to drive through.'

Anyway, my point is that even if we replace SMTP with a bright shiny new whizzy protocol, *unless* that protocol fixes that hole, and many others, we'll have spam.

And right now, it appears we can do the hole-fixing within existing SMTP -- using add-ons like SPF.