Internet News
Report: ISPs Block 17 Percent of Legit E-mail By Brian Morrissey

Top Internet service providers blocked 17 percent of legitimate permission-based e-mail in the first half of the year, according to a report issued by Return Path.

via Scott Mace

I pronounce email officially broken. If 17 percent of legit email is being blocked by spam filters, it's not officially working. No wonder I'm using blogs, IRC and IM for my primary modes of connecting with important people these days.

I don't care what excuses people give. The people who made smtp should have thought more about host authentication and the people who made IPv4 should have made longer IP addresses. My guess is that there were people who were voicing concerns who had more vision.

I have a feeling we are going to be kicking ourselves in the same way when we realize we "forgot" to put privacy into ID systems.

23 Comments

Right on. The missguided morons over at SPEWS.org just assumed that my hosting ISP is a spammer and they are now filtering out MY email with others. There is no way to refute or prove your innocence. It is a system based on massive false positives. If you are pissed at some one all you have to do is tell spews that they are a spammer and they are blacklisted with no investigation or reveiw.

But, but, all smtp servers can be configured to require username/password.

The article did not talk about 17% of all email being blocked, it talked about 17% of commercial messages being blocked without specifying whether they are solicited messages or not. Since this is regarding marketing email, it may well be spam to begin with. The article did not explain how this figure was derived so is the statistic even credible?

Next of all shame on you for speculating about the origins of SMTP&&IPv4 w/o doing your own research into the history. I know that you have been around long enough to know better.

I can only echo Chris_B's sentiment. I think it's amazing IPv4 and SMTP have remained more or less the same for the past since 81-82. When specific requirements outgrow current cabilities, it's time to update them. Also there are many articles about how IPv6 is being ignored by many U.S. ISP's even though it's been around for 5-6 years.

Hell it's sad the automobile is basically the same as it was since it was created.

Misanthropyst: But you can't configure a computer not to receive mail for an unauthorized host if it is addressed to you. What you need is to be able to ask any host connecting to port 25 for some sort of authentication (probably a certificate or something) and not allow a connection if it is not certified.

Chris and Ed. Maybe I'm being a bit rough on the IPv4 and SMTP guys, but my WHOLE POINT IS that once you set a standard, it's not going to change for awhile. I am going through this right now with the damn National ID in Japan. "Don't use 11 digit numbers. We have much better technology and you KNOW the risks." I can't say for sure about IPv4 or SMTP, but I DO know that I was thinking about security in the 80's. I've been ranting and raving about security on deaf ears since the early 90's. I assume some people in power at the time could have predicted spam or running out of IP addresses. Someone probably said, as you're saying, we can always change the standards when we need to. The whole point is that you can't.

A lot of the folks thinking about identity standards today will say, we can add privacy later or we can retrofit it. We can't. We will be like everyone pushing for IPv6 right now. Lots of good reasons, but switching cost is too high. Same with smtp. We should change it to deal with the spam issue, but we can't. Too much installed base. So Ed, "I think it's amazing IPv4 and SMTP have remained more or less the same for the past since 81-82"... That's the problem. With this kind of adoption volume, you CAN'T easily change so you need to design standards to be extensible and make them as flexible for the future as possible.

I don't want to bash the people who built the Net or the standards TOO much, but we should learn from the pickle we're in and make sure that we envision many of the new standards that we are working will end up being used for a long time.

Joi,

Surely you remeber in the early 90s when the NSF concented to commercial use of the Internet? How could any of the designers of SMTP anticipate spam in an environment where commercial use was prohibited in the first place? The same goes for IPv4 for that matter. Computers of any type, much less ones which could participate in a network were only affordable by institutions or wealthy individuals. Not to mention the fact that any one of the many other competing networking protocols could have just as easily become the defacto world standard so picking on IPv4 in hindsight does not do much good. The same can be said for prety much any defacto standard. Things tend to get widely adopted because they are either relatively painless to implement or becuase everyone else has already done so. Secure protocols of darn near any type do not tend to fit either of those criteria.

As for certificate based SMTP, its doable today. TLS SMTP can be configured to provide authentication and transit security, but then we fall back to the age old problem of who issues the certificates in the first place? Its a rather massive chicken and egg problem. How can I trust your mailserver when there is not CA to trust in the first place? This does not even begin to get into issues of who maintains Certificate Revocation Lists either. What we have here is a human problem, not a technological one.

I'd love to see things done "right" just as much as the next guy, but until the human trust issues are overcome I just dont see it happening.

Yup, if you could anticipate all future requirements and plan systems centrally and in advance, we would all be logging into MVS hosts in the morning and SNA would be the networking standard. It just doesn't work that way. You can't plan that far ahead and not end up looking like X.400.

The big issue with certificates, quite apart from who would issue them, is what exactly they would certify, other than the identity of the sender. You can get a certificate for just about anything. Just look at Gator Corporation, the people who do the dodgy scam with forcing your computer to dial a premium telephone number. Their ActiveX plugin is certified but that doesn't mean it isn't a dangerous piece of rubbish.

Realistically, we're still stuck with IPv4 and unauthenticated SMTP for "some" time to come.

A fairly effective anti-spam measure that is compatible with today's infrastructure is already known and implemented at a growing number of places:

- When a mail comes in, the mail server mails back an automated acknowledgement request to the purported sender address.
- If the return address is fake, the ACK request will bounce. The corresponding original mail is then deleted and not forwarded to your mailbox.
- If an acknowledgement confirming a human ia present at the other side arrives within a prescribed time limit, the mail is forwarded, and the e-mail address is added to a database of "authorized" sender addresses.
- You must manually seed the database of authorized senders with the addresses of your friends, colleagues, automated reporting systems and mailing lists to which you subscribe. Acknowledgement-based additions to the database for new correspondants should in principle be automatic.

But then you would have problems receiving emails from friends or from legitimate mailing lists. The system is also open to deception by wily spammers.

As for replacing SMTP, that's probably not such a good idea.

If you want to stop spam, you need a law which stops it. That's right, a law. The law has to apply everywhere and it has to be enforced. Spam (in the broadest sense of attracting the attention of people who don't want to hear from you) is a social problem, and you won't solve it by technical means alone.

I will concede that "Realistically, we're still stuck with IPv4 and unauthenticated SMTP for 'some' time to come." I also agree that centrally planning everything in advance is BAD and is how you build big ugly unusable standards. I think extensibility and foresight are important. We're going through this with the whole RSS debate. How to keep it simple but extensible.

As for the founding fathers of IPv4 and smtp. I had my first IP address when I was logging into ARPANet via an account on University of London (I think) in 1984 or so. I was on an Apple II. There were a few hundred computers on APRANet probably. But there were lots and lots of people buying Apple II's and I think the Mac had just come out. I remember imagining where this was going and thinking, "gee, I wonder if we have enough IP addresses."

I also remember when I first started using UUCP then SMTP and thinking. "Hmmm... How secure is this?"

My point is that if some dumbass high school kid was getting the "hmm...'s", I'm sure someone smarter could have though of this. I'm not saying that they should have "figured it all out" but it seems like they could made things more extensible or architected it better. Having said that, they did a pretty good job and I'm not going to blame it all on them. I'm probably wrong in placing too much blame on them. I just wanted an excuse to try to shake my finger at the identity folks today and warn them that if they don't bake privacy protection technology in identity technology now, it will be hard to retrofit it.

Antoin, I read the link in your comment. It's a good analysis. My personal opinion is that they should stop supporting UUCP "bang" routing, which I think many machines still support. I think server authentication will still help, but I don't think we could get everyone to retrofit it at this point.

So my dream of how certificates should work is a multi-modal one. Your mail server should be configured to have a set of rules for each mailbox of what certificates are authorized. You could make a whitelist of personal certificates that you accept, CA's you trust (your school, your company), you could have web of trust systems (LinkedIn network, FOAF), or you could trust some central CA if that was your preference, or you could just upload your PGP public keyring or something. The server should look up the sender's key in the profile of the recipient. I suppose the recipient and the key should be part of some sort of exchange that happens before the sender is actually allowed to connect to the recipient mail server. This does tie in a bit to the whole idea of identity servers. Maybe that's when we should "change smtp" when we switch away from email addresses as our primary form of identification.

SPEWS blacklists entire ISPs who refuse to drop their spamming clients, but only after proper investigation and warning. The idea is that the affected "innocents" will exert commercial pressure by demanding their ISP stop hosting spammers, or they will move to a legitimate ISP. Whether this effort is doomed to failure is uncertain. If enough people subscribe to SPEWS' blacklists, it probably will be.

Joi I admire you sticking to your guns no matter what, however, I think you are either being too stubborn here or you are not understanding some things.

As for your self congratulations on your teenage revelation, 1) I'm pretty sure that Apple ][ was at best using a serial link to a unix/vms/etc host. I cant find any evidence that there were any type of network cards for the Apple ][ in 1984. My point here is you were not using an IP address, the host you were connected to was. If you understood that and still forsaw a shortage of IP addresses, you were indeed smarter than the average bear.

"Webs of trust" are great when you trust the people in the web. PRZ has commented extensively regarding why he chose to use this model over a central key signing system. Out of the hundreds (thousands?) of people I've exchanged emails with over the years, only 3 people have ever used PGP to secure email with me and none of them requested that I authenticate the fingerprint of my key. This means that even paranoids cant be bothered to authenticate the sender of a message most of the time.

Even now that we can have organizations sign individual PGP keys, and there are standards to choose from for X509 signed or authenticated message transfer, have you thought out the issues involved with this? I dont trust the US govt to issue even a basic certificate beyond a passport. I certainly dont trust the clowns in Nagatacho to do even that well. Given that most people who use email are more trusting of their respective and many other governments, but you may understand my position here.

As someone mentioned before, there is the question of what is being certified? Identity? "Trustworthyness"? More than likely just that the requester has filled out their paperwork and paid the CA a fee. Even if something like a drivers license for sending email were created and recognized across borders, ("The State of New York Certifies that Chris_B meets the minimum requirements to send email on public networks until the expiry date below"), you can pretty well bet the system will be subverted faster than you can say apple pie. Wont take long before some local mafia somewhere figures out that the money to be made sending spam is much more than the cost of bribing a clerk to issue a license or the cost of just stealing people's email licenses. Who is liable in the case of a stolen identity being used to send spam? Who says anyone has to recognize certs issued by "rouge/failed states"? Or even France for that matter.

Lastly there is the anonymity issue. I for one prefer the benefits of anonymous free speech over the burden of deleting/filtering unwanted emails. Who is going to officially certify me to be anonymous? I would hope anyone reading this understands what I'm talking about.

I've gone on long enough in this post. If I have not made my points about why crytpo does not even come close to solving trust issues, please read some of what Bruce Schneier has written. As you probably know by now, I'll take working SMTP over a large scale trust model any day of the week.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Chris, thank you for admiring my stubborness.

You are right. The IP address I had was the address of the host that
I was accessing from. I was using a 300 baud acousting coupler on my
Apple II. I did imagine that my computer would someday have an IP
address though.

As for PGP. I agree that it has its problems, but I like the
architecture better than centralized systems. I personally use it
just about every day and have done probably close to 100 key
signings. I have my PGP fingerprint printed on my business card. My
attorney and my accountant both use PGP to communicate with me. So I
guess mileage may vary, but I couldn't live without PGP.

I agree that centralized CA's don't make a lot of sense and can be
subverted, but a network of CA's where you could determine which ones
you trust is probably not a bad architecture.

I agree that anonymity is important. I think there are cryptographic
methods that can prove and identify certain attributes without
identifying you. I think that this is a very interesting area of
research. I agree that we can't afford to lost anonymity just to get
rid of spam, but I'd like to try to think about alternative ways to
assure anonymity.

I'm a big fan of Bruce's and have read most of his books and ready
crypto-gram. I realize trust is a hard problem, but something we need
to deal with.

-----BEGIN PGP SIGNATURE-----
Version: PGP 8.0
Comment: http://joi.ito.com/

iQA/AwUBPzxnoeSNryotlGHxEQKP7wCg64rlgjvzq2ksjp6QeU7rZSv8W/IAoPkL
YXfTcEvksmsqrdUh32e5gc2L
=qfwU
-----END PGP SIGNATURE-----

;-)


Antoin O Lachtnain wrote: But then you would have problems receiving emails
from friends or from legitimate mailing lists. The system is also open to
deception by wily spammers.

If your friends whose addresses were unfortunately not manually seeded in
the "authorized address" list can't even be bothered to hit "reply" to an
automated ACK request, then I guess they don't really care about communicating
wtih you...

The system will effectively block a large number of automated mass-mailings,
as it enforces a replyable -- and possibly trackable -- originator address.
Spamming economics require automation, and having a human reply to the ACK
requests would thus make most spamming schemes unfeasible.

As to legislation banning spamming, sophisticated perpetrators could just
move spam origination to the hijacked machines of unsuspecting broadband-
connected Windows PC users, or even overseas -- e.g. a ship in international
waters linked by microwave/satellite/long-range 802.11 wireless.

Joi, I think you don't understand the problem.

Spam isn't the fault of SMTP, or unsigned IP packets, or of not having automatic replies.

Spam is the fault of _spammers_.

They have access to the same technology as you do. What makes you think that requiring a signed certificate to send e-mail to a server is going to stop a spammer for more than 5 minutes?

What makes you think that requiring an identity server would stop a spammer for more than 5 minutes?

The root of the problem stems out of this simple fact: anyone can e-mail anyone else.

The trick to stopping spam is to make it unprofitable for the spammer. That is generally done via the law. I would gladly accept a blanket law against spam, and against companies who contract with spammers. Spam someone else? $500 per e-mail. Is your company the one being advertised? $5000 per e-mail.

The Only Solution to Spam: Money

Only money can eradicate spam. The Internet is a globally distributed ecosystem complete with evolving organism/organizations that continuously adapt to change. The current spam epidemic is proof. Moreover, no legislative or technical solution (i.e. filters) will be able to stop it. Why? Because spam is fundamentally an economic problem.

Ross Mayfield and I attempted to get an anti-spam company off the ground two years ago based on this fact. Yet the noise of emerging technical solutions and lack of insight by "leading" venture capitalists reminded us that it takes more than being right to build a company. The current spam explosion is proof that technical solutions are only making the problem worse.

So here I go, I'm giving the world the answer. It's simple in theory, but incredibly complex to pull off in reality.

Put a price on your inbox. No email gets into your email inbox unless it has a dime attached. I pay you a dime the first time I want to communicate with you, and from there until infinity you and I can share that same dime back and forth. No money, no entry. This fundamentally shifts the economic cost of sending email back to original senders. Think a spammer would spend $100,000 to reach 1 million people now?

So there it is. Go build it, so we can all get on with our lives...oh and by the way, you need to be able to scale globally and have multi-currency functionality in 90 days or the system won't get adopted. Want more? The business plan is done. Just need $5m. Any takers?


http://www.corante.com/brainwaves/20030501.shtml#32424

Hey! I seem to have received about 20% less spam than usual yesterday. Could a solution to the spam problem be as simple as cutting off electrical power to a significant portion of the US? ;-)

I like Zack Lynch's idea of putting a price on your mailbox. It could perhaps be tied in with the "micropayment in exchange for information" schemes which could monetize blogs. Web pages and e-mails are both avatars of an information exchange process, anyway.

Zack Lynch on August 16, 2003 07:16 AM wrote:
>The Only Solution to Spam: Money

You have absolutely no imagination, and the only measurement about human being for you is money. How poor is the USA society !

The Internet is, by design, blind to content. Backbone carriers, and to some extent ISP's, make money from each bit they pump. We may soon be approaching the point where MOST of the bits on the Internet are bad bits: spam, viruses, hoax virus warnings, and terrible old jokes. But the carriers will still make money from them!

This situation will be stable so long as enough good bits get through, to deliver enough value, to enough people, to make them continue to pay for the bad bits as well.

The situation is a bit like 'junk DNA' in the genome. If that is any guide, we could see 97% noise and 3% signal in few years time. But the system should still work: it will 'eveolve' its own way of filtering out the noise.

So precisely how is SMTP broken?

The duity of SMTP is to transfer mail... which it does extremely well, right? So where's the breakage?

That ISPs block mail is not the fault of SMTP, it's the fault of the ISPs deciding to block it.

That spam gets in the system is not the fault of SMTP, it's the fault of people deciding that they want others to be able to send email (what's the point of email itself of this isn't true?).

With IPv4 you might have an argument, but SMTP is clicking along doing exactly what it's supposed to do. Don't pronounce it broken because you decide you don't like what you told it to do.

I like Mark's idea. Put a fine on both spammers and their contractors!

I mean, if you pay someone to kill someone, you're just as responsible for it right?

So if company solicits a spammer, the company should have to pay a fine, if the spammer is not found by law enforcement, the fine should be double (ie the company pays the spammers share of the fine).

There should be lots of media attention and special websites where people can go to report spam. Basicly an easy way for the law enforcement agencies to keep track of what spam comes from where so they can apply the fines.
I kinda like the idea, and if companies would have to pay 500$ or 5000$ per reported spam message, I am sure as hell many would shy away from it.

Also since the company itself is responsible for the spam, the spammer would not just be able to hide in a weird small country (although most spammers seem to live in trailers in the USA heh), since the company would be liable anyway.

Even if this law would be implemented in the US only, I believe it would be effective since most spam comes from there anyway! heh

I don`t think that email is broken. It`s really needed for many people who use it.

The big issue with certificates, quite apart from who would issue them, is what exactly they would certify, other than the identity of the sender. You can get a certificate for just about anything. Just look at Gator Corporation, the people who do the dodgy scam with forcing your computer to dial a premium telephone number. Their ActiveX plugin is certified but that doesn't mean it isn't a dangerous piece of rubbish.

Leave a comment

11 TrackBacks

Listed below are links to blogs that reference this entry: Email is officially broken.

TrackBack URL for this entry: http://joi.ito.com/MT-4.35-en/mt-tb.cgi/938

It makes me wonder. Joi Ito declares today -- Email is officially broken Reading the associated links it makes me wonder if excessive email filtering is actually constraining economic activity? Read More

Joi Ito's Web: Email is officially broken Well, here's my take. E-mail is modeled on postal mail. Anyone can send you something as long as they have (or guess) your address. Both afford us much spam/junk mail. I get 200 Read More

Joi Ito asks an important question. Is email broken? he thinks so, because of a new study by Return Path showing that on average, 17% of e-mail sent by their e-marketing clients is being blocked. Joi termed it legitimate email because Return Path did, ... Read More

Drawing the Line from Take the First Step
August 14, 2003 11:39 PM

One of the hardest tasks in developing new systems is knowing where to draw the line 〓include too much and you'll never finish, include too little and all you have is a toy. So I think that Joi is being a bit harsh in his guess that there were people... Read More

Joi has declaired e-mail to be officially broken and points to this Internet News story by Brian Morrissey as his... Read More

Email is Dead. from Ross Mayfield's Weblog
August 21, 2003 3:13 AM

Wired has a good article on how Aggregators Attack Info Overload, a perfect excuse for me to go on a rant on how information overload will kill email lists within the short term. I have posted at length on Blogging Read More

I think Joi Ito started it but lots of bloggers have been picking up on the meme that yes, email is indeed irrevocably broken. Gary Turner puts it in a nutshell and Tim Bray posts thoughtfully about the problem. Personally I haven't had any trouble wit... Read More

this is a very active topic in the front of my little brain, so it's a bit refreshing to find it at least resonating with some others (so, thanks, crystal flame - is that two words?) shortly after i made... Read More

this is a very active topic in the front of my little brain, so it's a bit refreshing to find it at least resonating with some others (so, thanks, crystal flame - is that two words?) shortly after i made... Read More

this is a very active topic in the front of my little brain, so it's a bit refreshing to find it at least resonating with some others (so, thanks, crystal flame - is that two words?) shortly after i made... Read More

Trackback Is Broken from AKMA’s Random Thoughts
February 7, 2005 10:02 AM

Listen, children, I remember way back when your Uncle Joi pronounced email officially broken. We’re still using email two years later, but today I shut down Trackback on this blog; I like Trackback when it links two related blogs, but the unmoder... Read More

About this Archive

This page is an archive of recent entries in the Business and the Economy category.

Books is the previous category.

Computer and Network Risks is the next category.

Find recent content on the main index.

Monthly Archives