A Plan For No Spam

INTRODUCTION
Unwanted and irrelevant mass mailings, commonly known as spam are becoming a serious nuisance that if left unchecked may soon be regarded as a Denial of Service Attack against the email infrastructure of the Internet itself. The term spam is derived from a Monty python sketch set in a cafeteria in which the principal protagonists have difficulty making them heard about a group of Vikings singing. "SPAM" in honor of the meat product manufactured by Hormel.

A Plan For No Spam

IDEAL SPAM CONTROL SYSTEM PROPERTIES

An ideal spam control would have the following properties Eliminate all unwanted mails Eliminate no wanted mails Require no user input on the part of either the sender or the receiver. Be compatible with all uses of mail. Be compatible with all email infrastructure configurations. Be scalable, that is remain effective if 90% of internet users adopt it. Resist attempts to evade it. Create no new problems. No perfect spam control solution has been found so far. Filtering approaches are compatible with a broad range of email uses and infrastructure but no filter perfectly identifies even a fraction of unwanted emails without eliminating at least some wanted emails. Further more the more widely a filter is used the greater the incentive becomes for the spam senders to test against it to ensure that their spam gets through.

A Plan For No Spam

TOWARDS SOLUTIONS
There are many types of spam and it is likely therefore that any realistic program to eliminate spam will have to address the problem in multiple ways. In particular we should not reject a mechanism out of hand simply because it fails to deal with a particular type of spam sender .For example it is highly unlikely that spam senders engaged in organized crime will respect Opt-Out lists or Legislative approaches. We should not however, we conclude from the fact that a mechanism fails in certain circumstances that it fails completely in all circumstances. The only mechanisms that rule out of hand are those based on vigilante actions that attempt to gain compliance by coercion.

A Plan For No Spam

BEST PRACTICES

The traditional response of the internet to problem uses administrators of deployed protocols is to specify some form of 'Best Practices'. Spam is an attack on the Internet community. The short survey and prosecutions by the FTC and others show that the spam senders are in many cases outright criminals, how then can best practices help? One area in which best practices can provide concrete benefit is in ensuring that the vast majority of Internet users who are acting in good faith do not inadvertently make the problem worse by poorly chosen or poorly coordinated mitigation strategies. Many of the problems being caused by irresponsible blacklists could be addressed through common agreement on best practices for blacklists setting out criteria for issues such as notice to the listed parties, appeals processes and the acceptability of 'collateral damage'. Best practices can also provide backing for legislative approaches. An appropriate statement by an authoritative body can help a court to decide that a particular form of behavior is unacceptable, even if such statements do not have the endorsement of a legislative body they can quickly acquire the force of law when the worst malefactors are being dealt with.

Acceptable Use policies

Perhaps the most important anti-spam measure to date has been the adoption of acceptable use policies that prohibit sending of spam by Internet service providers of all types. Such use policies do not just protect the other users of the Internet; they protect the Internet service provider against abuse by their own users. In a recent case a university discovered that some of its students had been renting out the use of their machines to spam senders, seriously degrading the network service for all the users.

A Plan For No Spam

Pull Vs Push
One of the major problems caused by spam is positive identification as spam of legitimate emails. This is a particular problem with commercial mailing lists and newsletter messages requested by the recipient which frequently carry advertising and make use of content that is likely to be flagged by content inspection filters as spam. While some have argued that the commercial nature of these emails means that this type of false positive is unimportant it seems odd to consider the loss of messages specifically requested by the user to be less important than the loss of unsolicited personal emails. The core problem with these messages is that although the sender and the recipient know that the user has consented to receive the messages, there is no way for that consent to be communicated to the mail infrastructure that transports the message. One possible solution to this problem would be to add some form of authentication mechanism that communicated the recipient's consent to the mail infrastructure .This is unlikely to prove practical in deployment however since any benefits would require changes to all parts of the email infrastructure-clients, originating servers and receiving server. Another option would be to distribute this form of content using the pull model of the Web [HTTP] and NNTP [NNTP] rather than the push model of the email. In the pull model, no content is sent until the client specifically requests it. Consent is thus implicit in the pull request since the client will not request content unless directed by the user. For the pull model to become practical a number of difficulties must be overcome. The most important of these being obtaining a commitment from the major software client to supporting the protocol to the commonly used email clients. A mechanism is also required that tells the client where the updates will be available from, the expected frequency of those updates, the format etc. The Really Simple Syndication (RSS)
5

A Plan For No Spam

Protocol [RSS] appears to have these properties, although it is currently being applied in a different way.

Opt-Out and Do-Not-Spam lists

Many spam messages carry a 'click here to Opt-Out' option. The research by the FTC has demonstrated that large proportion of these Opt-Out options are bogus [FTC 3] and it is not possible for the user to connect to the Opt-Out site. Opt-Out schemes of this type are unacceptable however, even if spam senders observe them since the spam victim is required to Opt-Out with each spam sender individually. This is impossibility if the victims email address has been listed on a CDROM, that has been sold to several thousand spam senders. For Opt-Out to be practical it must function in the same way that state Do-Not-call telephone lists do, the address owner Opt-Outs out once and it is the responsibility of the spam senders to ensure that their lists are clean. Opt-Out lists are proven to be of value in controlling the volume of unsolicited mail and unsolicited telephone calls from legitimate businesses. We know that a significant proportion of spam is sent on behalf of legitimate businesses that may be considered likely to respect such matters. Certainly a mechanism of this type widely deployed would deprive spam senders of the claim that they respect Opt-Out requests. The most powerful argument against Opt-Out lists is that it is likely that spam senders would routinely abuse lists of Opted-Out email addresses as the source of email addresses. This objection may be addressed by appropriate use of cryptography in which the entries in the lists are obscured using a one way message digest function such as SHA-1.The Opt-Out list consists of a sequence of message digest value to permit rapid lookup by binary search or similar means.

A Plan For No Spam

Accountability
One of the major difficulties faced with distinguishing legitimate bulk senders from illegitimate ones is the difficulty of determining whether the claims made that the bulk sender observes a particular set of best practices is true or not. This problem also faces the bulk senders themselves. At present there is no way that a outsourced provider of bulk mail services can tell if the email addresses on a list a customer wants used for a mailing have intact Opted-In as is claimed. Technical mechanisms are required that enable this form of accountability to be achieved.

A Plan For No Spam

CONTENT INSPECTION
Aspirin will not cure a cold but it will relieve the symptoms and make the cold more tolerable. Content inspection has a similar effect on spam. The symptoms of spam are relieved to a considerable degree, but the patient is still under attack from the infection.

Content inspection is a form of spam filtering that uses the content of the messages as the basis for the decision to filter. One of the principal difficulties with content inspection mechanism is the ease of evasion by the spam senders. If the spam sender knows the criteria applied by the Content inspection technique, then they can construct their messages so that they are not caught. There are many forms of Content inspection, each of which has advantages and disadvantages.

Naive Keyword Inspection

Messages are scanned for the presence of words or phrases that occur frequently in spam messages such as HGH or multi-level marketing. This type of filtering is implemented in many common email clients such as Outlook [MSFT]. Keyword Inspection alone is simple to implement but tends to have very high rate of false positives.

Naive Language Inspection

The Internet is an international medium and spam is sent in many languages. As a result a large number of spam messages are incomprehensive to the reader. While fully understanding the meaning of the mail message is a complex problem that is AI complete, that is it requires a solution of the artificial intelligence problem, detecting the language that the email is written in is
8

A Plan For No Spam

considerably easier. In many cases the character set in which the email is sent may be used as a proxy for language, although this technique can result in false positives as a Japanese user may send all his messages in a Japanese character set, using the ASCII subset for messages in English. Another problem with language inspection is that a large software vendors would face a considerable problem from foreign users of their product if they introduced language sensitive filters in their products.

Keyword Inspection with Statistical Techniques

The effectiveness of Keyword Inspection can be substantially improved if combined with Statistical Techniques that access the probability of a message being used as a spam based on the presence of multiple keywords. Various techniques may be used for this including Bayesian Inference and least square approaches.

Keyword Inspection with User Feedback

There has been considerable recent interest in content inspection mechanisms that employ user feedback, in most cases in combination with some form of Bayesian Inference. This approach provides some resistance to spam sender counter measures since the individual users maintain separate databases for desirable and undesirable messages. The drawback to this approach is that it requires user intervention on a per message basis which experience demonstrates limits the effectiveness of the approach severely.

Keyword Inspection with Dynamic Update

Another approach to improving the effectiveness of keyword inspection is the combination of Statistical methods with an online source that provides regular Updates. This approach is frequently combined with the Template approach described below.
9

A Plan For No Spam

Dynamic Template Response / Fuzzy Matching / Checksums

Dynamic Template Response uses templates or 'finger prints' of known spam messages to identify spam messages. The templates are constructed using the spam messages sent to 'honey pot' email addresses, either by hand or using some form of automated tool. While this technique can be very effective it is also costly to maintain, particularly since the spam sender may employ counter measures taken by the spam senders.

10

A Plan For No Spam

AUTHENTICATION AND AUTHORIZATION

Practically all spam messages sent today attempt to evade anti-spam measures by use of false header information. None of the spam messages that were examined in the writing of this paper carried a genuine sender address. Most of the massages contained from addresses that were obviously fake. In some cases the addresses were not even valid. Some contained no sender address at all. This suggests that a robust method of determining the false sender addresses would provide an effective means of eliminating spam. Detecting false sender addresses would be a simple task but for the fact that the SMTP protocol allows the sender to forge a message that purports to come from any sender. The use of forged email addresses is currently rare but becoming more common. A particularly insidious spam sender trick borrowed from the Klez computer virus is to harvest email addresses from the mailing lists archived on the web and send a large number of emails purporting to come from one member of the list to other members of the lists. The use of forged email address is likely to become very common if the use of anti-spam filters that detect missing or obviously false from addresses become common. To address this problem, some form of authentication scheme is required that provide an unequivocal proof that the sender address is valid. A Spam sender might attempt to circumvent an authentication scheme by sending messages with a legitimate, authenticated sender address. We find this objection to be a weak one since it is clear that Spam senders have a considerable motivation to conceal their identity. If every email carried an authenticated sender address spam senders would be forced to obtain a new DNS address frequently to
11

A Plan For No Spam

conceal their identity. This causes increase in costs of spam senders & reducing overall profit of the spam sender enterprise.

Authentication
Authentication techniques are broadly divided into two types, Network based and Crypto graphically based.

Network based
If a mail server knows the set of all possible IP addresses from which an email with a particular sender address is known the IP addresses may be used to provide a light weight means of authenticating the email sender. This mechanism is not completely reliable since an IP address is a valid source for a particular sender address. Some mail servers use the reverse DNS, which maps IP addresses to domain names for this purpose. This approach only works if the email is sent via a mail relay that is configured with reverse DNS entries for the domain of the sender address.

SMTP Routing information:

An SMTP message carries information that describes the path it has taken from one mail server to another.

Message Identifier Matching:

A mail service may use the In-Reply-To and References headers to identify messages that are replies to messages that originated at that service. Construction of the original message identifiers using a secret key and Message Authentication
12

A Plan For No Spam

code allows this to be archived without the need to maintain a list of all message identifiers issued by the service. Callback loop, Challenge/Response: When an email is received a message requesting confirmation is sent to the purported sender address. If the confirmation message is received the sender address is considered to be authentic. The callback loop mechanism is unusual in that is an active authentication mechanism that is applied at the request of the receiver rather than being applied passively to every message by the sender.

Cryptographic SSL

An extension to the SMTP protocol allows the use of the Secure Socket Layer via the START TLS operations allows authentication of both the sending and receiving email servers using X.509 digital certificates. Although SSL allows the email relay to use any IP address without the need for configuration of a reverse DNS address the sender must send their outgoing mail.

Cryptographic S/MIME

S/MIME provides end to end authentication of the sender address and message body. The sender need not send their message through any specific email relay.

Authentication Policy
For passive authentication mechanism to be useful as a means of detecting forged sender address it is necessary to know whether a purported sender has a policy of using authentication. Otherwise the recipient is unable to distinguish a message from a user does not use the authentication mechanism from a forged

13

A Plan For No Spam

message purporting to come from a user who always uses an authentication mechanism. The Internet architecture does not include a mechanism designed for the purpose of communicating security policies. Fortunately the DNS architecture provides the necessary functionality and may be readily adapted to the purpose without putting an undue load on the DNS. The Security Policy Advisory Mechanism provides a set of DNS extensions that address this need. The principal disadvantage of using the DNS as a means of communicating security policies is that the DNS itself is insecure. Fortunately the security weakness of the DNS do not lend themselves to exploitation on the scale necessary to make exploitation of these weakness a viable means of defeating an authentication based anti-spam measure.

Revocable Credentials

Cryptographic authentication schemes such as S/MIME or SSL use X.509 digital certificates as credentials.

14

A Plan For No Spam

LEGISLATION AND LITIGATION

The purpose of criminal legislation in a democratic is to deter persons from engaging in prohibited conduct. While it is unlikely that the criminal legislation alone would eliminate spam. Such legislation would certainly create a deterrent for both the spam senders and the advertisers seeking their services. The legislative process is very slow & time consuming. Legislators are reluctant to pass any legislation until they are confident that the implications are fully understood. Legislators will have to be convinced that any new legislation to address the problem of spam will bring benefits that significantly outweigh both the cost of enforcement and the political cost of committing the scarce resource of legislative time to the problem of spam rather than to other pressing problems.

Prosecutions and Litigation under Existing Legislation

As previously noted a substantial proportion of spam is illegal under the existing laws. Spam senders also violate law in their methods. Use of false sender address is an attempt to gain access to a computer system by fraud. A false sender address that impersonates another party is a form of identity theft and may also be actionable as defamation.

Spam Litigation
Spam has spawned two types of litigation, cases caused by ISPs & individuals who object to the use of their resources to send spam and cases caused by spam senders to prevent ISPs from cutting off their resources when they violate terms of use.

Anti-spam Legislation

15

A Plan For No Spam

In May 2002 the European Union issued a directive that directs member nations to introduce Legislation that prohibits sending of unsolicited marketing messages unless the recipients 'Opt-in'. A number of anti-spam measures have been proposed in the US congress that seeks to regulate spam by requiring spam senders to respect 'Opt-Out' requests and prohibiting the use of forged sender addresses. Most proposals are based on existing legislation concerning junk faxes and telemarketing calls that have already been tested with constitutional challenges. The chief objection to the 'Opt-Out' measures is that a recipient of a spam has no way to know the origin of a spam. It is therefore impossible for a spam recipient to know if the Opt-Out requests are being respected or not. While email spam senders have negligible support in Congress, the Direct Mail Association (DMA) that represents centers of junk mail and telemarketers has demonstrated a considerable degree of influence opposing opt-in requirements to protect privacy. The spam senders compete with the DMA members and so the DMA is opposed to spam but will oppose any measure requiring opt-in which might set a precedent that may later be applied to its members. One possible resolution of this problem would be to require spam senders to respect a one way encrypted opt-out list of the type described earlier. Such a list would meet the need for verification without threatening the interest of DMA and its members.

Pro-Spam Legislation
Not all the anti-spam legislation being proposed is intended to solve the problem of spam. Some of the 'anti-spam' proposals being raised in private are really 'pro-spam' legislation intended to solve problems for senders of spam. A common tactic used in the US to derail popular legislation is to introduce legislation at the federal level that proposes 'harmonization' of stateless that does
16

A Plan For No Spam

so by pre-empting the existing state laws with weaker or sometimes completely ineffectual federal laws.

Another potential tactic is to attempt to force ISPs to accept all mail that does not meet some definition of spam. If that definition is sufficiently loosely worded, ISPs might become legally obliged to accept any email content, however objectionable provided only that it was sent from a member of a particular association. It is therefore important that any spam legislation be carefully evaluated on its merits and not merely its title.

17

A Plan For No Spam

DEPLOYMENT
Content inspection based on approaches can be readily deployed but tends to loose their effectiveness as widespread adoption encourages spam senders to employ counter measures. Legislative approaches can increase the cost of spam senders by forcing them to employ costly counter measures such as moving offshore but are unlikely to eliminate spam together. Authentication based approaches provide a robust means of identifying messages that are not spam but are of limited utility unless widely adapted. This analysis suggests that content inspection, legislation and authentication are complementary approaches. Content inspection provides short-term mitigation of the effects of spam. Authentication provides a robust long-term solution. Legislation provides a means of slowing the rate of growth of spam so that the content inspection based approaches maintain their effectiveness long enough for the long-term authentication based solutions to be effective.

Protocol changes to facilitate spam measures

Previous sections have identified the need for mechanisms are that allow:Unsolicited call back request to be identified and suppressed.

Mail servers to communicate the results of server based authentication and filtering procedures to mail clients.

Mail clients to cause mail servers to apply filtering at the server Mail clients to disable server based filtering

18

A Plan For No Spam

Support for lightweight authentication mechanisms to be applied at the server level

19

A Plan For No Spam

APIs for spam filters

The definition of a standard Application Program Interface (API) for filtering modules would greatly simplify deployment and allow development of spam filtering mechanisms to take place independently of email client development.

Authentication mechanisms

We identify a need for the following:-

A means of advertising authentication policy through the DNS A lightweight authentication mechanism that can be deployed with minimal overhead Definition of a standard call back mechanism that allows a mail client to automatically detect call back loop requests that were initiated by the client and those that result from a message sent with a forged header

Mailing list management

Support for mailing list software in SMTP is less than satisfactory. We propose the following set of protocol changes to remedy this solution.

Mechanisms identifying messages sent through a mailing list and the means by which the subscription can be cancelled.

20

A Plan For No Spam

Mechanism that allows a mail server to determine that an email user solicited a message from a particular mailing list or other bulk sender

Authenticated subscription management mechanism Mechanism for communicating the authenticated mailing list subscription mechanisms to email clients.

Mechanisms for identifying automatically generated messages that should not be replied to under any circumstances.

Means of determining that an email client confirms to a consistent means of implementing the above

21

A Plan For No Spam

COMPREHENSIVE EMAIL MANAGEMENT

The various mechanisms described in this draft may be brought together to provide a comprehensive email management solution as shown in figure.

DENIAL OF SERVICE PROCESSOR

When the mail server receives a request to send an email, it first checks to see whether the email is being sent from an I P address that has attempted to perform a denial of service attack on the server by consulting the Denial Of Service blacklist.

AUTHENTICATION PROCESSOR
The mail server performs the authentication procedures that are available to it, including verification of I P address, use of authenticated SSL and S/MIME authentication on the message itself.

22

A Plan For No Spam

AUTHENTICATION POLICY PROCESSOR

If the sending domain has an authentication policy the mail server checks to see if the message meets the authentication policy. If the message passes the authentication policy or there is no authentication policy, the message is passed to the authorization processor. Otherwise the message is rejected.

AUTHORIZATION PROCESSOR
The authorization processor checks to see if the sender of the message is on the local white list of approved senders. If so the message is accepted, otherwise the message is passed to the content inspection module.

CONTENT INSPECTION PROCESSOR

The content inspection module uses the heuristic approaches described earlier such as keyword filtering, header verification, black listing and Bayesian inference to determine whether the message is likely to be a wanted message or Spam. The use of authentication techniques may be taken into account when determining the likelihood that the message is Spam. If the message is identified as likely to be good it is passed to the user, otherwise the message is quarantined, either by placing the message in a likely junk folder or by marking it in a manner that can be verified by the users mail client. The blacklist and the white list may contain information collected locally and information from external sources.

DENIAL OF SERVICE BLACK LIST

Denial Of Service attack typically targets a single site or a small group of sites. It is likely therefore that in most cases the source of information for the denial of service blacklist would be information collected at the site itself.

23

A Plan For No Spam

SPAM SENDER BLACKLIST

It is recommended that spam blacklists be used as one input to the content filtering decision making process than to provide an accept/deny decisions. In particular end users should not be deprived of wanted email messages because they are send from a source address that has been listed to create collateral damage as a means of establishing leverage to cause an ISP to change policy. In most cases neither the sender nor the receiver has direct influence over these policies and the annoyance caused to the use of these tactic is almost certain to be greater than that caused by the spam itself.

AUTHORIZED SENDER WHITELIST

The authorized sender white lists will typically be based on information configured locally from lists of known customers, suppliers and partners. In cases where robust authentication techniques such as digital certificates are used, it may prove adequate to list exceptions to the white lists. The use of feedback from end users is highly recommended to assist in the maintenance of the black lists and white lists and in the configuration of the content inspection processor. This feedback may employ Bayesian inference, least squares minimization etc.

24

A Plan For No Spam

CONCLUSION
There are many techniques that address a part of the spam problem. No currently known technique provides a complete solution and it is unlikely that address a part of the problem. No currently known technique provides a complete solution and it is unlikely that any technique will be found in the future that provides a complete and costless solution. Even so, there are many techniques that in combination can provide an effective strategy for addressing the spam problem. Do not send lists, protected using one-way encryption provide an objective definition of a spam message. Legislative initiatives provide a means of increasing the costs of the spam senders and reducing the profitability of their enterprise. Content inspection provides a means of identifying undesirable messages that can be deployed with immediate advantage to the user. Authentication provides a means of identifying desirable messages that allows the problem of false positives associated with content inspection to be reduced and with widespread deployment offers the possibility of a comprehensive solution. Authentication policy provides a means of knowing that a message that does not meet the Authentication policy specified by the domain name owner has been forged and should be discarded as spam. Callback loop authentication is acceptable as a last resort when a message has been identified as likely spam. it use as a first line of defense is not acceptable. Protocol improvements provide support for the content inspection and authentication mechanisms and should be pursued aggressively. Spam is a security problem. It is the lack of authentication and authorization in the email system that allows sit to be abused for any purpose spam is a community problem and it is the internet community as a whole that must find, implement and deploy solutions.

REFERENCES

25

A Plan For No Spam

Paul Graham, A Plan for Spam http://www.paulgraham.com/spam.html

Paul Graham, Better Bayesian Filtering http://www.paulgraham.com/better.html

Microsoft corp. Microsoft Junk E-Mail Filter Read me http://office.microsoft.com/Assistance/9798/newfilters.aspx

R. Fielding, J. Gettys, Mogul.H, Frystyk, L. Manister, P.Leach and T.Berners-Lee, Hypertext Transfer Protocol-HTTP 1.1.RFC 2616 http://www.ietf.org/rfc/rfc2616.txt

Kantor, B. and P.Lapsley,"Network News Transfer Protocol",RFC 977 http://www.ietf.org/rfc/rfc977.txt

Dave Winer, RSS 2.0 http://backend.userland.com/rss2 Postel. J,"Simple Mail Transfer Protocol",STD 10,RFC 821,ISI http://www.ietf.org/rfc/rfc821.txt

26

A Plan For No Spam

TABLE OF CONTENTS

INTRODUCTION IDEAL SPAM CONTROL SYSTEM PROPERTIES TOWARDS SOLUTIONS BEST PRACTICES CONTENT INSPECTION AUTHENTICATION AND AUTHORIZATION LEGISLATION AND LITIGATION DEPLOYMENT COMPREHENSIVE EMAIL MANAGEMENT CONCLUSIONS REFERENCES 21

1 2 3 4 8 11 15 18

24 25

27

A Plan For No Spam

ABSTRACT

The problem of unwanted and irrelevant mass mailings, commonly known as spam is starting to seriously degrade the usefulness of email. In this paper we provide a survey of the principal approaches currently being applied to spam control and propose a strategy by which these mechanisms may be combined to provide a comprehensive solution to the spam menace.

28

A Plan For No Spam

ACKNOWLEDGEMENT

29