Table
of Contents
I. Introduction
This document discusses anti-spam philosophies from
a variety of perspectives and provides information about available options
for dealing with spam.
II.
Who Should Be Reading this Document
This document could be useful for
- UNIX system administrators
- qmail administrators
- System Use/Abuse Policy Makers
- Casual users of e-mail concerned about SPAM
III.
General Issues
Spam
is defined here as unsolicited commercial e-mail, usually sent in bulk.
In other words, spam is simply electronic junk mail. Dealing with spam
is, at best, a very difficult task. This is mostly true because spammers
have a wide array of tools and circumstances available to them that make
it easy for them to send you mail but difficult for you to communicate
back with them or any authority over them. Spam is also difficult to deal
with because it almost always comes in under the guise of being a normal
e-mail message. No amount of technology can automatically decide what
content is undesirable to you, but there are many ways to use technology
to reduce the amount of unwanted e-mail you or your users receive.
For a more thorough explanation of
the issues here, I recommend reading the IETF
Anti Spam Recomendations. David E. Sorkin has produced an excellent
document called Technical
and Legal Approaches to Unsolicited Electronic Mail.
IV.
Specific Issues of Policy
Anyone dealing with spam prevention will have to make
definite and impacting decisions about the following issues:
- Is the prevention of spam worth the time and
resources required to reach a given level of spam reduction?
- Is the prevention of spam the responsibility
of a system administrator or the responsibility of the end user?
- Should e-mail identified as potential spam be
flatly rejected, or just tagged as spam and routed accordingly?
- Should system administrators (yours or anyone
else's) who have misconfigured their systems be held responsible for
any problems that result?
- Should you reject e-mail messages that are legitimate
in content but that do not conform to known and accepted standards?
- Should you accept for delivery mail that does
not have valid reply information (either in the envelope or From address)?
- What criteria should be met before an individual
or ISP is justifiably classified as "spam-friendly"?
V.
Basic Things You Can do to Prevent/Reduce Spam
- Don't use a dot-qmail-default file in a lazy
way
The dot-qmail-default file (e.g. '.qmail-default') makes any mailbox
name on your system valid, whether or not it actually exists. While
it can be useful as a catch-all for setups where there is a possibility
that the sender would misspell the address, it often leads to an increased
volume of spam from spammers who use made up aliases. Typical examples
of this are "sales@domain.com", "webmaster@domain.com",
and so on. Note that the dot-qmail-default file can actually be useful
in fighting spam, if you set it up to process mail sent to invalid addresses
and then reject the mail with useful error messages.
-
Report any spam that you do get.
Services like SpamCOP make this straightforward and efficient. See
the resources section below for links.
-
Educate
Educate your users, friends, and family about spam, why it is (or
is not) worth fighting, and what they can do. Spammers are successful
in getting mail through because the typical end user doesn't understand
the technical issues enough to make decisions about how to fight back.
-
Make sure your system is properly configured
All of the hosts on your system should have proper hostnames and corresponding
IP addresses. Other systems should be able to properly resolve your
hostname and IP address. Your mailer should issue standard
error and warning messages when appropriate, and you should be aware
of its configuration regarding rejection of messages. Adhering to
standards is a good thing.
VI. Commonly Held
Views About Spam
The following are some commonly held views about spam.
They are presented here to provide you with some perspective about the
issues involved in preventing spam. As you will note, the disparity
between the different views makes it even more difficult to develop
a commonly acceptable solution to preventing and reducing spam.
Commonly
Held View: "Spam Can't Be Stopped"
This view holds that, because of the seemingly impossible task of accurately
identifying e-mail messages as spam, and the difficulty in holding spammers
responsible for their actions once they have been identified, efforts
at stopping spam are a waste of time and resources, and can even result
in losing legitimate mail.
People holding this view usually think that the tolerable
ratio of lost legitimate e-mail to rejected spam is zero.
Commonly Held View: "Spam
Prevention is the Responsibility of the End-User"
This view holds that, because the identification of spam is so difficult,
the reduction of spam is the responsibility of the end user. This view
is usually held by system administrators because of the liability issues
in rejecting any e-mail on a system-wide basis. Many also think that
user-level spam prevention is more effective because it allows the user
to have more control over the kinds of messages that should be automatically
blocked or should automatically get through. People holding this view
sometimes acknowledge that, believe it or not, some people actually
like receiving commercial e-mail, and tend to believe that rejecting
any e-mail without obtaining the permission of the intended recipient
is an invasion of privacy.
People holding this view usually think that the tolerable
ratio of lost legitimate e-mail to rejected spam is a matter of personal
opinion to be determined by the user affected.
Commonly Held View: "Spam
Prevention is the Responsibility of the System Administrator"
This view holds that, because there are some tools available for rejecting
spam at the system level, and because the prevention of spam is a worthwhile
effort in the name of conserving system resources, spam prevention is
the responsibility of the person or entity maintaining the mail server.
This view is often held by administrative bodies in the context of reducing
the amount of time their staff (both sysadmins and end-users) spend
dealing with unwanted spam. It is also often held by system administrators
who have a particular dislike for spammers who are, as they perceive
it, misusing or abusing their systems for unwanted commercial e-mail.
This view also affects system administrators who maintain
systems that are configured in such a manner that spammers can more
easily use that system for spamming. The most common example of this
is when a system is classified as an "open relay", allowing
mail to pass through it that is neither to or from a user on the system.
Systems like this are sometimes seen as facilitating spam in a significant
manner, and that it is the responsibility of the system administrators
to alter that configuration in the name of fighting spam.
People holding this view usually think that the tolerable
ratio of lost legitimate e-mail to rejected spam is acceptable at low
levels. They are usually willing to accept that some legitimate e-mails
will be rejected because of a misconfiguration or error on the part
of another system administrator.
Variations
Within each of the above views, there are many variations. Often the
variations involve decisions about the issues stated above in "Specific
Issues". Some common variations:
-
Some people believe that any messages from senders who have been
listed in one of the various "black hole lists" (see resources
section for links) should be rejected without exception. Others sometime
believe that these black hole lists are not always fair or just in
their criteria for inclusion, and that depending on these lists would
result in too many legitimate e-mails being rejected. Even others
disagree with the use of the black hole lists because of grievances
with the methodology the owners use to develop and maintain the lists.
- Some people believe that messages not conforming to known standards
for mail delivery should be rejected or identified as potential spam.
The most common example of this involves "From" headers or
"envelope" addresses. These addresses are necessary to handle
the "bouncing" of messages and for any sort of reply. For
a variety of reasons, many spam messages do not have valid From or envelope
headers. So, while some believe that any such message should be rejected,
others hold that there are too many exceptions where these headers might
be invalid but the content or intent of the message is legitimate.
VII. Options for Individual
Users
This
section discusses spam prevention options for individual users. It assumes
that you have the access necessary to modify your personal mail setup
and that you have basic knowledge of software configuration procedures.
If either of the above is not true, you should ask your system administrator
for assistance.
The spam-prevention options for individual users at first seem limited,
given that if an e-mail has made its way to you, it has been successfully
accepted for delivery by your system, and is now your responsibility
for filtering. However, filtering mail (also sometimes called
"rule-based delivery") can be a powerful tool (in spam-prevention
and other arenas), and is probably your best option.
The most commonly used program for filtering mail under UN*X is procmail,
a recipe-based set of scripts that will route, reject, forward, and
modify your mail based on criteria you specify. Tools for filtering
mail under other platforms (Windows, Mac, etc) are numerous, and are
usually tied in to whatever mail client you use. Modern versions of
Eudora, Outlook, Netscape Mail, etc. all have filtering capabilities.
Using Realtime Third-Party Black-hole
Lists
With the basic procmail filter below (to be used in your .procmailrc
file) and two simple programs rblcheck
and origip.pl, you can block/route mail from
senders who have been listed in various realtime black-hole lists.
MAILDIR=$HOME/mail
SPAMFOLDER=$MAILDIR/junkmail
LOGFILE=$MAILDIR/log
:0h
TCPREMOTEIP=| origip.pl
LOG="Remote IP: \"$TCPREMOTEIP\""
:0
* !^From.*myfriend@domain.com
* ! ? if [ -n "$TCPREMOTEIP" ]; \
then /usr/local/bin/rblcheck -q "$TCPREMOTEIP"; fi
{
LOGABSTRACT=all
LOG="Filter: RBL-filtered address: \"$TCPREMOTEIP\""
:0:
$SPAMFOLDER
}
This recipe will filter any mail with IP addresses on the ORBS (Open
Relay Blocking System), RBL (Realtime Blackhole list), OR DUL (Dial-up
User List). (See Other Resources below for
links).
Does anyone know of RBL-checking tools for Windows, Mac, etc that
one could use in conjunction with client software filters?
Using Your Own Black-hole Lists
There are several different ways to set up procmail to use a locally
maintained list of potential spammers to automatically filter spam,
and a list of known good senders to automatically safeguard legitimate
mail.
Probably the best place to start here is to use SpamBouncer,
an extensive set of recipes for procmail designed for the novice procmail
user. To use SpamBouncer, just follow the instructions provided with
the software. After SpamBouncer is installed, you can modify the lists
of good and bad senders to meet your needs.
Using Tagged Messages and Special/Dedicated
E-mail Addresses.
Tagged messages can mean several different things.
- Requiring people sending you e-mail to verify the address they send
from before the mail will go through to you. While it usually requires
they do this only once in a lifetime of their e-mail address, it can
be a hassle for "new contacts."
- Allowing persons sending mail from known spam domains to bypass
spam filters by including a secret passphrase.
- Using e-mail addresses that "expire" or are dated when
you send mail, so that responses or new mail to that address (especially
those posted on newsgroups, websites, etc) are protected from spam.
Here are some tagged message resources:
- SpamBouncer has a mechanism
for tagged messenging
- Thomas Erskine wrote tms
(Tagged Message Sender), which also does this.
- Jason R. Mastaler has taken tms and extended the project into Tagged
Message Delivery Agent.
- Brett Randall has developed an avoidance
technique especially for users participating in usenet and mailing
list discussions.
VIII.
Options for qmail Administrators
This section discusses options for system administrators
who want to implement anti-spam mechanisms at the system-wide level. Please
note that you should resolve the specific issues
listed above before implementing any of these solutions, and that you
should always notify your users of any changes to the system that affect
the mail they do or don't receive.
Rejecting SMTP connections at
the network level from hosts with bad DNS
It is becoming common for the default installation of many Unix operating
systems like FreeBSD and Linux to include a mechanism to block network
traffic based on certain criteria, commonly referred to "host-based
access control" and commonly implemented using the tcp_wrappers
package. In some of these installations, network traffic from hostnames
that do not map to valid IP addresses is blocked. While not an e-mail
specific measure, this is one way to cut down on e-mail from hosts that
have misconfigured their DNS, and therefore are thought by some to be
more likely to be spam-friendly.
If you're using inetd,
an example line in a FreeBSD /etc/hosts.allow is here:
ALL : PARANOID : RFC931 20 : deny
One can also achieve this using the ucspi-tcp
package's tcpserver
(now the recommended alternative to inetd),
by enabling the "-p"
option, for paranoid, e.g.
tcpserver -p -v
-x/etc/tcp.smtp.cdb -u1007 -g1007 0 25 \
qmail-smtpd 2>&1
Using your SMTP daemon to reject
"known" spammers
In the ucspi-tcp package there is the rblsmtpd
package, an alternative to the usual qmail-smtpd, and works with any SMTP
server that runs under tcpserver.
So, if you follow the Life
With qmail Installation guide, and then update your supervise scripts
accordingly, your /var/qmail/supervise/qmail-smtpd/run
script looks like this:
#!/bin/sh
QMAILDUID=`id -u qmaild`
NOFILESGID=`id -g qmaild`
MAXSMTPD=`cat /var/qmail/control/concurrencyincoming`
exec /usr/local/bin/softlimit
-m 2000000 \
/usr/local/bin/tcpserver -v -p -x /etc/tcp.smtp.cdb -c "$MAXSMTPD"
\
-u "$QMAILDUID" -g "$NOFILESGID" 0 smtp \
/usr/local/bin/rblsmtpd /var/qmail/bin/qmail-smtpd 2>&1
Note that the above is an updated version of the call to rblsmtpd;
the previous version was only correct for older versions of daemontools,
and is now deprecated. The upgrade is worth it.
If you're still using inetd,
you can patch
qmail to do about the same thing, or you can patch tcp_wrappers to use
the RBL with software from
SmallWorks.
If you want to use the ORBS database (or any other database
with DNS access) in addition to the RBL, you can modify
your rblsmtpd configuration with the "-r"
option to do so.
rblsmtpd -rrelays.orbs.org
-rrbl.maps.vix.com
Mike Silbersack says "If you wish to use the *.mail-abuse.org
black-hole lists, you'll have to apply the patch to make rblsmptd work
with A records"
Using qmail-smtpd to reject
mail with invalid envelope or From headers
This area is a little blurry right now; it is the author's hope that
readers will contribute their experiences here to improve the recommended
options.
There are several patches out there that claim to make qmail
reject messages with bad envelopes or From headers (i.e. if the envelope
is blank or if the hostname in it doesn't have a valid DNS entry) or otherwise
deal with suspected SPAM mail. Note that most of these patches are not
featured on the qmail site and are therefore assumed to be "nonstandard".
- Nagy Balazs wrote a patch
to ensure that the domain name on the envelope sender is a valid DNS
name. This ensures that you do not receive email which you cannot bounce,
should that prove necessary.
- Erwin Hoffman has written a patch for qmail-smtpd called SPAMCONTROL
which improves qmail's filtering abilities and makes it RFC 2505 compliant.
He and Noel Mistula also produced some scripts
for filtering attachments and subject lines (something you can also
do with
procmail).
- The folks at flame.org wrote another
patch that performs various header checks and bounce/flagging functionality
- Will Harris wrote a patch
that allows you to use a new control file to specify Perl regular expressions
to be used when checking the validity of the envelope sender.
- Mark Delany wrote a patch
for qmail 1.01 which lets you match the envelope sender against
a regex and accept or reject the mail accordingly.
- Jonathan McDowell has written an X-Spam-Warning
header patch that adds warning headers for messages from senders
in ORBS, RSS, RBL and DUL without the use of any external programs.
This is useful if you want to allow your users to decide how to handle
SPAM while tagging it as such at the system level.
- Jay Soffian has a script that
also adds warning headers, but uses the existing QMAILQUEUE
patch instead of patching qmail source itself.
- Chris Johnson wrote a patch to log
attempted relay attempts
It should also be noted here that messages with recipient addresses in
the form "user%hosta@hostb"
are not going to be relayed through your system unless you have misconfigured
something. See the qmail.faqts
page on this issue for futher details.
Make it hard to spam from your system to
the outside world
There are a variety of ways to make it difficult for your users to create
spam. This is an important effort; while most of this document focuses
on avoiding incoming spam, don't forget that a lot of incoming spam is
generated because of overly lax mail sending policies.
-
Chris Johnson has written a patch for qmail called tarpit.
Tarpitting is "the practice of inserting a small sleep in an
SMTP session for each RCPT TO after some set number of RCPT TOs."
This discourages a user from using a given system as a relay.
IX. Other Resources
Real-time Third-Party Blocking Solutions
Third Party Spam Reporting Services
General Related Mail and System Tools
Writings on Spam
- Sorkin, David E. Technical
and Legal Approaches to Unsolicited Electronic Mail (April 2001)
PDF
- Kinnard, Shannon, Marketing
With Email : A Spam-Free Guide (December 1999)
- Loshin, Pete et al., Essential
E-Mail Standards : Rfcs and Protocols Made Practical (November,
1999)
- Mulligan, Geoff, Removing
the Spam : Email Processing and Filtering (April 1999)
- Schwartz, Alan et al., Stopping
Spam (October 1998)
- Wyman, Carolyn, Spam:
A Biography (July 1999) :)
Anti-Spam Manifestos and Organizations
X.
History
- April 28, 2001 - v0.31, added some new misc links
- April 21, 2001 - version 0.30; added a new section of resources
for administrators, updated some links, updated daemontools syntax
to be current, added misc links
- January 31, 2001 - minor update regarding -r syntax for rblsmtpd
- December 16, 2000 - added section numbering; removed links to contributor's
e-mail addresses (too much potential irony); added some new patches
and software; expanded user section to include non-Unix-centric perspectives;
other minor fixes and updates
- October 7, 2000 - added IETF recommendations, submitted by Robert
Dalton
- October 5, 2000 - updated some links, reviewed content
- July 14, 2000 - added rblsmtpd syntax that includes logging; added
more spam-related resources.
- July 3, 2000 - freshened up some links
- May 26, 2000 - updated rblsmtpd ORBS syntax with corrections provided
by Konstantin Riabitsev
- April 30, 2000 - added new links
- April 4, 2000 - version 0.2, implemented some suggested changes
from Dave Sill
- April 3, 2000 - Implemented suggested changes from Mikko Hanninen,
Peter van Dijk
- April 3, 2000 - version 0.1 created by Chris Hardie
XI. Comments/Feedback
If you have specific suggestions for changes, corrections, and additions
to this HOWTO, please send them to Chris
Hardie and they will be integrated into the document as appropriate.
If you have general comments, reactions, or points of discussion, please
use the comment form below. Please do NOT use this section to ask technical
questions about your specific system setup; use the qmail users list for
this.
7 most recent
User Comments on this Page [view all comments] [all comments on this site] [posting policy]
(powered by SSIComment)
blocking from and tos posted by Stefan Sels
on 07/11/2001 at 8:56:21 AMthere are some patches to use:
badrcpto solution :
with this patch, you can block sender and domains. this is done after the RCPT TO: handshake in SMTP. This is much better than bouncing. Spammer won´t use valid adresses.
you can get the patch from my hompage. this is a mirror, I am not the author. badrcptto patch
There are serveral solutions to stop stupid spammers who use always the same sender (badmailfrom) one is the patch (linked on qmail.org) called wildmat (= wild match).
it allows to mach sender against regular expressions.
you can get it
here
Another orbs replacement posted by Andres Salomon
on 07/10/2001 at 10:27:04 AMhttp://www.orbz.org/. This one is doing active scanning of relays, using lists imported from various other relay databases. Net-wide blocks aren't done, no politics involved; if an email finds it's way back to the scanner, it's blackholed. Plain and simple.
RBL style lists... an update posted by Larry M. Smith
on 07/03/2001 at 7:37:48 AMORBS has died last month (JUN 01)
However there are several replacements popping up;
http://www.orbl.org
http://www.orbs.org
http://www.orbd.org
http://www.dorkslayers.org
They may be others...
Also another site has come up in Feb 01 to deal with sites that do not want to follow the RFCs. However, as yet (Jul 01) no one has patched qmail to use this list.
http://www.rfc-ignorant.org
rblsmtpd patch posted by Ian Scott
on 07/02/2001 at 11:33:41 AMThe patch to rblsmtpd to allow it to use A records instead of TXT records is available here:
http://www.qmail.org/ucspi-rss.diff
rblsmtpd posted by Stefan Sels
on 06/09/2001 at 4:48:52 AMAs far as I know, rblsmtpd does not support MAPS because they changed their zone files. as you quote (Mike Silbersack says "If you wish to use the *.mail-abuse.org black-hole lists, you'll have to apply the patch to make rblsmptd work with A records")
What is "the patch" ? does ist exist ? have I missed a link ?
there seems to be everyone knows the problem but nobody wrote a patch/solution ...?
Any comment (preferable to my emailaccount) will be highly apreceated.
add this qmail anti-spam links posted by Noel
on 05/14/2001 at 8:30:53 AMhttp://www.fehcom.de/qmail/filter.html
http://www.ornl.gov/its/archives/mailing-lists/qmail/2001/03/msg01041.html
for filtering attachments, etc
cheers
Noel
rblsmtpd posted by John Nastasi
on 04/22/2001 at 11:46:28 AMFYI--
Rblsmtpd does in fact work with multiple r's. One thing I ran into, however, was the fact that RBL and DUL work with the following syntax:
-rblackholes.mail-abuse.org
-rdialups.mail-abuse.org
...but RSS "only" works with the following syntax:
-r 'relays.mail-abuse.org:Open Relay - see '
...that is to say, it will fail if you use:
-rrelays.mail-abuse.org
...haven't tried ORBS yet, but now have RBL, RSS, and DUL (in that order) working on the same rblsmtpd instance.
Thanks for the great resource.
Post your comment (all fields but email are required):
|