-<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
+<!-- -*-html-*- -->
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
- <title>HOWTO - Using exim and Mailman together</title>
+ <title>HOWTO - Using Exim 4 and Mailman together</title>
+ <style type="text/css">
+ body { color: #00005a; background: white; }
+ a:link { color: #ff6600; }
+ a:visited { color: #990000; }
+ a:hover { color: #ff9933; }
+ </style>
</head>
-
- <body bgcolor="#FFFFFF" text="#00005A" link="#FF6600" alink="#FF9933" vlink="#990000">
- <h1>HOWTO - Using exim and Mailman 2.1 together</h1>
+ <body>
+ <h1>HOWTO - Using Exim 4 and Mailman 2.1 together</h1>
<p>Mailman is a list manager with web front end and built in
- archiving functions. Details can be found at <a href="http://www.list.org/">
- http://www.list.org/</a>. This documentation describes the
- configuration of Exim (version 3 or 4) to work with Mailman
- version 2.1</p>
+ archiving functions. Details can be found at <a
+ href="http://www.list.org/"> http://www.list.org/</a>. This
+ documentation describes the configuration of Exim (version 4) to
+ work with Mailman version 2.1</p>
+ <h2><a name="index">Index</a></h2>
+ <ul>
+ <li><a href="#scope">Scope of this document</a></li>
+ <li><a href="#basic">Basic Configuration</a></li>
+ <ul>
+ <li><a href="#mmconf">Mailman configuration</a></li>
+ <li><a href="#exconf">Exim configuration</a></li>
+ <li><a href="#maconf">Main configuration settings</a></li>
+ <li><a href="#roconf">Exim Router</a></li>
+ <li><a href="#taconf">Exim Transport</a></li>
+ </ul>
+ <li><a href="#batune">Basic mailing list MTA tuning</a></li>
+ <ul>
+ <li><a href="#retune">Receiver verification</a></li>
+ <li><a href="#rctune">Tuning of numbers of recipients</a></li>
+ <li><a href="#smtune">SMTP callback</a></li>
+ </ul>
+ <li><a href="#verpin">Doing VERP and personalisation with exim
+ and Mailman</a></li>
+ <ul>
+ <li><a href="#verpmm">VERP within Mailman</a></li>
+ <li><a href="#persmm">Mailing list personalisation by Mailman</a></li>
+ <li><a href="#verpex">VERP expansion by Exim rather than
+ Mailman</a></li>
+ </ul>
+ <li><a href="#virdom">Virtual domains</a></li>
+ <li><a href="#lispol">List policy management</a></li>
+ <ul>
+ <li><a href="#conpol">Content scanning</a></li>
+ <li><a href="#incpol">Incoming message checks</a></li>
+ <li><a href="#mmapol">Mailman specific ACL checks</a></li>
+ </ul>
+ <li><a href="#lisver">List verification</a></li>
+ <li><a href="#problem">Possible Problems</a></li>
+ <li><a href="#dochis">Document History</a></li>
+ <li><a href="#doccha">Document Changes</a></li>
+ <li><a href="#docfin">Final Word</a></li>
+ </ul>
- <h2>Mailman configuration</h2>
+ <h2><a href="#index"><a name="scope">Scope of this document</a></a></h2>
- <p>There is no Mailman configuration needed other than the
- standard options detailed in the Mailman install documentation.
- The Exim configuration is transparent to Mailman. The user/group
- settings for Mailman must match those in the config fragments
- given below.</p>
+ <p>This document describes how to set up a basic working
+ configuration using Exim 4 as an MTA for the Mailman MLM. The
+ assumption is made that the receiving MTA, Mailman and the list
+ distribution MTA are all on the same machine, and that Mailman
+ talks to Exim using SMTP to address <tt>127.0.0.1</tt></p>
+ <p>It also describes ways of using VERP delivery, both
+ conventionally (doing VERP from Mailman), and an alternative more
+ efficient technique where VERP expansion is done within exim.</p>
- <h2>Exim configuration</h2>
+ <p>Tuning and setting appropriate mailing list protection policies
+ is also covered in passing.</p>
- <p>The Exim configuration is built so that a list created within
- Mailman automatically appears to Exim without the need for
- defining any additional aliases.</p>
+ <p>General installation, use, running and administration of either
+ Mailman or exim is not covered here - the documentation for the
+ programs concerned should be read for this information.</p>
- <p>The drawback of this configuration is that it will work poorly
- on systems supporting lists in several different mail domains.
- While Mailman handles virtual domains, it does not yet support
- having two distinct lists with the same name in different virtual
- domains, using the same Mailman installation. This will
- eventually change. (But see below for a variation on this scheme
- that should accomodate virtual domains better.)</p>
+ <h2><a href="#index"><a name="basic">Basic Configuration</a></a></h2>
- <p>The configuration file excerpts below are for use in an already
- functional Exim configuration, which accepts mail for the domain
- in which the list resides. If this domain is separate from the
- others handled by your Exim configuration, then you'll need to:</p>
+ <h3><a href="#index"><a name="mmconf">Mailman configuration</a></a></h3>
- <ul>
+ <p>For basic operation there is no Mailman configuration needed
+ other than the standard options detailed in the Mailman install
+ documentation. The user/group settings for Mailman must match
+ those in the config fragments given below, and you need to have at
+ least configured <tt>DEFAULT_URL_HOST</tt> and
+ <tt>DEFAULT_EMAIL_HOST</tt> within Mailman, for example by editing
+ <tt>~mailman/Mailman/mm_cfg.py</tt> and setting the following
+ (substituting in your own domains):-</p>
- <li>add the list domain, <tt>"my.list.domain"</tt> to local_domains</li>
+<pre>
+ # The host part of the URL used for your mailman install
+ DEFAULT_URL_HOST = 'www.example.com'
+ #
+ # The email domain of your lists
+ DEFAULT_EMAIL_HOST = 'list.example.com'
+ #
+ # Let Mailman know that the MTA needs no aliases setting
+ MTA = None
+</pre>
- <li>add a <tt>"domains=my.list.domain"</tt> option to the
- director (router) for the list</li>
+ <p>The final setting above informs Mailman that it does not need
+ to prompt you to add aliases when creating a list (like Sendmail),
+ or modify other settings (like Postfix). Not setting this will
+ mean that Mailman nags you to do things that aren't necessary in
+ an Exim configuration.</p>
- <li>(optional) exclude that domain from the other directors
- (routers)</li>
- </ul>
+ <p>After making a change to the Mailman configuration file you
+ need to restart the Mailman queue runners.</p>
+
+<pre>
+ ~mailman/bin/mailmanctl restart
+</pre>
+
+ <p>Mailman should also be set to deliver to the MTA using SMTP -
+ this is done by having <tt>DELIVERY_MODULE = 'SMTPDirect'</tt> in
+ the config file (which is the default mode of operation)</p>
- <p><i>[Note: the instructions in this document should work with
- either Exim 3 or Exim 4. In Exim 3, you must have a
- <tt>'local_domains'</tt> configuration setting; in Exim 4, you
- most likely have a <tt>'local_domains'</tt> domainlist. If you
- don't, you probably know what you're doing and can adjust
- accordingly. Similarly, in Exim 4 the concept of "directors" has
- disappeared -- there are only routers now. So if you're using
- Exim 4, whenever this document says "director", read
- "router".]</i></p>
- <p>Whether you are using Exim 3 or Exim 4, you will need to add
- some macros to the main section of your Exim config file. You
- will also need to define one new transport. With Exim 3, you'll
- need to add a new director; with Exim 4, a new router plays
- the same role.</p>
+ <h3><a href="#index"><a name="exconf">Exim configuration</a></a></h3>
- <p>Finally, the configuration supplied here should allow
- co-habiting Mailman 2.0 and 2.1 installations, with the proviso
- that you'll probably want to use <tt>"mm21"</tt> in place of
- <tt>"mailman"</tt> -- e.g., <tt>MM21_HOME</tt>,
- <tt>mm21_transport</tt>, etc.</p>
+ <p>The Exim configuration is built so that a list created within
+ Mailman automatically appears to Exim without the need for
+ defining any additional aliases (however Mailman may helpfully
+ show or email you a list of required aliases when you create a
+ list - you can just ignore those - if you have set the
+ <tt>MTA</tt> parameter it will stop doing this).</p>
+ <p>The drawback of this configuration is that it will work poorly
+ on systems supporting lists in several different mail domains.
+ While Mailman handles virtual domains, it does not yet support
+ having two distinct lists with the same name in different virtual
+ domains, using the same Mailman installation. This will
+ eventually change. (But see below for a variation on this scheme
+ that should accommodate virtual domains better.)</p>
- <h2>Main configuration settings</h2>
+ <p>The configuration file excerpts below are for use in an already
+ functional Exim configuration. You also need to have an alias for
+ <tt>mailman</tt> within the <tt>mm_domains</tt>, this picks up
+ mail sent to the site list (or basically just sent in error), and
+ should forward to the Mailman Administrator. It appears that a
+ couple of Mailman messages mention the <tt>mailman-admin</tt>
+ address (this appears to be an error in Mailman or maybe a
+ packaging error), so I would suggest that <tt>mailman-admin</tt>
+ is aliased also to the Mailman Administrator.</p>
+
+ <p><i>[Note: the instructions in this document will work only with
+ Exim 4. It may be possible to adapt them for Exim 3, but frankly
+ it is not worth the trouble]</i></p>
+
+ <p>You will need to add some macros to the main section of your
+ Exim config file. You will also need to define one new transport
+ and add new routers. Additional ACLs may be used to handle policy
+ enforcement. Remember that the exim daemon needs restarting
+ before it sees configuration changes.</p>
+
+ <h3><a href="#index"><a name="maconf">Main configuration settings</a></a></h3>
<p>First, you need to add some macros to the top of your Exim
- config file. These just make the directors (routers) and
- transport below a bit cleaner. Obviously, you'll need to edit
- these based on how you configured and installed Mailman.</p>
+ config file. These just make the routers and transport below a
+ bit cleaner. Obviously, you'll need to edit these based on how
+ you configured and installed Mailman.</p>
<pre>
# Home dir for your Mailman installation -- aka Mailman's prefix
# directory.
- MAILMAN_HOME=/usr/local/mailman
- MAILMAN_WRAP=MAILMAN_HOME/mail/mailman
-
+ # By default this is set to "/usr/local/mailman"
+ # On a Red Hat/Fedora system using the RPM use "/var/mailman"
+ # On Debian using the deb package use "/var/lib/mailman"
+ # This is normally the same as ~mailman
+ MM_HOME=/var/mailman
+ #
# User and group for Mailman, should match your --with-mail-gid
# switch to Mailman's configure script.
- MAILMAN_UID=mailman
- MAILMAN_GID=mailman
-</pre>
-
- <h2>Transport for Exim 3</h2>
-
- <p>Add this to the transports section of your Exim config file,
- i.e. somewhere between the first and second "end" line:</p>
-
-<pre>
- mailman_transport:
- driver = pipe
- command = MAILMAN_WRAP \
- '${if def:local_part_suffix \
- {${sg{$local_part_suffix}{-(\\w+)(\\+.*)?}{\$1}}} \
- {post}}' \
- $local_part
- current_directory = MAILMAN_HOME
- home_directory = MAILMAN_HOME
- user = MAILMAN_UID
- group = MAILMAN_GID
+ # Value is normally "mailman"
+ MM_UID=mailman
+ MM_GID=mailman
+ #
+ # Domains that your lists are in - colon separated list
+ # you may wish to add these into local_domains as well
+ domainlist mm_domains=list.example.com
+ #
+ # -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
+ #
+ # These values are derived from the ones above and should not need
+ # editing unless you have munged your mailman installation
+ #
+ # The path of the Mailman mail wrapper script
+ MM_WRAP=MM_HOME/mail/mailman
+ #
+ # The path of the list config file (used as a required file when
+ # verifying list addresses)
+ MM_LISTCHK=MM_HOME/lists/${lc::$local_part}/config.pck
</pre>
- <p><i>(XXX this is untested by me under Exim 3! Can someone using
- Exim 3 please let me know if it works?)</i></p>
-
-
- <h2>Director for Exim 3</h2>
-
-
- <p>If you're using Exim 3, you'll need to add the following
- director to your config file (directors go between the second and
- third "end" lines). Also, don't forget that order matters --
- eg. you can make Mailman lists take precedence over system aliases
- by putting this directors in front of your aliasfile director, or
- vice-versa.</p>
-
-<pre>
- # Handle all addresses related to a list 'foo': the posting address.
- # Automatically detects list existence by looking
- # for lists/$local_part/config.pck under MAILMAN_HOME.
- mailman_director:
- driver = smartuser
- require_files = MAILMAN_HOME/lists/$local_part/config.pck
- suffix_optional
- suffix = -bounces : -bounces+* : \
- -confirm+* : -join : -leave : \
- -owner : -request : -admin
- transport = mailman_transport
-</pre>
+ <h3><a href="#index"><a name="roconf">Exim Router</a></a></h3>
- <h2>Router for Exim 4</h2>
+ <p>This router picks up all the addresses going to the Mailman
+ lists. Initially it selects only the domains that have may have
+ lists in, then selects where <tt>local_part</tt> matches a list
+ name (ie you can see a list config file). The suffixes pick up
+ all the Mailman admin addresses<p>
- <p>In Exim 4, there are no such things as directors -- you instead
- need to add a router. Also, the canonical order of the
- configuration file was changed so routers come before transports,
- so the routers for Exim 4 come first here. Put these two routers
- somewhere after the <tt>"begin routers"</tt> line of your config
- file, and remember that order matters.</p>
+ <p>The router should be placed in the router section (ie somewhere
+ after the <tt>"begin routers"</tt> line of your config file).
+ Normally you would place it just after the aliases router (since
+ that will pick up the <tt>mailman</tt> master contact
+ address).</p>
<pre>
mailman_router:
driver = accept
- require_files = MAILMAN_HOME/lists/$local_part/config.pck
+ domains = +mm_domains
+ require_files = MM_LISTCHK
local_part_suffix_optional
- local_part_suffix = -bounces : -bounces+* : \
- -confirm+* : -join : -leave : \
- -owner : -request : -admin
+ local_part_suffix = -admin : \
+ -bounces : -bounces+* : \
+ -confirm : -confirm+* : \
+ -join : -leave : \
+ -owner : -request : \
+ -subscribe : -unsubscribe
transport = mailman_transport
</pre>
- <h2>Transports for Exim 4</h2>
-
- <p>The transport for Exim 4 is the same as for Exim 3; just copy
- the transport given above to somewhere under the "begin
- transports" line of your Exim config file.</p>
-
-
- <h2>Notes</h2>
+ <h3><a href="#index"><a name="taconf">Exim Transport</a></a></h3>
- <p>Exim should be configured to allow reasonable volume --
- e.g. don't set max_recipients down to a silly value -- and with
- normal degrees of security -- specifically, be sure to allow
- relaying from 127.0.0.1, but pretty much nothing else. Parallel
- deliveries and other tweaks can also be used if you like;
- experiment with your setup to see what works. Delay warning
- messages should be switched off or configured to only happen for
- non-list mail, unless you like receiving tons of mail when some
- random host is down.</p>
+ <p>The transport for Exim 4 can be placed anywhere where under the
+ <tt>begin transports</tt> line of your Exim config file.</p>
+ <p>The <tt>if def:local_part_suffix</tt> section selects whether
+ the suffix is used as the mailman command, or whether there is no
+ suffix and so <tt>post</tt> is passed as a command.</p>
- <h2>Problems</h2>
-
- <ul>
-
- <li> Mailman will send as many <tt>MAIL FROM/RCPT TO</tt> as it
- needs. It may result in more than 10 or 100 messages sent in one
- connection, which will exceed the default value of Exim's
- smtp_accept_queue_per_connection This is bad because it will
- cause Exim to switch into queue mode and severely delay delivery
- of your list messages. The way to fix this is to set mailman's
- <tt>SMTP_MAX_SESSIONS_PER_CONNECTION</tt> (in
- <tt>~mailman/Mailman/mm_cfg.py</tt>) to a smaller value than
- Exim's <tt>smtp_accept_queue_per_connection</tt></li>
+ <p>The <tt>sg</tt> phrase strips the VERP information (if any)
+ from the suffix,</p>
- <li>Mailman should ignore Exim delay warning messages, even
- though Exim should never send this to list messages. Mailman
- 2.1's general bounce detection and VERP support should greatly
- improve the bounce detector's hit rates.</li>
-
- <li>List existence is determined by the existence of a
- config.pck file for a list. If you delete lists by foul means,
- be aware of this.</li>
+<pre>
+ mailman_transport:
+ driver = pipe
+ command = MM_WRAP \
+ '${if def:local_part_suffix \
+ {${sg{$local_part_suffix}{-(\\w+)(\\+.*)?}{\$1}}} \
+ {post}}' \
+ $local_part
+ current_directory = MM_HOME
+ home_directory = MM_HOME
+ user = MM_UID
+ group = MM_GID
+</pre>
- <li>If you are getting Exim or Mailman complaining about user
- ids when you send mail to a list, check that the
- <tt>MAILMAN_UID</tt> and <tt>MAILMAN_GID</tt> match those of
- Mailman itself (i.e. what were used in the configure script).
- Also make sure you do not have aliases in the main alias file
- for the list.</li>
- </ul>
+ <h2><a href="#index"><a name="batune">Basic mailing list MTA tuning</a></a></h2>
+ <p>Exim has a lot configurability, especially where the ACL
+ (Access Control Lists) used during SMTP reception are concerned.
+ MTA policy needs to be tuned so that list traffic is not affected
+ by ACLs intended for qualifying traffic coming in from outside.
+ Later in this document some suggestions are made regarding
+ filtering traffic that is going into the mailing list, however
+ </p>
- <h2>Receiver Verification</h2>
+ <h3><a href="#index"><a name="retune">Receiver verification</a></a></h3>
- <P>Exim's receiver verification feature is very useful -- it lets
+ <p>Exim's receiver verification feature is very useful -- it lets
Exim reject unrouteable addresses at SMTP time. However, this is
most useful for externally-originating mail that is addresses to
mail in one of your local domains. For Mailman list traffic, mail
for trouble. In particular, Exim will attempt to route every
recipient addresses in outgoing Mailman list posts. Even though
this requires nothing more than a few DNS lookups for each
- address, it can still introduce significant delays. Therefore,
- you should disable recipient verification for Mailman traffic.</p>
-
- <p>Under Exim 3, put this in your main configuration section:</p>
-<pre>
- receiver_verify_hosts = !127.0.0.1
-</pre>
+ address, it can still introduce significant delays (because these
+ verifications have to be done serially as you attempt handoff to
+ exim). Therefore, you should disable recipient verification for
+ Mailman traffic.</p>
<p>Under Exim 4, this is probably already taken care of for you by
the default recipient verification ACL statement (in the "RCPT TO"
<p>which only does recipient verification on addresses in your
domain. (That's not exactly the same as doing recipient
verification only on messages coming from non-127.0.0.1 hosts, but
- it should do the trick for Mailman.)</p>
+ it should do the trick for Mailman). Obviously if the next ACL
+ does verification on non-local addresses you will need to deal
+ with that.</p>
-
- <h2>SMTP Callback</h2>
-
- <p>Exim's SMTP callback feature is an even more powerful way to
- detect bogus sender addresses than normal sender verification.
- Unfortunately, lots of servers send bounce messages with a bogus
- address in the header, and there are plenty that sound bounces
- with bogus envelope senders (even though they're supposed to just
- use an epmty envelope sender for bounces).</p>
-
- <p>In order to ensure that Mailman can disable/remove bouncing
- addresses, you generally want to receive bounces for Mailman
- lists, even if those bounces are themselves not bouncable. Thus,
- you might want to disable SMTP callback on bounce messages.</p>
-
- <p>If you do header and envelope callbacks, you can disable them
- for bounces to mailman lists (it is quite common for internal
- hosts to bounce with a non reachable internal address). The idea
- is that you typically don't want non bounceable Email, but you'd
- better accept bounces to mailman lists so that you can unsubscribe
- the people who are bouncing.</p>
-
- <p>With Exim 4, you can accomplish this using something like the
- following in your <tt>"RCPT TO"</tt> ACL:</p>
+ <p>Alternatively you can add an early get-out in the "RCPT TO"
+ ACL), which <i>trusts</i> all traffic coming from the loopback IP
+ address:</p>
<pre>
- # Accept bounces to lists even if callbacks or other checks would fail
- warn message = X-WhitelistedRCPT-nohdrfromcallback: Yes
- condition = \
- ${if and {{match{$local_part}{(.*)-bounces\+.*}}
- {exists {MAILMAN_HOME/lists/$1/config.pck}}} \
- {yes}{no}}
- {yes}{no}}
-
- accept condition = \
- ${if and {{match{$local_part}{(.*)-bounces\+.*}}
- {exists {MAILMAN_HOME/lists/$1/config.pck}}} \
- {yes}{no}}
- {yes}{no}}
-
- # Now, check sender address with SMTP callback.
- deny !verify = sender/callout=90s/check_postmaster
+ accept hosts = 127.0.0.1
</pre>
- <p>If you also do SMTP callbacks on header addresses, you'll want
- something like this in your "DATA" ACL:</p>
+ <h3><a href="#index"><a name="rctune">Tuning of numbers of recipients</a></a></h3>
+
+ <p>By default Mailman will send up to 500 recipients on each
+ message it injects into exim. However this is not necessarily a
+ good configuration for exim since it will route all those
+ recipients before starting deliveries to any of them.
+ Additionally some ACL configurations have tests on the maximum
+ number of recipients (which is a good reason for having a get out
+ ACL for list traffic as described above)</p>
+
+ <p>I would suggest setting Mailman to send a maximum of 5 to 50
+ recipients on a single mail (setting it lower decreases list
+ latency, but increases the work that Mailman and exim have to do),
+ and change it to send a maximum of 30 messages per SMTP
+ connection. To reflect this you should also change the exim
+ parameter <tt>smtp_accept_queue_per_connection</tt> to be 30 as
+ well.</p>
+
+ <p>For example, add the following lines to
+ <tt>~mailman/Mailman/mm_cfg.py</tt>:</p>
<pre>
- deny !condition = $header_X-WhitelistedRCPT-nohdrfromcallback:
- !verify = header_sender/callout=90s/check_postmaster
+ # Max recipients for each message
+ SMTP_MAX_RCPTS = 15
+ # Max messages sent in each SMTP connection
+ SMTP_MAX_SESSIONS_PER_CONNECTION = 30
</pre>
- <p><i>[XXX all this stuff is completely untested by me!
- -Greg]</i></p>
+ <p>Tuning a mailing list system is very much a black art, and
+ depends on the type of lists you host, their throughput, size and
+ the bandwidth available. In general, tuning is only a significant
+ issue if you are pushing your system near its operational
+ limits.</p>
+
+ <h2><a href="#index"><a name="verpin">Doing VERP and personalisation with exim and Mailman</a></a></h2>
+ <h3><a href="#index"><a name="verpmm">VERP within Mailman</a></a></h3>
- <h2>Doing VERP with Exim and Mailman</h2>
+ <p><a href="http://cr.yp.to/proto/verp.txt">VERP</a> (Variable
+ Envelope Return Paths) encodes the (original) receipient address
+ in the sender address. The reason for doing this is that it means
+ bounces are sent to an address which has the original recipient
+ address encoded in it - meaning you know which recipient address
+ caused the bounce. This makes automatic bounce handling very
+ effective - the normal method of parsing the bouncing address out
+ of the bounce message is very prone to failure, especially in the
+ case of foreign MTAs which use different addressing standards, or
+ where mail forwarding is involved.</p>
<p>VERP will send one email, with a separate envelope sender
- (return path), for each of your subscribers -- read the
- information in <tt>~mailman/Mailman/Default.py</tt> for the
- options that start with <tt>VERP</tt>. In a nutshell, all you
- need to do to enable VERP with Exim is to add these lines to
+ (return path), for each of your subscribers - this means that it
+ will generate more traffic since you cannot bundle up deliveries
+ together. An <a
+ href="http://www.python.org/cgi-bin/faqw-mm.py?req=show%26file=faq04.012.htp">analysis
+ of the costs of VERP</a> can be found in the <a
+ href="http://www.python.org/cgi-bin/faqw-mm.py">Mailman FAQ
+ Wizard</a>.</p>
+
+ <p>VERP settings within Mailman are done on a per-installation
+ basis - ie they affect all the lists within the installation. To
+ configure VERP within Mailman read the information in
+ <tt>~mailman/Mailman/Default.py</tt> for the options that start
+ with <tt>VERP</tt>. In a nutshell, all you need to do to enable
+ VERP with Exim is to add these lines to
<tt>~mailman/Mailman/mm_cfg.py</tt>:</p>
<pre>
VERP_PASSWORD_REMINDERS = 1
VERP_PERSONALIZED_DELIVERIES = 1
- VERP_DELIVERY_INTERVAL = 1
VERP_CONFIRMATIONS = 1
+ VERP_DELIVERY_INTERVAL = 1
</pre>
- <p>(The directors/routers above are smart enough to deal with VERP
+ <p>(The router and ACLs above are smart enough to deal with VERP
bounces.)</p>
+ <p>This configuration on its own will make the monthly password
+ reminders, confirmations and all list postings be sent out using
+ VERP</p>
+
+ <p>If you wish to have the advantages of VERP with a lower
+ bandwidth cost, you can enable VERP on occasional list postings
+ rather than on every posting. Mailman will still VERP on all
+ password reminders and confirmations (these are already inherently
+ single recipient mailings), but only on occasional list postings.
+ To make Mailman use VERP on every twentieth list posting (using
+ bulk delivery for the other 19), change:-</p>
+
+<pre>
+ VERP_DELIVERY_INTERVAL = 20
+</pre>
+
+ <p>The downside to doing this is that Mailman may fail to notice a
+ bouncing address if it does not receive at least one bounce per
+ day, so ideally this approach should only be taken if the lists
+ have more than 20 message per day throughput.</p>
+
+ <h3><a href="#index"><a name="persmm">Mailing list personalisation by Mailman</a></a></h3>
+
+ <p>Mailman can also personalise each message it sends out on a
+ list. This allows, for example, the recipient's own address to
+ appear as the To: header, or information specific to them to be
+ placed in the mail footer (although at present personalisation can
+ only be done for normal mail delivery - not for digest
+ subscribers). This personalisation comes at a cost of an
+ individual message per recipient (ie same bandwidth requirements
+ as full VERP) and some processing costs for Mailman.</p>
+
+ <p>To enable personalisation, add the following configuration item
+ to <tt>~mailman/Mailman/mm_cfg.py</tt> (you should also set the
+ VERP settings from above since you have already incurred the costs
+ of VERP):-</p>
+
+<pre>
+ OWNERS_CAN_ENABLE_PERSONALIZATION = 1
+</pre>
+
+ <p>You will then find that in the list administration web
+ interface a new set of options has appeared in the <i>Non-digest
+ options</i> section.</p>
+
+ <h3><a href="#index"><a name="verpex">VERP expansion by exim rather than Mailman</a></a></h3>
+
+ <p>One drawback of VERP is that as well as increasing the
+ bandwidth outgoing mail requires, it also causes Mailman to send
+ one separate message per recipient from Mailman to exim - causing
+ exim to have many many more queue entries and consequently more
+ queue disk space. For example a 20,000 recipient list would
+ require 400MB minimum temporary queue storage for each 20KB
+ message sent to the list. There are also issues of increasing
+ disk traffic/throughput and losing some disk caching
+ advantages.</p>
+
+ <p>These local load problems can be overcome by doing the VERP
+ expansion as the message is sent out from the MTA over network
+ SMTP rather than as the message is injected into the MTA. It will
+ come as no surprise that exim can be configured to do just
+ this.</p>
+
+ <p>Firstly we need to pick up outgoing Mailman mail and send it to
+ a specialised VERP transport. This is done using a router which
+ should be placed just before your normal <tt>dnslookup</tt> router
+ for remote addresses:-</p>
+
+<pre>
+ # Pick up on messages from our local mailman and route them via our
+ # special VERP-enabled transport
+ #
+ mailman_verp_router:
+ driver = dnslookup
+ # we only consider messages sent in through loopback
+ condition = ${if eq{$sender_host_address}{127.0.0.1}{yes}{no}}
+ # we do not do this for traffic going to the local machine
+ domains = !+local_domains:!+mm_domains
+ ignore_target_hosts = <; 0.0.0.0; \
+ 127.0.0.0/8; \
+ ::1/128; \
+ fe80::/10; \
+ ff00::/8
+ # only the un-VERPed bounce addresses are handled
+ senders = "*-bounces@*"
+ transport = mailman_verp_smtp
+</pre>
-<h2>Virtual Domains</h2>
+ <p>Addresses selected by this router should then be passed on to
+ an SMTP transport that does VERP expansion. This should be placed
+ anywhere within the transport section:-</p>
+
+<pre>
+ # Mailman VERP envelope sender address formatting. This seems not to use
+ # quoted-printable encoding of the address, but instead just replaces the
+ # '@' in the recipient address with '='.
+ #
+ mailman_verp_smtp:
+ driver = smtp
+ # put recipient address into return_path
+ return_path = \
+ ${local_part:$return_path}+$original_local_part=$original_domain@${domain:$return_path}
+ # must restrict to one recipient at a time
+ max_rcpt = 1
+ # Errors-To: may carry old return_path
+ headers_remove = Errors-To
+ headers_add = Errors-To: \
+ ${local_part:$return_path}+$original_local_part=$original_domain@${domain:$return_path}
+</pre>
+
+ <p>Once this has been configured, Mailman can be set to not do
+ VERP expansion on normal list deliveries - the VERP
+ configuration should now look like:-</p>
+<pre>
+ VERP_PASSWORD_REMINDERS = 1
+ VERP_PERSONALIZED_DELIVERIES = 1
+ VERP_CONFIRMATIONS = 1
+ VERP_DELIVERY_INTERVAL = 0
+</pre>
+
+ <p>If you have set personalisation on any list, this will still be
+ handled, and VERPed, by Mailman.</p>
+
+ <h2><a href="#index"><a name="virdom">Virtual domains</a></a></h2>
<p>One approach to handling virtual domains is to use a separate
Mailman installation for each virtual domain. (Currently, this is
the only way to have lists with the same name in different virtual
domains handled by the same machine.)</p>
- <p>In this case, the <tt>MAILMAN_HOME</tt> and
- <tt>MAILMAN_WRAP</tt> macros are useless -- you can remove them.
- Change your director (router) to something like this:</p>
+ <p>In this case, you must change the <tt>MM_HOME</tt> macro to
+ something like this:-</p>
<pre>
- require_files = /virtual/${domain}/mailman/lists/${lc:$local_part}/config.pck
+ MM_HOME=/virtual/${lc::$domain}/mailman
</pre>
- <p>and change your transport like this:</p>
+ <p>and modify the <tt>mm_domains</tt> domain list appropriately.</p>
+
+ <h2><a href="#index"><a name="lispol">List policy management</a></a></h2>
+
+ <p>Most list policy handling is done within Mailman using the Web
+ GUI. However some issues may be better handled by the MTA,
+ especially matters of overall site policy (not just mailing list
+ policy). For example you may wish to do virus or spam scanning on
+ incoming messages.</p>
+
+ <p>In general you exclude outgoing list mail from any policy
+ controls. This is because outgoing list mail has already been
+ through the policy controls on the way into the system.
+ Additionally spam scanning (for example) is a machine intensive
+ operation, and scanning a message that has already been scanned,
+ and then replicated to many recipients, is going to be very
+ expensive as well as ineffective. For this reason you will
+ normally have an <tt>accept</tt> clause early on in your ACLs that
+ causes Mailman generated traffic to bypass the machine intensive
+ checks.</p>
+
+ <h3><a href="#index"><a name="conpol">Content scanning</a></a></h3>
+
+ <p>I would recommend that mailing lists now scan for both spam and
+ viruses on incoming mail - this is due to the potential for a
+ compromised windows machine belonging to a subscriber managing to
+ distribute unwanted content via the list. This causes problems
+ not only to the list reputation, but also to the list manager who
+ will get many many bounces from subscribers who do content
+ scanning on their own incoming mail.</p>
+
+ <p>The best way to do this is using the <a
+ href="http://duncanthrax.net/exiscan-acl/">exiscan</a> extension
+ along with a virus scanner such as <a
+ href="http://duncanthrax.net/exiscan-acl/">clam-av</a> and a spam
+ content scanner such as <a
+ href="http://www.spamassassin.org/">SpamAssassin</a>. Configuring
+ these is beyond the scope of this document, however Tim Jackson
+ has a very good set of <a
+ href="http://www.timj.co.uk/linux/Exim-SpamAndVirusScanning.pdf">PDF
+ documentation</a> on integrating these.</p>
+
+ <p>One thing to note is that if you add full SpamAssassin headers
+ onto list messages this bulks up the messages significantly.
+ These headers are also available to list subscribers, which might
+ make it easier for someone malicious to work out how to evade your
+ spam scanning strategy. I would suggest that Spam headers are not
+ added for Mailman incoming mail, or minimal (short) headers added,
+ or that they are stripped somewhere. However having minimal
+ headers on means that you can, for example, moderate all messages
+ which have a given spam rating (as well as bouncing messages with
+ a very high rating).</p>
+
+ <h3><a href="#index"><a name="incpol">Incoming message checks</a></a></h3>
+
+ <p>You may wish to apply various checks to incoming messages to
+ ensure that they are sane. These may include:-</p>
+
+ <ul>
+ <li>DNSBL checks</li>
+ <li>Header checks</li>
+ <li>Sender callback verification</li>
+ </ul>
+
+ <p>However all of these do have some degree of false positive
+ ratings. You need to be aware of how much of your user base you
+ may alienate by imposing too strict a set of controls, and balance
+ that against the reduced amount of unwanted bulk mail.</p>
+
+ <h3><a href="#index"><a name="mmapol">Mailman specific ACL checks</a></a></h3>
+
+ <p>Lists should never receive bounce messages to the posting
+ address unless the bounced message is either a forgery using the
+ list address as the sender address, or the bouncing MTA is
+ terminally broken. In either of these cases we really are not
+ interested in receiving the messages and can reject them at SMTP
+ time with a clear conscience. The ACL to do this (as part of the
+ RCPT ACL) is:-</p>
<pre>
- command = /virtual/${domain}/mailman/mail/mailman \
- ${if def:local_part_suffix \
- {${sg{$local_part_suffix}{-(\\w+)(\\+.*)?}{\$1}}}
- {post}} \
- $local_part
- current_directory = /virtual/${domain}/mailman
- home_directory = /virtual/${domain}/mailman
+ # Reject bounce (null sender) messages to the list
+ reject message = "Recipient never sends mail so cannot cause bounces"
+ senders = :
+ domains = +mm_domains
+ condition = ${if exists{MM_LISTCHK} {yes}{no}}
+</pre>
+
+ <p>Additionally other mailman addresses do not generate mail (as
+ the envelope sender, although they may be mentioned in the header
+ addresses. The ACL is split into 2 so that it can be written
+ without the local_part condition wrapping.</p>
+
+<pre>
+ # Reject bounce (null sender) messages to the list
+ reject message = "Recipient never sends mail so cannot cause bounces"
+ senders = :
+ domains = +mm_domains
+ local_parts = \N^.*-(admin|join|leave|owner|request)$\N
+ reject message = "Recipient never sends mail so cannot cause bounces"
+ senders = :
+ domains = +mm_domains
+ local_parts = \N^.*-(subscribe|unsubscribe)$\N
+</pre>
+
+ <h3><a href="#index"><a name="smtune">SMTP callbacks</a></a></h3>
+
+ <p>Exim's SMTP callback feature is an even more powerful way to
+ detect bogus sender addresses than normal sender verification.
+ They are specially useful for checking envelope sender addresses
+ at RCPT time within SMTP, and have been to date the most effective
+ single anti-SPAM measure (however it should be noted that CBV is
+ hated vehemently by some mail admins, and does increase both
+ latency and traffic, as well as theoretically being a means to set
+ up a DDOS situation).</p>
+
+ <p>It is recommended that SMTP Sender CBV is not carried out on
+ messages to the Mailman bounce handlers, so that broken remote
+ MTAs (specifcally ones which send bounces with something other
+ than a null sender address) do not get excluded from being taken
+ off mailing lists</p>
+
+<pre>
+ # Do callback verification unless Mailman incoming bounce
+ reject !local_parts = *-bounces : *-bounces+*
+ !verify = sender/callout=30s,defer_ok
+
</pre>
-<h2>List Verification</h2>
+ <p>Callback verification can also be done on header addresses, but
+ care should be taken not to reject messages unnecessarily,
+ especially when the message is going to Mailman's bounce
+ processor</p>
+
+ <h2><a href="#index"><a name="lisver">List verification</a></a></h2>
<p>This is how a set of address tests for the Exim lists look on a
working system. The list in question is
- <tt>quixote-users@mems-exchange.org</tt>, and these commands were
- run on the <tt>mems-exchange.org</tt> mail server (<tt>"% "</tt>
+ <tt>testlist@list.example.com</tt>, and these commands were
+ run on the <tt>list.example.com</tt> mail server (<tt>"% "</tt>
indicates the Unix shell prompt):</p>
<pre>
- % exim -bt quixote-users
- quixote-users@mems-exchange.org
- router = mailman_main_router, transport = mailman_transport
-
- % exim -bt quixote-users-request
- quixote-users-request@mems-exchange.org
+ % exim -bt testlist@list.example.com
+ testlist@list.example.com
router = mailman_router, transport = mailman_transport
-
- % exim -bt quixote-users-bounces
- quixote-users-bounces@mems-exchange.org
+
+ % exim -bt testlist-request@list.example.com
+ testlist-request@list.example.com
router = mailman_router, transport = mailman_transport
-
- % exim -bt quixote-users-bounces+luser=example.com
- quixote-users-bounces+luser=example.com@mems-exchange.org
+
+ % exim -bt testlist-bounces@list.example.com
+ testlist-bounces@list.example.com
+ router = mailman_router, transport = mailman_transport
+
+ % exim -bt testlist-bounces+luser=example.com@list.example.com
+ testlist-bounces+luser=example.com@list.example.com
router = mailman_router, transport = mailman_transport
</pre>
that your Exim/Mailman installation is functioning perfectly,
though!</p>
+ <h2><a href="#index"><a name="problem">Possible Problems</a></a></h2>
+
+ <ul>
+
+ <li> Mailman will send as many <tt>MAIL FROM/RCPT TO</tt> as it
+ needs. It may result in more than 10 or 100 messages sent in one
+ connection, which will exceed the default value of Exim's
+ smtp_accept_queue_per_connection This is bad because it will
+ cause Exim to switch into queue mode and severely delay delivery
+ of your list messages. The way to fix this is to set mailman's
+ <tt>SMTP_MAX_SESSIONS_PER_CONNECTION</tt> (in
+ <tt>~mailman/Mailman/mm_cfg.py</tt>) to a smaller value than
+ Exim's <tt>smtp_accept_queue_per_connection</tt></li>
+
+ <li>Mailman should ignore Exim delay warning messages, even
+ though Exim should never send this to list messages. Mailman
+ 2.1's general bounce detection and VERP support should greatly
+ improve the bounce detector's hit rates.</li>
+
+ <li>List existence is determined by the existence of a
+ <tt>config.pck</tt> file for a list. If you delete lists by
+ foul means, be aware of this.</li>
+
+ <li>If you are getting Exim or Mailman complaining about user
+ ids when you send mail to a list, check that the
+ <tt>MM_UID</tt> and <tt>MM_GID</tt> match those of
+ Mailman itself (i.e. what were used in the configure script).
+ Also make sure you do not have aliases in the main alias file
+ for the list.</li>
+
+ </ul>
+
+ <h2><a href="#index"><a name="dochis">Document History</a></a></h2>
+
+ <ul>
+ <li>Originally written by Nigel Metheringham.</li>
+ <li>Updated by Marc Merlin for Mailman 2.1, Exim 4</li>
+ <li>Overhauled/reformatted/clarified/simplified by Greg
+ Ward.</li>
+ <li>Rehashed again by Nigel Metheringham</li>
+ </ul>
+
+ <h2><a href="#index"><a name="doccha">Document Changes</a></a></h2>
+
+ <ul>
+ <li><b>14 June 2004</b> Originally written by Nigel Metheringham.</li>
+ <li><b>28 June 2004</b> Fixed a problem with the Exim VERP
+ transport which caused locally aliased recipient addresses to
+ have incorrect VERP extensions added. Followed posting to
+ exim-users list by Arkadiusz Miskiewicz -- <i>Nigel Metheringham</i>.</li>
+ </ul>
+
- <h2>Document History</h2>
+ <h2><a href="#index"><a name="docfin">Final Word</a></a></h2>
- <p>Originally written by Nigel Metheringham. Updated by Marc
- Merlin for Mailman 2.1, Exim 4.
- Overhauled/reformatted/clarified/simplified by Greg Ward.</p>
+ <p>Like many documents of this type, it has evolved and taken on
+ contributions by many many helpful folks, mainly those on the
+ Mailman and exim mailing lists. To all of you, who have made
+ contributions yet had their names shamefully lost by me, <i>Thank
+ You</i>.</p>
<hr>
- <h4>$Id: mailman.html,v 1.3 2002/06/14 10:42:18 nigel Exp $</h4>
+ <h4>$Id: mailman21.html,v 1.4 2004/06/25 12:42:53 nigel Exp $</h4>
</body>
</html>
git://git.exim.org / exim-website.git / blobdiff