Add expansion variable $headers_added returning newline-sep list of headers

[users/heiko/exim.git] / doc / doc-txt / Exim4.upgrade

Upgrading Exim from Release 3.33 to 4.xx
----------------------------------------

Exim 4.00 represents the largest upheaval in Exim's history. There are a lot of
changes to the way some parts of Exim work, and a lot of incompatible changes
to the run time configuration file.

This document is in two parts. The first part contains instructions and
suggestions for how you might go about performing the upgrade. The second part
is a brief list of all the changes that have taken place. For full details of
all the new features, please consult the current version of the reference
manual.


HOW TO UPGRADE YOUR EXIM
------------------------

When you compile Exim 4, a Perl script called convert4r4 is built in the build
directory. It is not installed by the install script, because it is likely that
you will run it only once.

This script is provided to assist in updating Exim configuration files. It
reads an Exim 3 configuration file on the standard input, and writes a modified
file on the standard output. It also writes comments about what it has done to
the standard error file. It assumes that the input is a valid Exim 3
configuration file. A typical call to the conversion script might be

  ./convert4r4  </etc/exim/configure  >/etc/exim/configure.new

The output file MUST be checked and tested before trying to use it on a live
system. The conversion script is just an aid which does a lot of the "grunt
work". It does not guarantee to produce an Exim 4 configuration that behaves
exactly the same as the Exim 3 configuration it reads.

Each option change in the new file is preceded by an identifying comment. In
fact, the conversion script tends to make quite a mess of your configuration,
and you should expect to go through it afterwards and tidy it up by hand.

Unless you are running a very straightforward configuration, the automatic
conversion is likely to generate a non-optimal configuration. You should not
only check it thoroughly, but also run as many tests as you can, to ensure that
it is working as you expect. In particular, you should test address routing,
using -bt and -bv, and the policy controls, using -bh. If possible, you should
also do some live tests (i.e. send and receive some messages) before putting
Exim 4 into service.

If you have a very complicated configuration, it is possible that convert4r4
will break it in some situations, which is why thorough testing is strongly
recommended.

                   *********************************
                   ***** You Have Been Warned ******
                   *********************************


HOW TO MOVE FROM AN EXIM 3 RELEASE TO AN EXIM 4 RELEASE
-------------------------------------------------------

One way of upgrading to Exim 4 from a version 3 release is as follows:

1. Suppose your run time configuration file is called /usr/exim/configure, and
   you want to continue with this name after upgrading. The first thing to do
   is to make another copy of this file called, say, /usr/exim/configure.r3.

2. Rebuild your existing Exim to use the copy of the run time configuration
   file instead of the standard file. Install this version of Exim and HUP your
   daemon. You can check on the name of the configuration file by running

     exim -bP configure_file

   Ensure that everything is running smoothly. You now have something you can
   fall back to. IMPORTANT: when you do this re-install, you should also
   re-install the utilities because four of them (exicyclog, eximon, exinext,
   and exiwhat) also refer to the configuration file.

3. Build the new release, configured to use the standard configuration file.

4. Use the convert4r4 utility to upgrade your configuration file for the new
   release. After running the conversion utility, check the file by hand, and
   tidy it up.

5. Test, test, test! And test some more!

6. You can run complete tests, including actual deliveries, from an uninstalled
   binary, but you have to tell it where it is, so that any re-executions can
   be done. You can do this by temporarily inserting a setting such as

   exim_path = /source/exim/exim-4.00/build-SunOS5-5.8-sparc/exim

   into the run time configuration. If you want to, you can also insert
   settings for spool_directory and log_file_path to divert those away from
   their normal places. Remember to remove these temporary settings when you
   eventually install the binary for real.

7. The new installation script installs the new release as exim-4.00-1, and
   set a symbolic link called "exim" to point to it. The old version of Exim
   will be renamed to something like exim-3.33-1.

8. You can now easily change between the new and old releases simply by moving
   the symbolic link and HUPping your daemon. The format of message files on
   Exim's spool has _not_ changed, so there should be no problem in changing
   between releases while there are messages on the queue.

9. HOWEVER: If you do change back and forth between releases, you must also
   change the utilities exicyclog, eximon, exinext, and exiwhat if you are
   going to use them. Installing Exim 4 will have left the old versions with a
   .O suffix. It might be helpful to rename these so that you don't lose them.


WHAT HAS NOT CHANGED IN EXIM 4.00
---------------------------------

The basic overall philosophy, design, and process structure has not changed.
The format of spool files is the same. The transports have had only minor
modifications. The command line options remain the same, with a couple of
additions.

The general run time configuration approach has not changed, but the actual
details of the configuration file are different.

The Exim monitor has not changed, and there have been only very minor changes
to other Exim utilities.


WHAT HAS CHANGED IN EXIM 4.00
-----------------------------

The rest of this document lists the very many changes that have taken place.
I'm going to give only brief details here, because this part of the document is
intended as a way of alerting you to areas of difference. The reference manual
describes how the new features work in detail.


Named domain, host, address, and local part lists
-------------------------------------------------

A new feature makes it possible to give names to lists of domains, hosts,
addresses, and local parts. The syntax used is

  domainlist    <name> = <a domain list>
  hostlist      <name> = <a host list>
  addresslist   <name> = <an address list>
  localpartlist <name> = <a list of local parts>

For example:

  domainlist  local_domains = *.my.domain
  addresslist bad_senders = cdb;/etc/badsenders

These lists are referenced in options by giving the name preceded by a + sign.
For example, in a router you might have

  domains = +local_domains

At first sight, these lists might seem to be the same as macros, but they are
not. Macros are just textual substitutions. If you write

  ALIST = host1 : host2
  auth_advertise_hosts = !ALIST

it probably won't do what you want, because that is exactly the same as

  auth_advertise_hosts = !host1 : host2

Notice that the second host name is not negated. However, if you use a host
list, and write

  hostlist alist = host1 : host2
  auth_advertise_hosts = ! +alist

the negation applies to the whole list, and so that is equivalent to

  auth_advertise_hosts = !host1 : !host2

These named lists also have a performance advantage. When Exim is routing an
address or checking an incoming message, it caches the result of tests on the
lists. So, if you have a setting such as

  domains = +local_domains

on several of your routers, the actual test is done only for the first one.
However, this caching works only if there are no expansions within the list
itself or any sublists that it references. In other words, caching happens only
if the list is known to be the same each time it is referenced.

By default, there may be up to 16 named lists of each type. This limit can be
extended by changing a compile-time variable.

The use of domain and host lists is recommended for concepts such as local
domains, relay domains, and relay hosts. The default configuration is set up
like this.


Processing of domain, host, local part, and address lists
---------------------------------------------------------

The handling of these lists is now more uniform. Every list is expanded as a
single string before it is used. (In Exim 3, some options were expanded and
some weren't, and query-style lookup items were then re-expanded.)

If an expansion is forced to fail, Exim behaves as if the item has not been
found in the list.

The confusing $key variable has been abolished. When processing a domain list,
$domain contains the relevant domain and can be used in generating database
queries. Other appropriate variables are set when processing other kinds of
list; $sender_host and $sender_host_address for checking incoming hosts and
$host and $host_address for checking outgoing hosts.

Note that this means that any \ and $ characters in regular expressions must be
escaped if they appear in one of these lists. The new expansion feature to turn
off expansion (\N ... \N) which is described below can be helpful here.

IMPORTANT: The details of the processing of address lists has been revised. In
particular, the handling of the case of an item that is a single-key lookup has
changed. It no longer looks up the domain on its own before looking up the
complete address. You need to supply an explicit "*@" before the lookup if you
want just the domain to be looked up. Please check the manual for full details.

If an item in a host list is the empty string, it matches only when no host is
defined. If used when checking an incoming message, it matches only when the
message is arriving by SMTP on the standard input from a local process (using
-bs). This provides a way of distinguishing between SMTP mail from local
processes and from remote hosts.

The +allow_unknown and +warn_unknown settings for host lists have been replaced
by a single item, +include_unknown. By default, failure to find a host name
when needed causes Exim to behave as if the host does not match the list, but
if +include_unknown is set, the opposite behaviour happens. Whenever
+include_unknown is invoked, the incident is logged.


Merge of Directors and Routers
------------------------------

There are no longer any directors in Exim 4. There are just routers. All
addresses are passed to a single list of routers which typically makes use of
the "domains" option to choose which way to handle specific groups of domains.

A consequence of this is that the code no longer contains any concept of "local
domains". However, a typical configuration will probably define a named domain
list (see above) called local_domains, and use it to control routing something
like this:

  route_remote:
    driver = dnslookup
    domains = ! +local_domains
    transport = remote_smtp
    no_more

  system_aliases:
    ....

The first router does DNS routing for all domains that are not in the named
list of local domains, and no_more ensures that it is the last router for those
domains. All other domains fall through to the system_aliases and subsequent
routers. For a complete configuration example, look at the default
configuration file in src/configure.default.


Router Actions
--------------

The concept of how the routers work is as follows:

A number of pre-conditions are tested (details below). If any of them fails,
control is passed to the next router. We say "the router is skipped". Otherwise
the router is run, and can yield one of several different results:

. accept: The router accepts the address, and either queues it for a transport,
or generates one or more "child" addresses. Processing the original address
ceases, unless "unseen" is set on the router, in which case the address is
passed to the next router. Processing of any child addresses starts with the
first router by default, or at the router defined by redirect_router if it is
set. This may be any router in the list.

. decline: The router declines to accept the address because it does not
recognize it at all. The address is passed to the next router, unless no_more
is set, in which case the address fails.

. pass: The router recognizes the address, but cannot handle it itself. It
requests that the address be passed to another router. This overrides no_more.
By default the address is passed to the next router, but this can be changed by
setting pass_router. However, in this case (unlike redirect_router) the named
router must be below the current router (to avoid loops).

. fail: The router determines that the address should fail, and queues it for
the generation of a bounce message. There is no further processing of the
original address, unless "unseen" is set.

. defer: The router cannot handle the address at the present time. (For
example, a database may be offline.) No further processing of the address
happens in this delivery attempt. It is tried again next time.

. error: There is some error in the router (for example, a syntax error in
its configuration). The action is as for defer.


Router pre-conditions
---------------------

In Exim 3 there are some strange interactions between the generic options that
test things before running a director or router and the no_more test that
happens afterwards.

In Exim 4 it is all more straightforward. If any of the pre-condition tests
fail, the router is skipped and control passes to the next router. The no_more
option has an effect only if the router is actually run - that is, if all the
pre-condition tests succeed. The order in which these tests are run is:

  verify status, expn status, domains, local_parts, check_local_user

If all those match, the debug_print string is output when debugging. Exim then
goes on to test

  senders, require_files, condition

Note that require_files comes near the end of the list, so you cannot use it to
check for the existence of a file in which to lookup up a domain, local part,
or sender. However, as these options are all expanded, you can use the "exists"
expansion condition to make such tests. The require_files option is intended
for checking files that the router may be going to use internally, or which are
needed by a specific transport (e.g. .procmailrc).

In Exim 4, local part prefixes and suffixes are recognized and removed before
any of the other pre-condition tests are done (in Exim 3 they were removed
afterwards). Note that this means that the local_parts option now tests the
local part without its prefix or suffix.

If you want to use local parts that include any affixes in a pre-condition
test, you can do so by using a "condition" option that uses the variables
$local_part, $local_part_prefix, and $local_part_suffix as necessary.


A New Set of Routers
--------------------

The two sets of routers and directors of Exim 3 have been replaced by a single
set of routers for Exim 4. These are as follows:

. accept        Always accepts an address. It has no private options.

. dnslookup     Routes by DNS lookup (descended from lookuphost).

. ipliteral     Routes IP literal addresses (unchanged).

. iplookup      Special-purpose lookup router (unchanged).

. manualroute   Routes domains from explicit data (descended from domainlist).

. queryprogram  Routes addresses by running a program (detail changed).

. redirect      Redirects addresses; handles all the functions previously
                supported by aliasfile, forwardfile, and smartuser without
                a transport.


Saving duplication of effort while routing
------------------------------------------

Early versions of Exim used to copy the routing of one address for all other
addresses in the same domain, thereby possibly saving some repeated DNS
lookups. This feature was removed for release 2.12, after the possibility of
varying the router actions according to the local part (the local_parts option)
was added. (In fact, the use of $local_part could have broken it earlier.)

For Exim 4, I have added an option called same_domain_copy_routing to the
dnslookup and manualroute routers. When one of these routers routes an address
to a remote transport and this option is set, any other addresses in the
message that have the same domain are automatically given the same routing, but
only if the router does not set headers_add or headers_remove, and does not
`widen' the domain during the routing.


Generic Router Options
----------------------

. The global locally_caseless option is replaced by a generic router option
  called caseful_local_part. By default, routers handle local parts caselessly.

. check_local_user is now a generic option that is needed to check for a local
  account. Typically used on redirect (for user's forward files) and on accept
  (for local deliveries).

. The setting self=local has been removed (since there's no concept of local
  domains in the code). The same kind of effect can be achieved by using
  self=reroute or self=pass.

. expn is now a generic router option.

. local_part_prefix and local_part_suffix are now generic router options,
  replacing prefix and suffix on directors.

. Exim 3 has two logging styles for delivery, depending on whether the domain
  is a local domain or not. For local domains, the address is given just as the
  local part - this makes these deliveries easier to spot in the log. In Exim 4
  there's no concept of local domains, so this functionality cannot be
  automatic. Instead, there's a generic router option called log_as_local which
  requests "local-style" logging. This option defaults on for the "accept"
  router, and off for all the others.

. There's an option called retry_use_local_part which is the default for any
  router that has check_local_user set, and it applies to routing delays. (The
  same option for transports applies to transport delays.)

. transport_home_directory and transport_current_directory are new generic
  options on all routers. They set up default values for home_directory and
  current_directory on the transport to which they route an address. Any
  settings in the transport override.

. If transport_home_directory is not set, but check_local_user is set, the
  user's home directory is used as a default value.

. The special fudge that exists in Exim 3 for handling home_directory settings
  in forwardfile directors is not needed in Exim 4. It has therefore been
  removed.

. The new_director option in Exim 3 allows the direction of redirected
  addresses to start at a given director, instead of the first one. In Exim 4,
  this option is now called redirect_router. The option is used when a redirect
  router succeeds, and when a queryprogram router returns a "redirect"
  response.

. There is a new option called pass_router, which specifies the router to go to
  when a router "passes" on an address. The named router must follow the
  current router (to avoid routing loops). Note: if a router declines, control
  always passes to the next router, unless no_more is set.

. There is a new router option called address_data. This is set to a string
  which is expanded just before the router is run, that is, after all the
  pre-tests have succeeded. If the expansion is forced to fail, the router
  declines. Other expansion failures cause delivery of the address to be
  deferred.

  When the expansion succeeds, the value is retained with the address, and can
  be accessed using the variable $address_data. Even if the router declines or
  passes, the value remains with the address, though it can be changed by
  another address_data setting on a subsequent router. If a router generates
  child addresses, the value of $address_data propagates to them.

  The idea of address_data is that you can use it to look up a lot of data for
  the address once, then then pick out parts of the data later. For example,
  you could use an LDAP lookup to return a string of the form

    uid=1234 gid=5678 mailbox=/mail/xyz forward=/home/xyz/.forward

  In the transport you could then pick out the mailbox by a setting such as

    file = ${extract{mailbox}{$address_data}}

  This makes the configuration file less messy, and also reduces the number of
  lookups. (Exim does cache the most recent lookup, but there may be several
  addresses with different lookups.)

. When a transport is run for several addresses simultaneously, the values of
  $address_data, $local_part_data, and $domain_data are taken from the first
  address that the transport handles. However, the order in which multiple
  addresses are processed is not defined. You therefore need to be careful if
  you want to use these variables with multiple addresses. The smtp transport
  is the only one which by default handles multiple addresses.

. When an address is routed by a router with the "unseen" option set, a "clone"
  address is created, and it starts being routed at the next router. (This is
  what people expect. In Exim 3 it starts at the top - in simple cases that has
  the same effect because of the anti-looping rule, but if aliases are involved
  it sometimes doesn't do what you want.)

. The way that require_files works has been changed. Each item in the list is
  now separately expanded as the test proceeds. The use of leading ! and +
  characters is unchanged. However, user and group checking is done differently.
  Previously, seteuid() was used, but seteuid() is no longer used in Exim (see
  "Security" below). Instead, Exim now scans along the components of the file
  path and checks the access for the given uid and gid. It expects "x" access
  on directories and "r" on the final file. This means that file access control
  lists (on those operating systems that have them) are ignored.


Other Consequences of the Director/Router Merge
-----------------------------------------------

. The -odqr option is abolished, as there is no inbuilt concept of remote
  domains.

. The -odqs option is equivalent to queue_smtp_domains = *.

. queue_remote_domains is renamed queue_domains, and applies to any domain.

. The -ql option now suppresses remote delivery; routing always happens.

. The "remote" facility of queue_only_file has been removed.

. The match_directory option for forwardfile and localuser has been entirely
  abolished. Its function can be achieved using the "condition" option in
  conjunction with check_local_user.

. When an address is being verified, if it is redirected to a single new
  address, verification continues with that address. If it is redirected to
  more than one address, verification ceases with a success result. (In Exim 3,
  this applied only to aliasing, not to forwarding.)


The dnslookup router
--------------------

This router replaces the lookuphost router of Exim 3. It is much the same,
except that the "gethostbyname" option has been removed. It now does only DNS
routing - hence the change of name. Routing using gethostbyname() can be done
by the manualroute router.


The manualroute router
----------------------

This is the new name for the domainlist router, supposedly to make its function
clearer and to avoid confusion with the "domainlist" that is used to set up
named domain lists. Several things have been removed and reorganized.

. The old search mechanism (route_file, route_query, route_queries,
  search_type) have been removed. Instead there is a new option called
  route_data, which is an expanded string. It should expand to a single routing
  entry. If the expansion ends up empty (or is forced to fail), the router
  declines. The route_list option still exists, for convenient listing of a few
  inline routes.

. There is no longer any MX processing function in this router. The keywords
  bydns_mx and bydns_a have been removed, leaving just

    bydns  => find IP addresses from address records in the DNS
    byname => find IP addresses by calling gethostbyname()

  The default lookup type is "byname", and this can be omitted from a route
  data line. If an IP address is given, both "byname" and "bydns" are ignored
  (so typically you omit this field).

. The qualify_single and search_parents options have also been removed.

. A transport is always required to be set, unless verify_only is set.

. The host_find_failed option can be set to "decline", to cause the router to
  decline if it can't find an IP address for a listed host.

. If manualroute routes to a local transport, there is no need to specify
  byname or bydns in the routing data. Any supplied host list is passed as a
  string in $host, but $host_address is unset.


The queryprogram router
-----------------------

This router has been re-designed:

. You must now specify a user and group for the program to be run using
  command_user and (if necessary) command_group. It no longer defaults to
  "nobody". These options are expanded.

. The command is now split up and each argument expanded separately, as happens
  for the pipe transport. The command name is also expanded.

. The return value "forcefail" has been renamed "fail", and it causes delivery
  to fail. (The original usage of "fail" meaning "decline" has finally been
  removed.)

. The $route_option variable, which queryprogram used to be able to set has
  been abolished. A facility to set the new $address_data variable replaces it.

. The string returned from queryprogram must now be one of:

  DECLINE
  FAIL text
  DEFER text
  PASS
  FREEZE text
  REDIRECT text
  ACCEPT TRANSPORT=transport HOSTS=host list LOOKUP=byname|bydns DATA=text

The text returned for "redirect" is a list of new addresses. The text for FAIL
is returned in the SMTP dialogue when the router is run as part of address
verification. It is also logged. The text for DEFER and FREEZE is just logged.

The data items in the "accept" return can be given in any order, and all are
optional. If no transport is included in the "accept" return, the router's
default transport is used. The host list and lookup type are needed only if the
transport is an smtp transport that does not itself have a host list. The
default lookup type is "byname". If the "data" field is set, its value is
placed in the $address_data variable.


The redirect router
-------------------

This router replaces forwardfile, appendfile, and the use of smartuser without
a transport. It has two mutually exclusive options for specifying the data that
it uses. If "file" is set, the data is taken from a file. Otherwise "data" must
be set, and the data is the expanded value of that option.

The data may be an alias list, possibly including special entries such as
:fail:, or it may be a list of filtering instructions.

If "file" is set, but the file does not exist or is empty, or its contents have
no effect (entirely comments, or a filter that does nothing), the router
declines. This also happens if the expansion of "file" is forced to fail. Any
other expansion failure causes the router to defer.

Ownership of the file is checked if check_local_user is set or if owners is
set, unless check_owner is explicitly set false.

Likewise, the group is checked if owngroups is set, or if check_local_user is
set and a modemask not containing 020 is set, unless check_group is explicitly
set false.

If "data" is set, a forced expansion causes the router to decline. This also
happens if "data" is an empty string or a string that causes nothing to be
generated and no action to be taken.

Because "data" is now used for traditional /etc/aliases lookups, an empty alias
no longer gives an error. It behaves in the same way as :unknown: (which is
still recognized, but ignored).

. If no_repeat_use is set, the router is skipped if _any_ ancestor of the
  current address was routed by this router. This pre-test happens before any
  of the others. (Contrast the default loop avoidance logic, which skips a
  router if an ancestor with the same local part was routed by the router.)

. If include_directory is set, :include: files are constrained to this
  directory.

. When an address is redirected to a file or a pipe, $address_file or
  $address_pipe (as appropriate) is set when expanding the value of
  file_transport or directory_transport.

. There are new options forbid_filter_readfile and forbid_filter_run to lock
  out the use of the new ${readfile and ${run expansion items in filters.

. If one_time is set, forbid_pipe, forbid_file, and forbid_filter_reply are
  forced to be true, and headers_add and headers_remove are forbidden.


Generic transport options
-------------------------

. All remote deliveries are now done in subprocesses running with specified
  UIDs and GIDs. (Formerly, only remote parallel deliveries were done in
  subprocesses.) As a result, user and group are now generic options that can
  be used on all transports. The default for both local and remote transports
  is to run as the Exim user and group. For remote transports, this should not
  normally be changed, but if it is, the user or group should be able to access
  the hints databases, though failure to open a hints database is always
  ignored.

  If it turns out that a transport user is in the never_users list, Exim now
  defers delivery and writes to the panic log. (Previously it just ran the
  delivery as "nobody".) Because subprocesses (usually running as "exim")
  are now always used for remote deliveries, you should *not* include "exim" in
  the never_users list.

. initgroups is now also a generic transport option.

. home_directory and current_directory are generic options on all transports,
  though some transports (e.g. smtp) make no use of them. If they are unset,
  values supplied by the router are used.

. The message_size_limit option is now expanded, which makes it possible to
  have different limits for different hosts, for example.


Multiple (batch) deliveries in the appendfile, lmtp, and pipe transports
------------------------------------------------------------------------

The options controlling batch deliveries, including BSMTP, were a mess, and
have been reworked.

. The batch option has been removed from all three transports, and the bsmtp
  and bsmtp_helo options have been removed from appendfile and pipe.

. The batch_max option defaults to 1 in all three transports.

. A new option called use_bsmtp has been added to appendfile and pipe. When
  set, the message is delivered in BSMTP format. If you want to have a HELO
  line at the start of the message, you can configure this by making use of the
  message_prefix option. You must include the terminating newline.

. A new option called batch_id has been added to all three transports.

Batching is now achieved by setting batch_max to a value greater than 1. This
is recommended for lmtp. When multiple addresses are routed to the same
transport that has a batch_max value greater than one, the addresses are
delivered in a batch, subject to certain conditions:

. If any of the transport's options contain a reference to "$local_part", no
  batching is possible.

. If any of the transport's options contain a reference to "$domain", only
  addresses with the same domain are batched.

. If batch_id is set, it is expanded for each address, and only those addresses
  with the same expanded value are batched.

. Batched addresses must also have the same errors address (where to send
  delivery errors), the same header additions and removals, the same user and
  group for the transport, and if a host list is present, the first host must
  be the same.


The appendfile transport
------------------------

. The prefix and suffix options have been renamed message_prefix and
  message_suffix to avoid confusion with address affixes. The default values,
  which are suitable for mbox deliveries, now apply only if "file" is set and
  use_bsmtp is not set. Otherwise, the default values for these options are
  unset. They can, of course, always be overridden.

. If "directory" is set (which means that "file" is not set), the check_string
  and escape_string options now default unset.

. The require_lockfile options has been abolished. If use_lockfile is set, a
  lock file is always required.

. The quota_filecount option is now expanded.

. The create_file option now also applies when delivering into an individual
  file in a given directory, as well as when appending to a single file. In the
  case of maildir delivery, the restriction applies to the top directory of the
  maildir folder.

. There's a new option called directory_file which is expanded to form the
  final leaf name of files when "directory" is set, but neither maildir nor
  mailstore is set. The default is "q${base62:$tod_epoch}-$inode", which
  reproduces the old fixed value. The variable $inode is available only when
  expanding this new option.


The pipe transport
------------------

. The prefix and suffix options have been renamed message_prefix and
  message_suffix to avoid confusion with address affixes. The default values
  that are suitable for vacation deliveries now apply only if use_bsmtp is not
  set. Otherwise the default values for these options are unset. They can, of
  course, always be overridden.


The smtp transport
------------------

. The badly-named batch_max option is now called connection_max_messages.

. If hosts_randomize is set, it now affects host lists that come from a router
  as well as the contents of the "hosts" option, but only if the hosts were not
  obtained from MX records. Typically, such lists come from the manualroute
  router. This change means that the router can provide the same host list for
  multiple addresses - causing them all to be sent to the transport at once.
  Randomizing is then done each time the transport is called. (If you set
  hosts_randomize on the router, the randomizing happens for each address.)

. The way that smtp operates when there are multiple addresses to be sent to
  the same host is now different. Previously, the transport was called many
  times, with a maximum of max_rcpt addresses per call. Each call made a new
  connection to the host. When remote_max_parallel = 1, all the addresses are
  now passed to the transport at once. It makes a single TCP/IP call, but may
  send multiple copies of the message, each with no more than max_rcpt
  recipients.

  When remote_max_parallel is greater than 1, a heuristic is used. The number
  of addresses passed to a single call of the transport is limited to

    (the total number of recipients) / (the value of remote_max_parallel)

  so, for example, if there are 100 recipients and remote_max_parallel is 2, no
  more than 50 are passed in one call, even if max_rcpt is 100. (The idea is
  that the other 50 will be passed to another call running in parallel.)

  There is an option of the smtp transport called connection_max_messages
  which limits the number of messages, or copies of a message, that Exim sends
  down a single TCP/IP connection. This applies both to this mechanism for
  multiple copies of a single message, and the re-use of a TCP/IP connection
  for sending other messages destined for the same host, after a delivery
  delay. The default value is 500.

. The "interface" option is now expanded. If the result is a forced failure or
  an empty string, it is ignored. Otherwise, the result must be a list of IP
  addresses. The first one of the correct type (IPv4 or IPv6) for the outgoing
  connection is used. If there isn't one of the correct type, the option is
  ignored.

. At the start of running the transport, the value of $host is taken from the
  first host in a multi-host list. However, just before the transport connects
  to a host, the value is changed to refer to that particular host. (This
  applies to $host_address as well.) This means that options such as helo_data
  and the tls_options can be made host-specific.

. The tls_verify_ciphers option has been renamed tls_require_ciphers, in order
  to leave the word "verify" as something that refers to the verification of
  certificates.

. The resolution of hosts and fallback_hosts used to look up MX records. This
  was some kind of ancient silliness that I recently noticed. These are
  definitely hosts, not mail domains. Exim 4 just looks up address records.
  As a consequence of this, the mx_domains option of the smtp transport is
  removed.

. The authenticate_hosts option has been renamed as hosts_try_auth. A new
  option called hosts_require_auth has been added; if authentication fails for
  one of these hosts, Exim does _not_ try to send unauthenticated. It defers
  instead. The deferal error is detectable in the retry rules, so this can be
  turned into a hard failure if required.


The System Filter
-----------------

. The system filter options that were called message_filter_xxx have all been
  renamed as system_filter_xxx.

. The value of system_filter is expanded.

. message_filter_directory_transport and message_filter_file_transport are now
  both expanded before use. If the filter set up any file or pipe deliveries,
  $address_file and $address_pipe are set as appropriate while doing the
  expansions.

. message_filter_directory2_transport has been removed. The effect of using
  different directory-style transports can be achieved by specifying a suitable
  expansion string to system_filter_directory_transport.

. When a system filter added recipients to a message, Exim 3 added an
  X-Envelope-To: header, listing the real recipients (up to 100). This has been
  abolished because you can do this kind of thing using "headers_add" nowadays.

. The "fail" command has been extended to allow for two different messages, one
  for Exim's log and the other to be returned to the sender. The syntax is

    fail "<<log message>>user message"

  That is, if the first two characters of the message are "<<" the following
  text, up to ">>", is written to the log, and the remainder is returned to the
  user. If there is no log message, the user message is logged. The motivation
  for this feature was to reduce the amount of text logged, while being able to
  send quite long (maybe even multi-line) messages back to the sender.


Changes to Lookups
------------------

. Oracle support is available. It works like the mysql and pgsql support,
  except that there is no "database name" involved, and the "host name" field
  is used for what is called "service name" in Oracle. This often looks like a
  host name. Also, semicolons are not used at the end of an SQL query for
  Oracle.

. There's a new single-key lookup type called dsearch. It searches a directory
  for a file whose name matches the key. The result of a successful search is
  the key. One possible use of this could be for recognizing virtual domains.
  If each domain is represented by a file whose name is the domain name, you
  needn't make a separate list of the domains. You could test for them in an
  ACL (see below), for example, by a line like this

     accept domains = dsearch;/etc/virtual/domains

. The format of LDAP output has been changed for cases where multiple
  attributes are requested. The data for each attribute is now always quoted.
  Within the quotes, the quote character, backslash, and newline are escaped
  with backslashes and commas are used to separate multiple values for the
  attribute. Thus, the string in quotes takes the same form as the output when
  a single attribute is requested. If multiple entries are found, their data is
  still separated by a newline.

. There's a new expansion condition called ldapauth which exists so that the
  LDAP authentication mechanism can be used for user authentication. It is
  described under "string expansion" below.

. Exim now supports ldaps:// URLs as well as ldap:// URLs. The former do LDAP
  over TLS (i.e. encrypted) connections.

. There is now support for the "whoson" mechanism for doing "POP-before-SMTP"
  authentication. This is provided by new query-style lookup type called
  "whoson", with queries that consist of IP addresses. For example, in an ACL
  you can write

    require condition = ${lookup whoson {$sender_host_address}{yes}{no}}


Special items in domain and host lists
--------------------------------------

. In a domain list, the special item @ matches the primary host name, and the
  special item @[] matches any local interface address enclosed in square
  brackets (as in domain literal email addresses). The special item @mx_any
  matches any domain that has an MX record pointing to the local host. The
  special items @mx_primary and @mx_secondary are similar, except that the
  first matches only when the primary MX is to the local host, and the second
  only when the primary MX is not the local host, but a secondary MX is.

. In a host list, the special item @ matches the primary host name, and the
  special item @[] matches any local interface address (not in brackets).


Access Control Lists (ACLs)
---------------------------

All the policy control options for incoming messages have been replaced by
Access Control Lists (ACLs). These give more flexibility to the sysadmin, and
allow the order of testing to be specified. For example, using an ACL, it is
possible to specify "accept if authenticated, even if from an RBL host, but
otherwise deny if from an RBL host", which is not possible in Exim 3.

ACLs are defined in a new part of the configuration file, and given names.
Which ones to run are controlled by a new set of options that are placed in the
main part of the configuration.

  acl_smtp_auth   specifies the ACL to run when AUTH is received
  acl_smtp_data   specifies the ACL to run after a message has been received
  acl_smtp_etrn   specifies the ACL to run when ETRN is received
  acl_smtp_expn   specifies the ACL to run when EXPN is received
  acl_smtp_rcpt   specifies the ACL to run when RCPT is received
  acl_smtp_vrfy   specifies the ACL to run when VRFY is received

The default actions vary. If acl_smtp_auth is not defined, AUTH is always
accepted (and an attempt is made to authenticate the session). If acl_smtp_data
is not defined, no checks are done after a message has been received, and it is
always accepted at that point.

However, if any of the others are not defined, the relevant SMTP command is
rejected. In particular, this means that acl_smtp_rcpt must be defined in order
to receive any messages over an SMTP connection. The default configuration file
contains a suitable default for this.

ACLs can be provided in line, or in files, or looked up from databases. One ACL
can call another in a subroutine-like manner. String expansion is used, and
which ACL to run can be varied according to sender host or any other criterion
that a string expansion can test.

This is not the place to give a full specification of ACLs, but here is a
typical example for checking RCPT commands, taken from the default
configuration. The tests are performed in order.

acl_check_rcpt:
  # Accept if source is local SMTP (i.e. not over TCP/IP - undefined host)
  accept  hosts = :

  # Deny if the local part contains @ or % or /
  deny    local_parts   = ^.*[@%/]

  # Accept mail to postmaster in any local domain, regardless of the source,
  # and without verifying the sender.
  accept  domains       = +local_domains
          local_parts   = postmaster

  # Deny unless the sender address can be verified.
  require verify        = sender

  # Accept if the address is in a local domain, but only if the recipient can
  # be verified. Otherwise deny. The "endpass" line is the border between
  # passing on to the next ACL statement (if tests above it fail) or denying
  # access (if tests below it fail).
  accept  domains       = +local_domains
          endpass
          message       = unknown user
          verify        = recipient

  # We get here only for non-local domains. Accept if the message arrived over
  # an authenticated connection, from any host. These messages are usually from
  # MUAs, so recipient verification is omitted.
  accept  authenticated = *

  # Reaching the end of the ACL causes a "deny", but we might as well give
  # an explicit message.
  deny    message       = relay not permitted

The following options have been abolished as a consequence of the introduction
of ACLs:

auth_hosts, auth_over_tls_hosts, headers_checks_fail, headers_check_syntax,
headers_sender_verify, headers_sender_verify_errmsg, host_accept_relay,
host_auth_accept_relay, host_reject_recipients, prohibition_message,
rbl_domains, rbl_hosts, rbl_log_headers, rbl_log_rcpt_count,
rbl_reject_recipients, rbl_warn_header, receiver_try_verify, receiver_verify,
receiver_verify_addresses, receiver_verify_hosts, receiver_verify_senders,
recipients_reject_except, recipients_reject_except_senders, relay_domains,
relay_domains_include_local_mx, relay_match_host_or_sender,
sender_address_relay, sender_address_relay_hosts, sender_reject,
sender_reject_recipients, sender_try_verify, sender_verify,
sender_verify_batch, sender_verify_hosts, sender_verify_fixup,
sender_verify_hosts_callback, sender_verify_callback_domains,
sender_verify_callback_timeout, sender_verify_max_retry_rate,
sender_verify_reject, smtp_etrn_hosts, smtp_expn_hosts. smtp_verify, tls_hosts.

The variable $prohibition_reason has been abolished.

The host_reject option has been retained, but with its name changed to
host_reject_connection, to emphasize that it causes a rejection at connection
time. I've left it available just in case it is needed - but its use is not
recommended in normal circumstances.


Other Incoming SMTP Session Controls
------------------------------------

. The option smtp_accept_max_per_connection (default 1000) limits the number of
  messages accepted over a single SMTP connection. This is a safety catch in
  case some sender goes mad (incidents of this kind have been seen). After the
  limit is reached, a 421 response is given to MAIL commands.

. Some sites find it helpful to be able to limit the rate at which certain
  hosts can send them messages, and the rate at which an individual message can
  specify recipients. There are now options for controlling these two different
  rates.

  Rate limiting applies only to those hosts that match smtp_ratelimit_hosts,
  whose value is a host list. When a host matches, one or both of the options
  smtp_ratelimit_mail and smtp_ratelimit_rcpt may be set. They apply to the
  rate of acceptance of MAIL and RCPT commands in a single SMTP session,
  respectively.

  The value of each option is a set of four comma-separated values:

    1. A threshold, before which there is no rate limiting.
    2. An initial time delay. Unlike other times in Exim, fractions are allowed
       here.
    3. A factor by which to increase the delay each time.
    4. A maximum value for the delay.

  For example, these settings have been used successfully at the site which
  first suggested this feature, for controlling mail from their customers:

    smtp_ratelimit_mail = 2, 0.5s, 1.05, 4m
    smtp_ratelimit_rcpt = 4, 0.25s, 1.015, 4m

. The default value for smtp_connect_backlog has been increased to 20.

. The SMTP protocol specification requires the client to wait for a response
  from the server at certain points in the dialogue. (Without PIPELINING these
  are after every command; with PIPELINING they are fewer, but still exist.)
  Some spamming sites send out a complete set of SMTP commands without waiting
  for any response. Exim 4 protects against this by rejecting messages if the
  client has sent further input when it should not have. The error response
  "554 SMTP synchronization error" is sent, and the connection is dropped.

  This check is controlled by smtp_enforce_sync, which is true by default.

. helo_strict_syntax has been abolished. The default is now to enforce strict
  domain syntax for HELO/EHLO arguments. You can use helo_accept_junk_hosts if
  you want to avoid this.

. There's a new option called helo_lookup_domains. If the domain given in a
  HELO or EHLO command matches this list, a reverse lookup is done in order to
  establish the host's true name. The default setting is

    helo_lookup_domains = @ : @[]

  That is, a lookup is forced if the client host gives the server's name or
  [one of its IP addresses] in HELO or EHLO. (In Exim 3 this happened
  automatically and was not configurable.)

. The value of the global message_size_limit option is now expanded. For
  locally submitted messages this happens at the start of message reception.
  For messages from remote hosts, the expansion is done just after the host
  connects, so that the value can depend on the host.


Handling of Resent- Fields
--------------------------

RFC 2822 makes it clear that Resent- fields are purely informational. Exim used
to make use of Resent-Reply-To: which does not actually exist, and it also used
to use the last set of resent- fields for all the address fields it recognized.

In Exim 4, resent- headers are dealt with as follows:

. A Resent-From: header that just contains the login id as the address is
  automatically rewritten in the same way as From: is (using qualify domain,
  and user name from the passwd data).

. If there's a rewrite rule for a header, it is also applied to resent- headers
  of the same type. For example, a rule that rewrites From: headers also
  rewrites Resent-From: headers.

. For local messages, if Sender: is being removed on input, Resent-Sender: is
  also removed.

. If there are any resent- headers but no Resent-Date: or Resent-From: they are
  added.

. The logic for adding Sender: is now duplicated for Resent-Sender.

. If there's no Resent-Message-Id: one is created, and it is the
  Resent-Message-Id: which is included in the log line.


Authentication
--------------

. The auth_hosts option has been abolished; this functionality is now
  controlled by ACLs.

. The auth_always_advertise option has been abolished because it depended on
  auth_hosts and host_auth_accept_relay, both of which are no more. In its
  place there is a new option called auth_advertise_hosts, whose default value
  is *, meaning "advertise AUTH to all".

. The value of server_setid is now used when logging failed authentication
  attempts.

. The -oMaa option allows trusted users to set the value of
  $sender_host_authenticated (the authenticator name). This is normally used in
  conjunction with -oMa.


Encryption
----------

. Because tls_hosts is no more, tls_advertise_hosts is now the only means of
  controlling the advertisement of STARTTLS (previously, tls_hosts overrode).

. The global option tls_verify_ciphers has been abolished. There are now
  facilities for checking which cipher is in use in ACLs.

. There's a new option called tls_try_verify_hosts. Like tls_verify_hosts, this
  causes the server to request a certificate from a client, and it verifies the
  certificate that it receives. However, unlike tls_verify_hosts, Exim
  continues with the SMTP connection (encrypted) if a client certificate is not
  received, or if the certificate does not verify. This state can be detected
  in an ACL, which makes it possible to implement policies such as "accept for
  relay only if a verified certificate has been received but accept for local
  delivery if encrypted, even without a verified certificate".

  A match in tls_verify_hosts overrides tls_try_verify_hosts.


The Daemon
----------

. local_interfaces can now specify a port number with each address, thus
  allowing a single Exim daemon to listen on multiple ports. The format of each
  address is either [aaaa]:ppp or aaaa.ppp where aaaa is an IP address and ppp
  is a port number. For example:

    local_interfaces = 192.168.3.4.25 : 192.168.3.4.26

  If an address is listed without a port, the setting of daemon_smtp_port, or
  the value of the -oX option, is the default.

. The -oX option can now override local_interfaces. That is, it can supply IP
  addresses as well as just a port. It is interpreted in this way if its value
  contains any of the characters . : or []. For example:

  exim -bd -oX 10.9.8.7:10.11.12.13.2525

  The format of the string is identical to the format recognized by the
  local_interfaces option.

. The way the daemon wrote PID files was overly complicated and messy. It no
  longer tries to be clever. A PID file is written if, and only if, -bd is used
  and -oX is _not_ used. In other words, only if the daemon is started with its
  standard options. There is only one PID file. If pid_file_path is unset, it
  is exim-daemon.pid in Exim's spool directory. Otherwise the value of
  pid_file_path is used. For backwards compatibility, "%s" in this value is
  replaced by an empty string.


Logging
-------

The log_level option and all the various independent logging control options
have been abolished. In their place there is a single option called
log_selector. It takes a string argument composed of names preceded by + or -
characters. These turn on or off the logging of different things. For example:

  log_selector = +arguments -retry_defer

The optional logging items (defaults marked *) are:

  address_rewrite             address rewriting
  all_parents                 all parents in => lines
  arguments                   exim arguments
 *connection_reject           connection rejections
 *delay_delivery              immediate delivery delayed (message queued)
  delivery_size               add S=nnn to delivery lines
 *dnslist_defer               defers of DNS list (aka RBL) lookups
  incoming_interface          incoming interface on <= lines
  incoming_port               incoming port on <= lines
 *lost_incoming_connection    as it says (includes timeouts)
 *queue_run                   start and end queue runs
  received_sender             sender on <= lines
  received_recipients         recipients on <= lines
 *retry_defer                 "retry time not reached"
  sender_on_delivery          add sender to => lines
 *size_reject                 rejection because too big
 *skip_delivery               "message is frozen"
  smtp_confirmation           SMTP confirmation on <= lines
  smtp_connection             SMTP connections
  smtp_protocol_error         SMTP protocol errors
  smtp_syntax_error           SMTP syntax errors
  subject                     contents of Subject: on <= lines
 *tls_cipher                  TLS cipher on <= lines
  tls_peerdn                  TLS peer DN on <= lines

  all                         all of the above

"retry time not reached" is always omitted from individual message logs after
the first delivery attempt.

The log line "error message sent to" has been abolished, because the R= item on
the incoming message line gives the relationship between the original message
and the bounce.

The logging options that have been abolished are: log_all_parents,
log_arguments, log_incoming_port, log_interface, log_ip_options,
log_level, log_queue_run_level, log_received_sender, log_received_rceipients,
log_rewrites, log_sender_on_delivery, log_smtp_confirmation,
log_smtp_connections, log_smtp_syntax_errors, log_subject, tls_log_cipher,
tls_log_peerdn.


Debugging
---------

The debug_level option has been removed. The -dm option has been removed. The
-df option has also be removed, along with its related build-time option
STDERR_FILE. (To debug inetd usage, an auxiliary script should be used.)

The -d option has been reworked. It no longer takes a debug level number
argument, but instead takes a list of debugging names, each preceded by + or -
to turn on or off individual sets of debugging messages.

. The -v option now shows just the SMTP dialog and any log lines.

. -d with no argument gives a lot of standard debugging data. This is in effect
  the equivalent of the old -d9, the thing you ask people to set for an initial
  debugging test.

. -d+x      adds debugging option x to the default set
  -d-x      removes debugging option x from the default set
  -d-all+x  leaves only debugging option x

The available debugging names are:

  acl            ACL interpretation
  auth           authenticators
  deliver        general delivery logic
  dns            DNS lookups (see also resolver)
  dnsbl          DNS black list (aka RBL) code
  exec           arguments for execv() calls
  filter         filter handling
  hints_lookup   hints data lookups
  host_lookup    all types of name->IP address handling
  ident          ident lookup
  interface      lists of local interfaces
  lists          matching things in lists
  load           system load checks
  lookup         general lookup code and all lookups
  memory         memory handling (replaces the old -dm)
  process_info   setting info for the process log
  queue_run      queue runs
  receive        general message reception logic
  resolver       turn on the DNS resolver's debugging output; goes to stdout
  retry          retry handling
  rewrite        rewriting
  route          address routing
  tls            TLS logic
  transport      transports
  uid            changes of uid/gid and looking up uid/gid
  verify         address verification logic

  all            all of the above, and also -v

The default (-d with no argument) includes all of the above, plus -v, with the
exception of filter, interface, load, memory, and resolver. Some debugging
output always happens unconditionally whenever any debugging is selected. This
includes some initial output and every log line.

-d without any value was previously allowed for non-admin users because it used
to be synonymous with -v. In Exim 4, non-admin users may use -v, but not -d.

If the debug_print option is set in any driver, it produces output whenever any
debugging is selected, or if -v is used.


Local Scan Function
-------------------

For customized message scanning, you can now supply a C function that is linked
into the Exim binary. The function is called local_scan(), and it is called
when Exim has received a message, but has not yet sent a final
acknowledgement to the sender. This applies to all messages, whether local or
remote, SMTP or not.

From within your function you can inspect the message, change the recipients,
add or remove headers, and tell Exim whether to accept or reject the message.

The manual contains the specification of the API for this function.


String Expansion
----------------

. The lookup feature that allowed for subkeys using the syntax

    ${lookup {key:subkey} type {data...

  has been abolished (a) because the effect can be achieved using ${extract,
  and (b) because in non-lsearch lookups, a colon can be a valid character in a
  key.

. When a string key is used in a ${extract expansion item, it is now handled
  case-insensitively.

. A new expansion variable called $tod_epoch gives the time as a single decimal
  number representing the number of seconds from the start of the Unix epoch.

. There's a new expansion operator that can turn numbers into base 62, for
  example, ${base62:$tod_epoch}.

. ${extract{number} now recognizes a negative number as a request to count
  fields from the right.

. There's a new expansion feature for reading files:

    ${readfile{/some/file}{eolstring}}

  The contents of the file replace the item. If {eolstring} is present (it's
  optional) any newlines in the file are replaced by that string.

. There's a new expansion feature for running commands:

    ${run{comand args}{yes}{no}}

  Like all the other conditional items, the {yes} and {no} strings are
  optional. Omitting both is equivalent to {$value}. The standard output of the
  command is put into $value if the command succeeds (returns a zero code). The
  value of the code itself is put into $runrc, and this remains set afterwards,
  so in a filter file you can do things like

    if "${run{x y z}{}}$runrc" is 1 then ...
    elsif $runrc is 2 then ...

  As in other command executions from Exim, a shell is not used by default.
  If you want a shell, you must explicitly code it.

. The redirect router has options for forbidding ${readfile and ${run in
  filters.

. A feature is provided to suppress expansion of part of a string. Any
  characters between two occurrences of \N are copied to the output string
  verbatim. This is particularly useful for protecting regular expressions from
  unwanted expansion effects. For example:

    queue_smtp_domains = ! \N^ten-\d+\.testing\.com$\N

  Without \N the \ and $ characters in the regex would have to be escaped.

. Radius authentication is supported in a similar way to PAM. You must set
  RADIUS_CONFIG_FILE in Local/Makefile to specify the location of the Radius
  client configuration file. Then you can use expansions such as

    server_condition = ${if radius{arguments}{yes}{no}}

. User authentication can now also be done by attempting to bind to an LDAP
  server. The syntax is again similar to PAM and Radius.

    server_condition = ${if ldapauth{ldap query}{yes}{no}}

  A user and password are required to be supplied with the query. No actual
  data is looked up; Exim just does a bind to the LDAP server and sets the
  condition according to the result. Here's an example of an SMTP
  authenticator:

    login:
      driver = plaintext
      public_name = LOGIN
      server_prompts = "Username:: : Password::"
      server_condition = ${if ldapauth \
        {user="uid=${quote_ldap:$1},ou=people,o=example.org" pass="$2" \
        ldap://ldap.example.org/}{yes}{no}}
      server_set_id = uid=$1,ou=people,o=example.org



Security
--------

Exim 3 could be run in a variety of ways as far as security was concerned. This
has all been simplified in Exim 4. The security-conscious might like to know
that it no longer makes any use of the seteuid() function.

. A UID and GID are required to be specified when Exim is compiled. They can be
  now specified by name as well as by number, so the relevant options are now
  called EXIM_USER and EXIM_GROUP. If you really feel you have to run Exim as
  root, you can specify root here, but it is not recommended.

. The "security" option has been abolished. Exim always releases its root
  privilege when it can. In a conventional configuration, that means when it is
  receiving a message, when it is delivering a message, when it is running a
  queryprogram router, and when it is obeying users' filter files (and system
  filters if it has been given a user for that purpose).

. One important change is that Exim 4 runs as root while routing addresses for
  delivery. Exim 3 used seteuid() to give up privilege temporarily while
  routing. Apart from the unliked use of seteuid(), this sometimes gave rise to
  permissions problems on configuration files.

. However, Exim still runs as the Exim user while receiving messages, and
  therefore while using the routing logic for verifying at SMTP time.

. There is a new option called deliver_drop_privilege. If this is set, Exim
  gives up its privilege right at the start of a delivery process, and runs the
  entire delivery as the Exim user. This is the same action that used to be
  requested by setting security=unprivileged.


Hints Databases
---------------

. A single "misc" hints database is now used for ETRN and host serialization.
  There have been appropriate consequential changes to the utilities for
  managing the hints.

. The exim_tidydb -f option has been abolished. A full tidy is now always done
  (it hasn't proved to be very expensive).


The run time Configuration File
------------------------------

. The format of the configuration file has changed. Instead of using "end" to
  terminate sections, it now uses "begin <name>" to start sections. This means
  that the sections, apart from the first, may appear in any order.

. You can now include other files inside Exim run time configuration files, by
  using this syntax:

  .include <file name>

. Quotes round the file name are optional. Includes may be nested to any depth,
  but remember that Exim reads its configuration file often. The processing of
  .include happens early, at a physical line level, so, like comment lines, it
  can be used in the middle of an options setting, for example:

  hosts_lookup = a.b.c \
                 .include /some/file

  Include processing happens _before_ macro processing. Its effect is simply to
  process the lines of the file as if they occurred inline where the .include
  appears.

. A macro at the start of a configuration line can now turn the line into an
  empty line or a comment line. This applies to _logical_ input lines, that is,
  after any concatenations have been done.


Format of spool files
---------------------

. -local_scan is used in spool files to record the value of $local_scan_data,
  the string returned from the locally-provided local_scan() function.


Renamed Options
---------------

Some options have been renamed, to make their function clearer, or for
consistency.

. receiver_unqualified_hosts has been renamed as recipient_unqualified_hosts.
  I'm going to use "recipient" everywhere in future.

. helo_verify has become helo_verify_hosts.

. remote_sort has become remote_sort_domains.

. In the appendfile and pipe transports, "prefix" and "suffix" have become
  "message_prefix" and "message_suffix". In the generic router options,
  "prefix" and "suffix" have become "local_part_prefix" and "local_part_suffix".


Miscellaneous
-------------

. ETRN serialization now uses a double fork, so that an Exim process (detached
  from the original input process) can wait for the command to finish. This
  means that it works whatever command ETRN causes to run. (Previously it
  worked only if ETRN ran "exim -Rxxx".)

. For incoming messages, the server's port number is preserved, and is
  available in $interface_port. The privileged option -oMi can be used to
  set this value.

. The -Mmd option (to mark addresses delivered) now operates in a
  case-sensitive manner.

. Checks for duplicate deliveries are now case-sensitive in the local part.

. The number of situations where Exim panics has been reduced. For example,
  expansion failures for the "domains" or "local_parts" options in a router now
  cause deferral instead of a panic.

. EXPN no longer attempts to distinguish local and remote addresses (but you
  can cause it to be rejected for certain arguments in the ACL).

. accept_timeout has been renamed as receive_timeout, to match
  smtp_receive_timeout.

. The ability to check an ident value as part of an item in a host list has
  been removed.

. The reject log shows a message's headers only if the rejection happens after
  the SMTP DATA command (because they aren't available for earlier checks). The
  sender, and up to five recipients are listed in Envelope-from: and
  Envelope-to: header lines. After the headers, a line of separator characters
  is output. Separators are no longer used for other reject log entries.

. Because header checks are now done as part of ACLs, they now apply only to
  SMTP input.

. The port number on SMTP connections is now logged in the format [aaaa]:ppp
  where aaaa is an IP address and ppp is a port, instead of in the format
  [aaaa.ppp] because the former format causes some software to complain about
  bad IP addresses.

. The -oMa and -oMi options can now use the [aaaa]:ppp notation to set a port
  number, but they still also recognize the aaaa.ppp notation.

. The build-time option HAVE_AUTH is abolished. Exim automatically includes
  authentication code if any authenticators are configured.

. The nobody_user and nobody_group options have been abolished.

. The $message_precedence variable has been abolished. The value is now
  available as $h_precedence:.

. There's a new utility script called exim_checkaccess which packages up a call
  to Exim with the -bh option, for access control checking. The syntax is

    exim_checkaccess <IP address> <email address> [exim options]

  It runs "exim -bh <IP address>", does the SMTP dialogue, tests the result and
  outputs either "accepted" or "Rejected" and the SMTP response to the RCPT TO
  command. The sender is <> by default, but can be changed by the use of the
  -f option.

. The default state of Exim is now to forbid domain literals. For this reason,
  the option that changes this has been renamed as allow_domain_literals.

. The dns_check_names boolean option has been abolished. Checking is now turned
  off by unsetting dns_check_names_pattern.

. The errors_address and freeze_tell_mailmaster options have been abolished. In
  their place there is a new option called freeze_tell, which can be set to a
  list of addresses. A message is sent to these addresses whenever a message is
  frozen - with the exception of failed bounce messages (this is not changed).

. The message_size_limit_count_recipients option has been abolished on the
  grounds that it was a failed experiment.

. The very-special-purpose X rewrite flag has been abolished. The facility it
  provided can now be done using the features of ACLs.

. The timestamps_utc option has been abolished. The facility is now provided by
  setting timezone = utc.

. The value of remote_max_parallel now defaults to 2.

. ignore_errmsg_errors has been abolished. The effect can be achieved by
  setting ignore_bounce_errors_after = 0s. This option has been renamed from
  ignore_errmsg_errors_after to make its function clearer. The default value
  for ignore_bounce_errors_after is now 10w (10 weeks - i.e. likely to be
  longer than any other timeouts, thereby disabling the facility).

. The default for message_size_limit is now 50M as a guard against DoS attacks.

. The -qi option does only initial (first time) deliveries. This can be helpful
  if you are injecting message onto the queue using -odq and want a queue
  runner just to process new messages. You can also use -qqi if you want.

. Rewriting and retry patterns are now anything that can be single address list
  items. They are processed by the same code, and are therefore expanded before
  the matching takes place. Regular expressions must be suitably quoted. These
  patterns may now be enclosed in double quotes so that white space may be
  included. Normal quote processing applies.

. Some scripts were built in the util directory, which was a mistake, because
  they might be different for different platforms. Everything that is built is
  now built in the build directory. The util directory just contains a couple
  of scripts that are not modified at build time.

. The installation script now installs the Exim binary as exim-v.vv-bb (where
  v.vv is the version number and bb is the build number), and points a symbolic
  link called "exim" to this binary. It does this in an atomic way so that
  there is no time when "exim" is non-existent. The script is clever enough to
  cope with an existing non-symbolic-link binary, converting it to the new
  scheme automatically (and atomically).

. When installing utilities, Exim now uses cp instead of mv to add .O to the
  old ones, in order to preserve the permissions.

. If the installation script is installing the default configuration, and
  /etc/aliases does not exist, the script installs a default version. This does
  not actually contain any aliases, but it does contain comments about ones
  that should be created. A warning is output to the user.

. A delay warning message is not sent if all the addresses in a message get a
  "retry time not reached" error. Exim waits until a delivery is actually
  attempted, so as to be able to give a more informative message.

. The existence of the three options deliver_load_max, queue_only_load, and
  deliver_queue_load_max was confusing, because their function overlapped. The
  first of them has been abolished. We are left with

    queue_only_load         no immediate delivery if load is high when
                            message arrives
    deliver_queue_load_max  no queued delivery if load is too high

. The ability to edit message bodies (-Meb and the Eximon menu item) has been
  removed, on the grounds that it is bad practice to do this.

. Eximstats is now Steve Campbell's patched version, which displays sizes in K
  and M and G, and can optionally generate HTML.

. If bounce_sender_authentication is set to an email address, this address is
  used in an AUTH option of the MAIL command when sending bounce messages, if
  authentication is being used. For example, if you set

    bounce_sender_authentication = mailer-daemon@your.domain

  a bounce message will be sent over an authenticated connection using

    MAIL FROM:<> AUTH=mailer-daemon@your.domain

. untrusted_set_sender has changed from a boolean to an address pattern. It
  permits untrusted users to set sender addresses that match the pattern. Like
  all address patterns, it is expanded. The identity of the user is in
  $sender_ident, so you can, for example, restrict users to setting senders
  that start with their login ids by setting

    untrusted_set_sender = ^$sender_ident-

  The effect of the previous boolean can be achieved by setting the value to *.
  This option applies to all forms of local input.

. The always_bcc option has been abolished. If an incoming message has no To:
  or Cc: headers, Exim now always adds an empty Bcc: line. This makes the
  message valid for RFC 822 (sic). In time, this can be removed, because RFC
  2822 does not require there to be a recipient header.

. ACTION_OUTPUT=no is now the default in the Exim monitor.

. dns_ipv4_lookup has changed from a boolean into a domain list, and it now
  applies only to those domains. Setting this option does not stop Exim from
  making IPv6 calls: if an MX lookup returns AAAA records, Exim will use them.
  What it does is to stop Exim looking for AAAA records explicitly.

. The -G option is ignored (another Sendmail thing).

. If no_bounce_return_message is set, the original message is not included in
  bounce messages. If you want to include additional information in the bounce
  message itself, you can use the existing errmsg_file and errmsg_text
  facilities.

. -bdf runs the daemon in the foreground (i.e. not detached from the terminal),
  even when no debugging is requested.

. Options for changing Exim's behaviour on receiving IPv4 options have been
  abolished. Exim now always refuses calls that set these options, and logs the
  incident. The abolished options are kill_ip_options, log_ip_options, and
  refuse_ip_options.

. The pattern for each errors_copy entry is now matched as an item in an
  address list.

. A number of options and variables that used the word "errmsg" have been
  changed to use "bounce" instead, because it seems that "bounce message" is
  now a reasonably well-understood term. I used it in the book and am now using
  it in the manual; it's a lot less cumbersome than "delivery error
  notification message". The changes are:

  $errmsg_recipient          => $bounce_recipient
  errmsg_file                => bounce_message_file
  errmsg_text                => bounce_message_text
  ignore_errmsg_errors_after => ignore_bounce_errors_after

  For consistency, warnmsg_file has been changed to warn_message_file. However,
  the two variables $warnmsg_delay and $warnmsg_recipients are unchanged.

  The hide_child_in_errmsg option has not changed, because it applies to both
  bounce and delay warning messages.

. smtp_accept_max_per_host is now an expanded string, so it can be varied on
  a per-host basis. However, because this test happens in the daemon before it
  forks, the expansion should be kept as simple as possible (e.g. just inline
  tests of $sender_host_address).

. The retry rules can now recognize the error "auth_failed", which happens when
  authentication is required, but cannot be done.

. There's a new option called local_sender_retain which can be set if
  no_local_from_check is set. It causes Sender: headers to be retained in
  locally-submitted messages.

. The -dropcr command line option now turns CRLF into LF, and leaves isolated
  CRs alone. Previously it simply dropped _all_ CR characters. There is now
  also a drop_cr main option which, if turned on, assumes -dropcr for all
  non-SMTP input.


Removal of Obsolete Things
--------------------------

. The obsolete values "fail_soft" and "fail_hard" for the "self" option have
  been removed.

. The obsolete "log" command has been removed from the filter language.

. "service" was an obsolete synonym for "port" when specifying IP port numbers.
  It has been removed.

. The obsolete option collapse_source_routes has been removed. It has done
  nothing since release 3.10.

. The obsolete from_hack option in appendfile and pipe transports has been
  removed.

. The obsolete ipv4_address_lookup has been abolished (dns_ipv4_lookup has been
  a synonym for some time, but it's changed - see above).

. The obsolete generic transport options add_headers and remove_headers have
  been abolished. The new names, headers_add and headers_remove, have been
  available for some time.

Philip Hazel
February 2002