Exigrep patch to allow it to handle syslog files.

[exim.git] / doc / doc-txt / NewStuff

$Cambridge: exim/doc/doc-txt/NewStuff,v 1.57 2005/08/01 13:20:28 ph10 Exp $

New Features in Exim
--------------------

This file contains descriptions of new features that have been added to Exim,
but have not yet made it into the main manual (which is most conveniently
updated when there is a relatively large batch of changes). The doc/ChangeLog
file contains a listing of all changes, including bug fixes.

Exim version 4.53
-----------------

TK/01 Added the "success_on_redirect" address verification option. When an
      address generates new addresses during routing, Exim will abort
      verification with "success" when more than one address has been
      generated, but continue to verify a single new address. The latter
      does not happen when the new "success_on_redirect" option is set, like

      require verify = recipient/success_on_redirect/callout=10s

      In that case, verification will succeed when a router generates a new
      address.

PH/01 Support for SQLite database lookups has been added. This is another
      query-style lookup, but it is slightly different from the others because
      a file name is required in addition to the SQL query. This is because an
      SQLite database is a single file and there is no daemon as in other SQL
      databases. The interface to Exim requires the name of the file, as an
      absolute path, to be given at the start of the query. It is separated
      from the query by white space. This means that the path name cannot
      contain white space. Here is a lookup expansion example:

        ${lookup sqlite {/some/thing/sqlitedb \
          select name from aliases where id='ph10';}}

      In a list, the syntax is similar. For example:

        domainlist relay_domains = sqlite;/some/thing/sqlitedb \
           select * from relays where ip='$sender_host_address';

      The only character affected by the ${quote_sqlite: operator is a single
      quote, which it doubles.

      Note that you must set LOOKUP_SQLITE=yes in Local/Makefile in order to
      obtain SQLite support, and you will also need to add -lsqlite3 to the
      EXTRALIBS setting. And of course, you have to install SQLite on your
      host first.


Exim version 4.52
-----------------

TF/01 Support for checking Client SMTP Authorization has been added. CSA is a
      system which allows a site to advertise which machines are and are not
      permitted to send email. This is done by placing special SRV records in
      the DNS, which are looked up using the client's HELO domain. At this
      time CSA is still an Internet-Draft.

      Client SMTP Authorization checks are performed by the ACL condition
      verify=csa. This will fail if the client is not authorized. If there is
      a DNS problem, or if no valid CSA SRV record is found, or if the client
      is authorized, the condition succeeds. These three cases can be
      distinguished using the expansion variable $csa_status, which can take
      one of the values "fail", "defer", "unknown", or "ok". The condition
      does not itself defer because that would be likely to cause problems
      for legitimate email.

      The error messages produced by the CSA code include slightly more
      detail. If $csa_status is "defer" this may be because of problems
      looking up the CSA SRV record, or problems looking up the CSA target
      address record. There are four reasons for $csa_status being "fail":
      the client's host name is explicitly not authorized; the client's IP
      address does not match any of the CSA target IP addresses; the client's
      host name is authorized but it has no valid target IP addresses (e.g.
      the target's addresses are IPv6 and the client is using IPv4); or the
      client's host name has no CSA SRV record but a parent domain has
      asserted that all subdomains must be explicitly authorized.

      The verify=csa condition can take an argument which is the domain to
      use for the DNS query. The default is verify=csa/$sender_helo_name.

      This implementation includes an extension to CSA. If the query domain
      is an address literal such as [192.0.2.95], or if it is a bare IP
      address, Exim will search for CSA SRV records in the reverse DNS as if
      the HELO domain was e.g. 95.2.0.192.in-addr.arpa. Therefore it is
      meaningful to say, for example, verify=csa/$sender_host_address - in
      fact, this is the check that Exim performs if the client does not say
      HELO. This extension can be turned off by setting the main
      configuration option dns_csa_use_reverse = false.

      If a CSA SRV record is not found for the domain itself, then a search
      is performed through its parent domains for a record which might be
      making assertions about subdomains. The maximum depth of this search is
      limited using the main configuration option dns_csa_search_limit, which
      takes the value 5 by default. Exim does not look for CSA SRV records in
      a top level domain, so the default settings handle HELO domains as long
      as seven (hostname.five.four.three.two.one.com) which encompasses the
      vast majority of legitimate HELO domains.

      The dnsdb lookup also has support for CSA. Although dnsdb already
      supports SRV lookups, this is not sufficient because of the extra
      parent domain search behaviour of CSA, and (as with PTR lookups)
      dnsdb also turns IP addresses into lookups in the reverse DNS space.
      The result of ${lookup dnsdb {csa=$sender_helo_name} } has two
      space-separated fields: an authorization code and a target host name.
      The authorization code can be "Y" for yes, "N" for no, "X" for explicit
      authorization required but absent, or "?" for unknown.

PH/01 The amount of output produced by the "make" process has been reduced,
      because the compile lines are often rather long, making it all pretty
      unreadable. The new style is along the lines of the 2.6 Linux kernel:
      just a short line for each module that is being compiled or linked.
      However, it is still possible to get the full output, by calling "make"
      like this:

        FULLECHO='' make -e

      The value of FULLECHO defaults to "@", the flag character that suppresses
      command reflection in "make". When you ask for the full output, it is
      given in addition to the the short output.

TF/02 There have been two changes concerned with submission mode:

      Until now submission mode always left the return path alone, whereas
      locally-submitted messages from untrusted users have the return path
      fixed to the user's email address. Submission mode now fixes the return
      path to the same address as is used to create the Sender: header. If
      /sender_retain is specified then both the Sender: header and the return
      path are left alone.

      Note that the changes caused by submission mode take effect after the
      predata ACL. This means that any sender checks performed before the
      fix-ups will use the untrusted sender address specified by the user, not
      the trusted sender address specified by submission mode. Although this
      might be slightly unexpected, it does mean that you can configure ACL
      checks to spot that a user is trying to spoof another's address, for
      example.

      There is also a new /name= option for submission mode which allows you
      to specify the user's full name to be included in the Sender: header.
      For example:

        accept authenticated = *
               control = submission/name=${lookup {$authenticated_id} \
                                           lsearch {/etc/exim/namelist} }

      The namelist file contains entries like

        fanf: Tony Finch

      And the resulting Sender: header looks like

        Sender: Tony Finch <fanf@exim.org>

TF/03 The control = fakereject ACL modifier now has a fakedefer counterpart,
      which works in exactly the same way except it causes a fake SMTP 450
      response after the message data instead of a fake SMTP 550 response.
      You must take care when using fakedefer because it will cause messages
      to be duplicated when the sender retries. Therefore you should not use
      fakedefer if the message will be delivered normally.

TF/04 There is a new ratelimit ACL condition which can be used to measure
      and control the rate at which clients can send email. This is more
      powerful than the existing smtp_ratelimit_* options, because those
      options only control the rate of commands in a single SMTP session,
      whereas the new ratelimit condition works across all connections
      (concurrent and sequential) to the same host.

      The syntax of the ratelimit condition is:

        ratelimit = <m> / <p> / <options> / <key>

      If the average client sending rate is less than m messages per time
      period p then the condition is false, otherwise it is true.

      The parameter p is the smoothing time constant, in the form of an Exim
      time interval e.g. 8h for eight hours. A larger time constant means it
      takes Exim longer to forget a client's past behaviour. The parameter m is
      the maximum number of messages that a client can send in a fast burst. By
      increasing both m and p but keeping m/p constant, you can allow a client
      to send more messages in a burst without changing its overall sending
      rate limit. Conversely, if m and p are both small then messages must be
      sent at an even rate.

      The key is used to look up the data used to calcluate the client's
      average sending rate. This data is stored in a database maintained by
      Exim in its spool directory alongside the retry database etc. For
      example, you can limit the sending rate of each authenticated user,
      independent of the computer they are sending from, by setting the key
      to $authenticated_id. The default key is $sender_host_address.
      Internally, Exim includes the smoothing constant p and the options in
      the lookup key because they alter the meaning of the stored data.
      This is not true for the limit m, so you can alter the configured
      maximum rate and Exim will still remember clients' past behaviour,
      but if you alter the other ratelimit parameters Exim will effectively
      forget their past behaviour.

      Each ratelimit condition can have up to two options. The first option
      specifies what Exim measures the rate of, and the second specifies how
      Exim handles excessively fast clients.

      The per_mail option means that it measures the client's rate of sending
      messages. This is the default if none of the per_* options is specified.

      The per_conn option means that it measures the client's connection rate.

      The per_byte option limits the sender's email bandwidth. Note that it
      is best to use this option in the DATA ACL; if it is used in an earlier
      ACL it relies on the SIZE parameter on the MAIL command, which may be
      inaccurate or completely missing. You can follow the limit m in the
      configuration with K, M, or G to specify limits in kilobytes,
      megabytes, or gigabytes respectively.

      The per_cmd option means that Exim recomputes the rate every time the
      condition is processed, which can be used to limit the SMTP command rate.
      The alias per_rcpt is provided for use in the RCPT ACL instead of per_cmd
      to make it clear that the effect is to limit the rate at which recipients
      are accepted. Note that in this case the rate limiting engine will see a
      message with many recipients as a large high-speed burst.

      If a client's average rate is greater than the maximum, the rate
      limiting engine can react in two possible ways, depending on the
      presence of the strict or leaky options. This is independent of the
      other counter-measures (e.g. rejecting the message) that may be
      specified by the rest of the ACL. The default mode is leaky, which
      avoids a sender's over-aggressive retry rate preventing it from getting
      any email through.

      The strict option means that the client's recorded rate is always
      updated. The effect of this is that Exim measures the client's average
      rate of attempts to send email, which can be much higher than the
      maximum. If the client is over the limit it will be subjected to
      counter-measures until it slows down below the maximum rate.

      The leaky option means that the client's recorded rate is not updated
      if it is above the limit. The effect of this is that Exim measures the
      client's average rate of successfully sent email, which cannot be
      greater than the maximum. If the client is over the limit it will
      suffer some counter-measures, but it will still be able to send email
      at the configured maximum rate, whatever the rate of its attempts.

      As a side-effect, the ratelimit condition will set the expansion
      variables $sender_rate containing the client's computed rate,
      $sender_rate_limit containing the configured value of m, and
      $sender_rate_period containing the configured value of p.

      Exim's other ACL facilities are used to define what counter-measures
      are taken when the rate limit is exceeded. This might be anything from
      logging a warning (e.g. while measuring existing sending rates in order
      to define our policy), through time delays to slow down fast senders,
      up to rejecting the message. For example,

        # Log all senders' rates
        warn
          ratelimit = 0 / 1h / strict
          log_message = \
            Sender rate $sender_rate > $sender_rate_limit / $sender_rate_period

        # Slow down fast senders
        warn
          ratelimit = 100 / 1h / per_rcpt / strict
          delay     = ${eval: 10 * ($sender_rate - $sender_rate_limit) }

        # Keep authenticated users under control
        deny
          ratelimit = 100 / 1d / strict / $authenticated_id

        # System-wide rate limit
        defer
          message = Sorry, too busy. Try again later.
          ratelimit = 10 / 1s / $primary_hostname

        # Restrict incoming rate from each host, with a default rate limit
        # set using a macro and special cases looked up in a table.
        defer
          message = Sender rate $sender_rate exceeds \
                    $sender_rate_limit messages per $sender_rate_period
          ratelimit = ${lookup {$sender_host_address} \
                        cdb {DB/ratelimits.cdb} \
                        {$value} {RATELIMIT} }

      Warning: if you have a busy server with a lot of ratelimit tests,
      especially with the per_rcpt option, you may suffer from a performance
      bottleneck caused by locking on the ratelimit hints database. Apart from
      making your ACLs less complicated, you can reduce the problem by using a
      RAM disk for Exim's hints directory, /var/spool/exim/db/. However this
      means that Exim will lose its hints data after a reboot (including retry
      hints, the callout cache, and ratelimit data).

TK/01 Added an 'spf' lookup type that will return an SPF result for a given
      email address (the key) and an IP address (the database):

      ${lookup {tom@duncanthrax.net} spf{217.115.139.137}}

      The lookup will return the same result strings as they can appear in
      $spf_result (pass,fail,softfail,neutral,none,err_perm,err_temp). The
      lookup is armored in EXPERIMENTAL_SPF. Currently, only IPv4 addresses
      are supported.

      Patch submitted by Chris Webb <chris@arachsys.com>.

PH/02 There's a new verify callout option, "fullpostmaster", which first acts
      as "postmaster" and checks the recipient <postmaster@domain>. If that
      fails, it tries just <postmaster>, without a domain, in accordance with
      the specification in RFC 2821.

PH/03 The action of the auto_thaw option has been changed. It no longer applies
      to frozen bounce messages.

TK/02 There are two new expansion items to help with the implementation of
      the BATV "prvs" scheme in an Exim configuration:


      ${prvs {<ADDRESS>}{<KEY>}{[KEYNUM]}}

      The "prvs" expansion item takes three arguments: A qualified RFC2821
      email address, a key and an (optional) key number. All arguments are
      expanded before being used, so it is easily possible to lookup a key
      and key number using the address as the lookup key. The key number is
      optional and defaults to "0". The item will expand to a "prvs"-signed
      email address, to be typically used with the "return_path" option on
      a smtp transport. The decision if BATV should be used with a given
      sender/recipient pair should be done on router level, to avoid having
      to set "max_rcpt = 1" on the transport.


      ${prvscheck {<ADDRESS>}{<SECRET>}{<RETURN_STRING>}}

      The "prvscheck" expansion item takes three arguments. Argument 1 is
      expanded first. When the expansion does not yield a SYNTACTICALLY
      valid "prvs"-scheme address, the whole "prvscheck" item expands to
      the empty string. If <ADDRESS> is a "prvs"-encoded address after
      expansion, two expansion variables are set up:

        $prvscheck_address   Contains the "prvs"-decoded version of
                             the address from argument 1.

        $prvscheck_keynum    Contains the key number extracted from
                             the "prvs"-address in argument 1.

      These two variables can be used in the expansion code of argument 2
      to retrieve the <SECRET>. The VALIDITY of the "prvs"-signed address
      is then checked. The result is stored in yet another expansion
      variable:

        $prvscheck_result    Contains the result of a "prvscheck"
                             expansion: Unset (the empty string) for
                             failure, "1" for success.

      The "prvscheck" expansion expands to the empty string if <ADDRESS>
      is not a SYNTACTICALLY valid "prvs"-scheme address. Otherwise,
      argument 3 defines what "prvscheck" expands to: If argument 3
      is the empty string, "prvscheck" expands to the decoded version
      of the address (no matter if it is CRYPTOGRAPHICALLY valid or not).
      If argument 3 expands to a non-empty string, "prvscheck" expands
      to that string.


      Usage example
      -------------

      Macro:

      PRVSCHECK_SQL = ${lookup mysql{SELECT secret FROM batv_prvs WHERE \
                      sender='${quote_mysql:$prvscheck_address}'}{$value}}

      RCPT ACL:

      # Bounces: drop unsigned addresses for BATV senders
      deny message = This address does not send an unsigned reverse path.
           senders = :
           recipients = +batv_recipients

      # Bounces: In case of prvs-signed address, check signature.
      deny message = Invalid reverse path signature.
           senders = :
           condition = ${prvscheck {$local_part@$domain}{PRVSCHECK_SQL}{1}}
           !condition = $prvscheck_result

      Top-Level Router:

      batv_redirect:
        driver = redirect
        data = ${prvscheck {$local_part@$domain}{PRVSCHECK_SQL}{}}

      Transport (referenced by router that makes decision if
      BATV is applicable):

        external_smtp_batv:
          driver = smtp
          return_path = ${prvs {$return_path} \
                               {${lookup mysql{SELECT \
                               secret FROM batv_prvs WHERE \
                               sender='${quote_mysql:$sender_address}'} \
                           {$value}fail}}}

PH/04 There are two new options that control the retrying done by the daemon
      at startup when it cannot immediately bind a socket (typically because
      the socket is already in use). The default values reproduce what were
      built-in constants previously: daemon_startup_retries defines the number
      of retries after the first failure (default 9); daemon_startup_sleep
      defines the length of time to wait between retries (default 30s).

PH/05 There is now a new ${if condition called "match_ip". It is similar to
      match_domain, etc. It must be followed by two argument strings. The first
      (after expansion) must be an IP address or an empty string. The second
      (after expansion) is a restricted host list that can match only an IP
      address, not a host name. For example:

        ${if match_ip{$sender_host_address}{1.2.3.4:5.6.7.8}{...}{...}}

      The specific types of host list item that are permitted in the list are
      shown below. Consult the manual section on host lists for further
      details.

      . An IP address, optionally with a CIDR mask.

      . A single asterisk matches any IP address.

      . An empty item matches only if the IP address is empty. This could be
        useful for testing for a locally submitted message or one from specific
        hosts in a single test such as

          ${if match_ip{$sender_host_address}{:4.3.2.1:...}{...}{...}}

        where the first item in the list is the empty string.

      . The item @[] matches any of the local host's interface addresses.

      . Lookups are assumed to be "net-" style lookups, even if "net-" is not
        specified. Thus, the following are equivalent:

          ${if match_ip{$sender_host_address}{lsearch;/some/file}...
          ${if match_ip{$sender_host_address}{net-lsearch;/some/file}...

        You do need to specify the "net-" prefix if you want to specify a
        specific address mask, for example, by using "net24-".

PH/06 The "+all" debug selector used to set the flags for all possible output;
      it is something that people tend to use semi-automatically when
      generating debug output for me or for the list. However, by including
      "+memory", an awful lot of output that is very rarely of interest was
      generated. I have changed this so that "+all" no longer includes
      "+memory". However, "-all" still turns everything off.


Version 4.51
------------

PH/01 The format in which GnuTLS parameters are written to the gnutls-param
      file in the spool directory has been changed. This change has been made
      to alleviate problems that some people had with the generation of the
      parameters by Exim when /dev/random was exhausted. In this situation,
      Exim would hang until /dev/random acquired some more entropy.

      The new code exports and imports the DH and RSA parameters in PEM
      format. This means that the parameters can be generated externally using
      the certtool command that is part of GnuTLS.

      To replace the parameters with new ones, instead of deleting the file
      and letting Exim re-create it, you can generate new parameters using
      certtool and, when this has been done, replace Exim's cache file by
      renaming. The relevant commands are something like this:

        # rm -f new.params
        # touch new.params
        # chown exim:exim new.params
        # chmod 0400 new.params
        # certtool --generate-privkey --bits 512 >new.params
        # echo "" >>new.params
        # certtool --generate-dh-params --bits 1024 >> new.params
        # mv new.params params

      If Exim never has to generate the parameters itself, the possibility of
      stalling is removed.

PH/02 A new expansion item for dynamically loading and calling a locally-
      written C function is now provided, if Exim is compiled with

        EXPAND_DLFUNC=yes

      set in Local/Makefile. The facility is not included by default (a
      suitable error is given if you try to use it when it is not there.)

      If you enable EXPAND_DLFUNC, you should also be aware of the new redirect
      router option forbid_filter_dlfunc. If you have unprivileged users on
      your system who are permitted to create filter files, you might want to
      set forbid_filter_dlfunc=true in the appropriate router, to stop them
      using ${dlfunc to run code within Exim.

      You load and call an external function like this:

        ${dlfunc{/some/file}{function}{arg1}{arg2}...}

      Once loaded, Exim remembers the dynamically loaded object so that it
      doesn't reload the same object file in the same Exim process (but of
      course Exim does start new processes frequently).

      There may be from zero to eight arguments to the function. When compiling
      a local function that is to be called in this way, local_scan.h should be
      included. The Exim variables and functions that are defined by that API
      are also available for dynamically loaded functions. The function itself
      must have the following type:

        int dlfunction(uschar **yield, int argc, uschar *argv[])

      Where "uschar" is a typedef for "unsigned char" in local_scan.h. The
      function should return one of the following values:

      OK            Success. The string that is placed in "yield" is put into
                    the expanded string that is being built.

      FAIL          A non-forced expansion failure occurs, with the error
                    message taken from "yield", if it is set.

      FAIL_FORCED   A forced expansion failure occurs, with the error message
                    taken from "yield" if it is set.

      ERROR         Same as FAIL, except that a panic log entry is written.

      When compiling a function that is to be used in this way with gcc,
      you need to add -shared to the gcc command. Also, in the Exim build-time
      configuration, you must add -export-dynamic to EXTRALIBS.

TF/01 $received_time is a new expansion variable containing the time and date
      as a number of seconds since the start of the Unix epoch when the
      current message was received.

PH/03 There is a new value for RADIUS_LIB_TYPE that can be set in
      Local/Makefile. It is RADIUSCLIENTNEW, and it requests that the new API,
      in use from radiusclient 0.4.0 onwards, be used. It does not appear to be
      possible to detect the different versions automatically.

PH/04 There is a new option called acl_not_smtp_mime that allows you to scan
      MIME parts in non-SMTP messages. It operates in exactly the same way as
      acl_smtp_mime

PH/05 It is now possible to redefine a macro within the configuration file.
      The macro must have been previously defined within the configuration (or
      an included file). A definition on the command line using the -D option
      causes all definitions and redefinitions within the file to be ignored.
      In other words, -D overrides any values that are set in the file.
      Redefinition is specified by using '==' instead of '='. For example:

        MAC1 =  initial value
        ...
        MAC1 == updated value

      Redefinition does not alter the order in which the macros are applied to
      the subsequent lines of the configuration file. It is still the same
      order in which the macros were originally defined. All that changes is
      the macro's value. Redefinition makes it possible to accumulate values.
      For example:

        MAC1 =  initial value
        ...
        MAC1 == MAC1 and something added

      This can be helpful in situations where the configuration file is built
      from a number of other files.

PH/06 Macros may now be defined or redefined between router, transport,
      authenticator, or ACL definitions, as well as in the main part of the
      configuration. They may not, however, be changed within an individual
      driver or ACL, or in the local_scan, retry, or rewrite sections of the
      configuration.

PH/07 $acl_verify_message is now set immediately after the failure of a
      verification in an ACL, and so is available in subsequent modifiers. In
      particular, the message can be preserved by coding like this:

         warn  !verify = sender
               set acl_m0 = $acl_verify_message

      Previously, $acl_verify_message was set only while expanding "message"
      and "log_message" when a very denied access.

PH/08 The redirect router has two new options, sieve_useraddress and
      sieve_subaddress. These are passed to a Sieve filter to specify the :user
      and :subaddress parts of an address. Both options are unset by default.
      However, when a Sieve filter is run, if sieve_useraddress is unset, the
      entire original local part (including any prefix or suffix) is used for
      :user. An unset subaddress is treated as an empty subaddress.

PH/09 Quota values can be followed by G as well as K and M.

PH/10 $message_linecount is a new variable that contains the total number of
      lines in the header and body of the message. Compare $body_linecount,
      which is the count for the body only. During the DATA and
      content-scanning ACLs, $message_linecount contains the number of lines
      received. Before delivery happens (that is, before filters, routers, and
      transports run) the count is increased to include the Received: header
      line that Exim standardly adds, and also any other header lines that are
      added by ACLs. The blank line that separates the message header from the
      body is not counted. Here is an example of the use of this variable in a
      DATA ACL:

        deny message   = Too many lines in message header
             condition = \
               ${if <{250}{${eval: $message_linecount - $body_linecount}}}

      In the MAIL and RCPT ACLs, the value is zero because at that stage the
      message has not yet been received.

PH/11 In a ${run expansion, the variable $value (which contains the standard
      output) is now also usable in the "else" string.

PH/12 In a pipe transport, although a timeout while waiting for the pipe
      process to complete was treated as a delivery failure, a timeout while
      writing the message to the pipe was logged, but erroneously treated as a
      successful delivery. Such timeouts include transport filter timeouts. For
      consistency with the overall process timeout, these timeouts are now
      treated as errors, giving rise to delivery failures by default. However,
      there is now a new Boolean option for the pipe transport called
      timeout_defer, which, if set TRUE, converts the failures into defers for
      both kinds of timeout. A transport filter timeout is now identified in
      the log output.


Version 4.50
------------

The documentation is up-to-date for the 4.50 release.

****