$Cambridge: exim/doc/doc-txt/NewStuff,v 1.24 2004/12/29 10:16:52 ph10 Exp $ New Features in Exim -------------------- This file contains descriptions of new features that have been added to Exim, but have not yet made it into the main manual (which is most conveniently updated when there is a relatively large batch of changes). The doc/ChangeLog file contains a listing of all changes, including bug fixes. Version 4.50 ------------ 1. There is a new build-time option called CONFIGURE_GROUP which works like CONFIGURE_OWNER. It specifies one additional group that is permitted for the runtime configuration file when the group write permission is set. 2. The "control=submission" facility has a new option /sender_retain. This has the effect of setting local_sender_retain true and local_from_check false for the incoming message in which it is encountered. 3. $recipients is now available in the predata ACL (oversight). 4. The value of address_data from a sender verification is now available in $sender_address_data in subsequent conditions in the ACL statement. Note: this is just like $address_data. The value does not persist after the end of the current ACL statement. If you want to preserve it, you can use one of the ACL variables. 5. The redirect router has two new options: forbid_sieve_filter and forbid_exim_filter. When filtering is enabled by allow_filter, these options control which type(s) of filtering are permitted. By default, both Exim and Sieve filters are allowed. 6. A new option for callouts makes it possible to set a different (usually smaller) timeout for making the SMTP connection. The keyword is "connect". For example: verify = sender/callout=5s,connect=1s If not specified, it defaults to the general timeout value. 7. The new variables $sender_verify_failure and $recipient_verify_failure contain information about exactly what failed. In an ACL, after one of these failures, the relevant variable contains one of the following words: qualify the address was unqualified (no domain), and the message was neither local nor came from an exempted host; route routing failed; mail routing succeeded, and a callout was attempted; rejection occurred at or before the MAIL command (that is, on initial connection, HELO, or MAIL); recipient the RCPT command in a callout was rejected; postmaster the postmaster check in a callout was rejected. The main use of these variables is expected to be to distinguish between rejections of MAIL and rejections of RCPT. 8. The command line option -dd behaves exactly like -d except when used on a command that starts a daemon process. In that case, debugging is turned off for the subprocesses that the daemon creates. Thus, it is useful for monitoring the behaviour of the daemon without creating as much output as full debugging. 9. $host_address is now set to the target address during the checking of ignore_target_hosts. 10. There are four new variables called $spool_space, $log_space, $spool_inodes, and $log_inodes. The first two contain the amount of free space in the disk partitions where Exim has its spool directory and log directory, respectively. (When these are in the same partition, the values will, of course, be the same.) The second two variables contain the numbers of free inodes in the respective partitions. NOTE: Because disks can nowadays be very large, the values in the space variables are in kilobytes rather than in bytes. Thus, for example, to check in an ACL that there is at least 50M free on the spool, you would write: condition = ${if > {$spool_space}{50000}{yes}{no}} The values are recalculated whenever any of these variables is referenced. If the relevant file system does not have the concept of inodes, the value of those variables is -1. If the operating system does not have the ability to find the amount of free space (only true for experimental systems), the space value is -1. 11. It is now permitted to omit both strings after an "if" condition; if the condition is true, the result is the string "true". As before, when the second string is omitted, a false condition yields an empty string. This makes it less cumbersome to write custom ACL and router conditions. For example, instead of condition = ${if eq {$acl_m4}{1}{yes}{no}} or the shorter form condition = ${if eq {$acl_m4}{1}{yes}} (because the second string has always defaulted to ""), you can now write condition = ${if eq {$acl_m4}{1}} Previously this was a syntax error. 12. There is a new "record type" that can be specified in dnsdb lookups. It is "zns" (for "zone NS"). It performs a lookup for NS records on the given domain, but if none are found, it removes the first component of the domain name, and tries again. This process continues until NS records are found or there are no more components left (or there's a DNS error). In other words, it may return the name servers for a top-level domain, but it never returns the root name servers. If there are no NS records for the top-level domain, the lookup fails. For example, ${lookup dnsdb{zns=xxx.quercite.com}} returns the name servers for quercite.com, whereas ${lookup dnsdb{zns=xxx.edu}} returns the name servers for edu, assuming in each case that there are no NS records for the full domain name. You should be careful about how you use this lookup because, unless the top-level domain does not exist, the lookup will always return some host names. The sort of use to which this might be put is for seeing if the name servers for a given domain are on a blacklist. You can probably assume that the name servers for the high-level domains such as .com or .co.uk are not going to be on such a list. 13. Another new "record type" is "mxh"; this looks up MX records just as "mx" does, but it returns only the names of the hosts, omitting the priority values. 14. It is now possible to specify a list of domains or IP addresses to be looked up in a dnsdb lookup. The list is specified in the normal Exim way, with colon as the default separator, but with the ability to change this. For example: ${lookup dnsdb{one.domain.com:two.domain.com}} ${lookup dnsdb{a=one.host.com:two.host.com}} ${lookup dnsdb{ptr = <; 1.2.3.4 ; 4.5.6.8}} In order to retain backwards compatibility, there is one special case: if the lookup type is PTR and no change of separator is specified, Exim looks to see if the rest of the string is precisely one IPv6 address. In this case, it does not treat it as a list. The data from each lookup is concatenated, with newline separators (by default - see 14 below), in the same way that multiple DNS records for a single item are handled. The dnsdb lookup fails only if all the DNS lookups fail. If there is a temporary DNS error for any of them, the behaviour is controlled by an optional keyword followed by a comma that may appear before the record type. The possible keywords are "defer_strict", "defer_never", and "defer_lax". With "strict" behaviour, any temporary DNS error causes the whole lookup to defer. With "never" behaviour, a temporary DNS error is ignored, and the behaviour is as if the DNS lookup failed to find anything. With "lax" behaviour, all the queries are attempted, but a temporary DNS error causes the whole lookup to defer only if none of the other lookups succeed. The default is "lax", so the following lookups are equivalent: ${lookup dnsdb{defer_lax,a=one.host.com:two.host.com}} ${lookup dnsdb{a=one.host.com:two.host.com}} Thus, in the default case, as long as at least one of the DNS lookups yields some data, the dnsdb lookup succeeds. 15. It is now possible to specify the character to be used as a separator when a dnsdb lookup returns data from more than one DNS record. The default is a newline. To specify a different character, put '>' followed by the new character at the start of the query. For example: ${lookup dnsdb{>: a=h1.test.ex:h2.test.ex}} ${lookup dnsdb{>| mxh=<;m1.test.ex;m2.test.ex}} It is permitted to specify a space as the separator character. Note that more than one DNS record can be found for a single lookup item; this feature is relevant even when you do not specify a list. The same effect could be achieved by wrapping the lookup in ${tr...}; this feature is just a syntactic simplification. 16. It is now possible to supply a list of domains and/or IP addresses to be lookup up in a DNS blacklist. Previously, only a single domain name could be given, for example: dnslists = black.list.tld/$sender_host_name What follows the slash can now be a list. As with all lists, the default separator is a colon. However, because this is a sublist within the list of DNS blacklist domains, it is necessary either to double the separators like this: dnslists = black.list.tld/name.1::name.2 or to change the separator character, like this: dnslists = black.list.tld/<;name.1;name.2 If an item in the list is an IP address, it is inverted before the DNS blacklist domain is appended. If it is not an IP address, no inversion occurs. Consider this condition: dnslists = black.list.tls/<;192.168.1.2;a.domain The DNS lookups that occur are for 2.1.168.192.black.list.tld and a.domain.black.list.tld Once a DNS record has been found (that matches a specific IP return address, if specified), no further lookups are done. If there is a temporary DNS error, the rest of the sublist of domains or IP addresses is tried. The dnslists item itself defers only if none of the other DNS lookups in this sublist succeeds. In other words, a successful lookup for any of the items in the sublist overrides a defer for a previous item. 17. The log selector queue_time_overall causes Exim to output the time spent on the queue as an addition to the "Completed" message. Like queue_time (which puts the queue time on individual delivery lines), the time is tagged with "QT=", and it is measured from the time that the message starts to be received, so it includes the reception time. 18. It is now possible to use both -bF and -bf on the same command, in order to test a system filter and a user filter in the same run. For example: exim -bF /system/filter -bf /user/filter } to the expansion operators. This operator converts an arbitrary string into one that is base64 encoded. 10. A new authenticator, called cyrus_sasl, has been added. This requires the presence of the Cyrus SASL library; it authenticates by calling this library, which supports a number of authentication mechanisms, including PLAIN and LOGIN, but also several others that Exim does not support directly. The code for this authenticator was provided by Matthew Byng-Maddick of A L Digital Ltd (http://www.aldigital.co.uk). Here follows draft documentation: xx. THE CYRUS_SASL AUTHENTICATOR The cyrus_sasl authenticator provides server support for the Cyrus library Implementation of the RFC 2222 "Simple Authentication and Security Layer". It provides a gatewaying mechanism directly to the Cyrus interface, so if your Cyrus library can do, for example, CRAM-MD5, then so can the cyrus_sasl authenticator. By default it uses the public name of the driver to determine which mechanism to support. Where access to some kind of secret file is required, for example in GSSAPI or CRAM-MD5, it is worth noting that the authenticator runs as the exim user, and that the Cyrus SASL library has no way of escalating privileges by default. You may also find you need to set environment variables, depending on the driver you are using. xx.1 Using cyrus_sasl as a server The cyrus_sasl authenticator has four private options. It puts the username (on a successful authentication) into $1. server_hostname Type: string* Default: $primary_hostname This option selects the hostname that is used when communicating with the library. It is up to the underlying SASL plug-in what it does with this data. server_mech Type: string Default: public_name This option selects the authentication mechanism this driver should use. It allows you to use a different underlying mechanism from the advertised name. For example: sasl: driver = cyrus_sasl public_name = X-ANYTHING server_mech = CRAM-MD5 server_set_id = $1 server_realm Type: string Default: unset This is the SASL realm that the server is claiming to be in. server_service Type: string Default: "smtp" This is the SASL service that the server claims to implement. For straigthforward cases, you do not need to set any of the authenticator's private options. All you need to do is to specify an appropriate mechanism as the public name. Thus, if you have a SASL library that supports CRAM-MD5 and PLAIN, you might have two authenticators as follows: sasl_cram_md5: driver = cyrus_sasl public_name = CRAM-MD5 server_set_id = $1 sasl_plain: driver = cyrus_sasl public_name = PLAIN server_set_id = $1 11. There is a new global option called tls_on_connect_ports. Its value must be a list of port numbers; the most common use is expected to be tls_on_connect_ports = 465 Setting this option has the same effect as -tls-on-connect on the command line, but only for the specified ports. It applies to all connections, both via the daemon and via inetd. You still need to specify all the ports for the daemon (using daemon_smtp_ports or local_interfaces or the -X command line option) because this option does not add an extra port -- rather, it specifies different behaviour on a port that is defined elsewhere. The -tls-on-connect command line option overrides tls_on_connect_ports, and forces tls-on-connect for all ports. 12. There is a new ACL that is run when a DATA command is received, before the data itself is received. The ACL is defined by acl_smtp_predata. (Compare acl_smtp_data, which is run after the data has been received.) This new ACL allows a negative response to be given to the DATA command itself. Header lines added by MAIL or RCPT ACLs are not visible at this time, but any that are defined here are visible when the acl_smtp_data ACL is run. 13. The "control=submission" ACL modifier has an option "/domain=xxx" which specifies the domain to be used when creating From: or Sender: lines using the authenticated id as a local part. If the option is supplied with an empty domain, that is, just "/domain=", Exim assumes that the authenticated id is a complete email address, and it uses it as is when creating From: or Sender: lines. 14. It is now possible to make retry rules that apply only when the failing message has a specific sender. In particular, this can be used to define retry rules that apply only to bounce messages. The syntax is to add a new third item to a retry rule, of the form "senders=
". The retry timings themselves then become the fourth item. For example: * * senders=: F,1h,30m would match all bounce messages. If the address list contains white space, it must be enclosed in quotes. For example: a.domain timeout senders="x@b.dom : y@c.dom" G,8h,10m,1.5 When testing retry rules using -brt, you can supply a sender using the -f command line option, like this: exim -f "" -brt user@dom.ain If you do not set -f with -brt, a retry rule that contains a senders list will never be matched. 15. Two new control modifiers have been added to ACLs: "control = enforce_sync" and "control = no_enforce_sync". This makes it possible to be selective about when SMTP synchronization is enforced. The global option smtp_enforce_sync now specifies the default state of the switch. These controls can appear in any ACL, but the most obvious place to put them is in the ACL defined by acl_smtp_connect, which is run at the start of an incoming SMTP connection, before the first synchronization check. 16. Another two new control modifiers are "control = caseful_local_part" and "control = caselower_local_part". These are permitted only in the ACL specified by acl_smtp_rcpt (i.e. during RCPT processing). By default, the contents of $local_part are lower cased before ACL processing. After "control = caseful_local_part", any uppercase letters in the original local part are restored in $local_part for the rest of the ACL, or until "control = caselower_local_part" is encountered. However, this applies only to local part handling that takes place directly in the ACL (for example, as a key in lookups). If a "verify = recipient" test is obeyed, the case-related handling of the local part during the verification is controlled by the router configuration (see the caseful_local_part generic router option). This facility could be used, for example, to add a spam score to local parts containing upper case letters. For example, using $acl_m4 to accumulate the spam score: warn control = caseful_local_part set acl_m4 = ${eval:\ $acl_m4 + \ ${if match{$local_part}{[A-Z]}{1}{0}}\ } control = caselower_local_part Notice that we put back the lower cased version afterwards, assuming that is what is wanted for subsequent tests. 17. The option hosts_connection_nolog is provided so that certain hosts can be excepted from logging when the +smtp_connection log selector is set. For example, you might want not to log SMTP connections from local processes, or from 127.0.0.1, or from your local LAN. The option is a host list with an unset default. Because it is consulted in the main loop of the daemon, you should strive to restrict its value to a short inline list of IP addresses and networks. To disable logging SMTP connections from local processes, you must create a host list with an empty item. For example: hosts_connection_nolog = : If the +smtp_connection log selector is not set, this option has no effect. 18. There is now an acl called acl_smtp_quit, which is run for the QUIT command. The outcome of the ACL does not affect the response code to QUIT, which is always 221. Thus, the ACL does not in fact control any access. For this reason, the only verbs that are permitted are "accept" and "warn". The ACL can be used for tasks such as custom logging at the end of an SMTP session. For example, you can use ACL variables in other ACLs to count messages, recipients, etc., and log the totals at QUIT time using one or more "logwrite" modifiers on a "warn" command. You do not need to have a final "accept", but if you do, you can use a "message" modifier to specify custom text that is sent as part of the 221 response. This ACL is run only for a "normal" QUIT. For certain kinds of disastrous failure (for example, failure to open a log file, or when Exim is bombing out because it has detected an unrecoverable error), all SMTP commands from the client are given temporary error responses until QUIT is received or the connection is closed. In these special cases, the ACL is not run. 19. The appendfile transport has two new options, mailbox_size and mailbox_ filecount. If either these options are set, it is expanded, and the result is taken as the current size of the mailbox or the number of files in the mailbox, respectively. This makes it possible to use some external means of maintaining the data about the size of a mailbox for enforcing quota limits. The result of expanding these option values must be a decimal number, optionally followed by "K" or "M". 20. It seems that there are broken clients in use that cannot handle multiline SMTP responses. Can't people who implement these braindead programs read? RFC 821 mentions multiline responses, and it is over 20 years old. They must handle multiline responses for EHLO, or do they still use HELO? Anyway, here is YAWFAB (yet another workaround for asinine brokenness). There's a new ACL switch that can be set by control = no_multiline_responses If this is set, it suppresses multiline SMTP responses from ACL rejections. One way of doing this would have been just to put out these responses as one long line. However, RFC 2821 specifies a maximum of 512 bytes per response ("use multiline responses for more" it says), and some of the responses might get close to that. So I have implemented this by doing two very easy things: (1) Extra information that is normally output as part of a rejection caused by sender verification failure is omitted. Only the final line (typically "sender verification failed") is now sent. (2) If a "message" modifier supplies a multiline response, only the first line is output. The setting of the switch can, of course, be made conditional on the calling host. 21. There is now support for the libradius library that comes with FreeBSD. This is an alternative to the radiusclient library that Exim already supports. To use the FreeBSD library, you need to set RADIUS_LIB_TYPE=RADLIB in Local/Makefile, in addition to RADIUS_CONFIGURE_FILE, and you probably also need -libradius in EXTRALIBS. Version 4.42 ------------ 1. The "personal" filter test is brought up-to-date with recommendations from the Sieve specification: (a) The list of non-personal From: addresses now includes "listserv", "majordomo", and "*-request"; (b) If the message contains any header line starting with "List=-" it is treated as non-personal. 2. The Sieve functionality has been extended to support the "copy" and "vacation" extensions, and comparison tests. 3. There is now an overall timeout for performing a callout verification. It defaults to 4 times the callout timeout, which applies to individual SMTP commands during the callout. The overall timeout applies when there is more than one host that can be tried. The timeout is checked before trying the next host. This prevents very long delays if there are a large number of hosts and all are timing out (e.g. when the network connections are timing out). The value of the overall timeout can be changed by specifying an additional sub-option for "callout", called "maxwait". For example: verify = sender/callout=5s,maxwait=20s 4. Changes to the "personal" filter test: (1) The list of non-personal local parts in From: addresses has been extended to include "listserv", "majordomo", "*-request", and "owner-*", taken from the Sieve specification recommendations. (2) If the message contains any header line starting with "List-" it is treated as non-personal. (3) The test for "circular" in the Subject: header line has been removed because it now seems ill-conceived. 5. The autoreply transport has a new option called never_mail. This is an address list. If any run of the transport creates a message with a recipient that matches any item in the list, that recipient is quietly discarded. If all recipients are discarded, no message is created. Version 4.40 ------------ The documentation is up-to-date for the 4.40 release. What follows here is a brief list of the new features that have been added since 4.30. 1. log_incoming_interface affects more log lines. 2. New ACL modifier "control = submission". 3. CONFIGURE_OWNER can be set at build time to define an alternative owner for the configuration file, in addition to root and exim. 4. Added expansion variables $body_zerocount, $recipient_data, and $sender_data. 5. The time of last modification of the "new" subdirectory is now used as the "mailbox time last read" when there is a quota error for a maildir delivery. 6. The special item "+ignore_unknown" may now appear in host lists. 7. The special domain-matching patterns @mx_any, @mx_primary, and @mx_secondary can now be followed by "/ignore=". 8. New expansion conditions: match_domain, match_address, match_local_part, lt, lti, le, lei, gt, gti, ge, and new expansion operators time_interval, eval10, and base62d. 9. New lookup type called "iplsearch". 10. New log selectors ident_timeout, tls_certificate_verified, queue_time, deliver_time, outgoing_port, return_path_on_delivery. 11. New global options smtp_active_hostname and tls_require_ciphers. 12. Exinext has -C and -D options. 13. "domainlist_cache" forces caching of an apparently variable list. 14. For compatibility with Sendmail, the command line option -prval:sval is equivalent to -oMr rval -oMs sval. 15. New callout options use_sender and use_postmaster for use when verifying recipients. 16. John Jetmore's "exipick" utility has been added to the distribution. 17. The TLS code now supports CRLs. 18. The dnslookup router and the dnsdb lookup type now support the use of SRV records. 19. The redirect router has a new option called qualify_domain. 20. exigrep's output now also includes lines that are not related to any particular message, but which do match the pattern. 21. New global option write_rejectlog. If it is set false, Exim no longer writes anything to the reject log. ****