1 $Cambridge: exim/doc/doc-txt/NewStuff,v 1.5 2004/11/04 10:42:11 ph10 Exp $
6 This file contains descriptions of new features that have been added to Exim,
7 but have not yet made it into the main manual (which is most conveniently
8 updated when there is a relatively large batch of changes). The doc/ChangeLog
9 file contains a listing of all changes, including bug fixes.
15 1. There is a new build-time option called CONFIGURE_GROUP which works like
16 CONFIGURE_OWNER. It specifies one additional group that is permitted for
17 the runtime configuration file when the group write permission is set.
19 2. The "control=submission" facility has a new option /retain_sender. This
20 has the effect of setting local_sender_retain true and local_from_check
21 false for the incoming message in which it is encountered.
23 3. $recipients is now available in the predata ACL (oversight).
25 4. The value of address_data from a sender verification is now available in
26 $sender_address_data in subsequent conditions in the ACL statement. Note:
27 this is just like $address_data. The value does not persist after the end
28 of the current ACL statement. If you want to preserve it, you can use one
31 5. The redirect router has two new options: forbid_sieve_filter and
32 forbid_exim_filter. When filtering is enabled by allow_filter, these
33 options control which type(s) of filtering are permitted. By default, both
34 Exim and Sieve filters are allowed.
40 1. There is a new Boolean global option called mua_wrapper, defaulting false.
41 This causes Exim to run an a restricted mode, in order to provide a very
44 Background: On a personal computer, it is a common requirement for all
45 email to be sent to a smarthost. There are plenty of MUAs that can be
46 configured to operate that way, for all the popular operating systems.
47 However, there are MUAs for Unix-like systems that cannot be so configured:
48 they submit messages using the command line interface of
49 /usr/sbin/sendmail. In addition, utility programs such as cron submit
52 Requirement: The requirement is for something that can provide the
53 /usr/sbin/sendmail interface and deliver messages to a smarthost, but not
54 provide any queueing or retrying facilities. Furthermore, the delivery to
55 the smarthost should be synchronous, so that if it fails, the sending MUA
56 is immediately informed. In other words, we want something that in effect
57 converts a command-line MUA into a TCP/SMTP MUA.
59 Solutions: There are a number of applications (for example, ssmtp) that do
60 this job. However, people have found them to be lacking in various ways.
61 For instance, some sites want to allow aliasing and forwarding before
62 sending to the smarthost.
64 Using Exim: Exim already had the necessary infrastructure for doing this
65 job. Just a few tweaks were needed to make it behave as required, though it
66 is somewhat of an overkill to use a fully-featured MTA for this purpose.
68 Setting mua_wrapper=true causes Exim to run in a special mode where it
69 assumes that it is being used to "wrap" a command-line MUA in the manner
72 If you set mua_wrapper=true, you also need to provide a compatible router
73 and transport configuration. Typically there will be just one router and
74 one transport, sending everything to a smarthost.
76 When run in MUA wrapping mode, the behaviour of Exim changes in the
79 (a) A daemon cannot be run, nor will Exim accept incoming messages from
80 inetd. In other words, the only way to submit messages is via the
83 (b) Each message is synchonously delivered as soon as it is received (-odi
84 is assumed). All queueing options (queue_only, queue_smtp_domains,
85 control=queue, control=freeze in an ACL etc.) are quietly ignored. The
86 Exim reception process does not finish until the delivery attempt is
87 complete. If the delivery was successful, a zero return code is given.
89 (c) Address redirection is permitted, but the final routing for all
90 addresses must be to the same remote transport, and to the same list of
91 hosts. Furthermore, the return_address must be the same for all
92 recipients, as must any added or deleted header lines. In other words,
93 it must be possible to deliver the message in a single SMTP
94 transaction, however many recipients there are.
96 (d) If the conditions in (c) are not met, or if routing any address results
97 in a failure or defer status, or if Exim is unable to deliver all the
98 recipients successfully to one of the hosts immediately, delivery of
99 the entire message fails.
101 (e) Because no queueing is allowed, all failures are treated as permanent;
102 there is no distinction between 4xx and 5xx SMTP response codes from
103 the smarthost. Furthermore, because only a single yes/no response can
104 be given to the caller, it is not possible to deliver to some
105 recipients and not others. If there is an error (temporary or
106 permanent) for any recipient, all are failed.
108 (f) If more than one host is listed, Exim will try another host after a
109 connection failure or a timeout, in the normal way. However, if this
110 kind of failure happens for all the hosts, the delivery fails.
112 (g) When delivery fails, an error message is written to the standard error
113 stream (as well as to Exim's log), and Exim exits to the caller with a
114 return code value 1. The message is expunged from Exim's spool files.
115 No bounce messages are ever generated.
117 (h) No retry data is maintained, and any retry rules are ignored.
119 (i) A number of Exim options are overridden: deliver_drop_privilege is
120 forced true, max_rcpt in the smtp transport is forced to "unlimited",
121 remote_max_parallel is forced to one, and fallback hosts are ignored.
123 The overall effect is that Exim makes a single synchronous attempt to
124 deliver the message, failing if there is any kind of problem. Because no
125 local deliveries are done and no daemon can be run, Exim does not need root
126 privilege. It should be possible to run it setuid=exim instead of
127 setuid=root. See section 48.3 in the 4.40 manual for a general discussion
128 about the advantages and disadvantages of running without root privilege.
130 2. There have been problems with DNS servers when SRV records are looked up.
131 Some mis-behaving servers return a DNS error or timeout when a non-existent
132 SRV record is sought. Similar problems have in the past been reported for
133 MX records. The global dns_again_means_nonexist option can help with this
134 problem, but it is heavy-handed because it is a global option. There are
135 now two new options for the dnslookup router. They are called
136 srv_fail_domains and mx_fail_domains. In each case, the value is a domain
137 list. If an attempt to look up an SRV or MX record results in a DNS failure
138 or "try again" response, and the domain matches the relevant list, Exim
139 behaves as if the DNS had responded "no such record". In the case of an SRV
140 lookup, this means that the router proceeds to look for MX records; in the
141 case of an MX lookup, it proceeds to look for A or AAAA records, unless the
142 domain matches mx_domains.
144 3. The following functions are now available in the local_scan() API:
146 (a) void header_remove(int occurrence, uschar *name)
148 This function removes header lines. If "occurrence" is zero or negative,
149 all occurrences of the header are removed. If occurrence is greater
150 than zero, that particular instance of the header is removed. If no
151 header(s) can be found that match the specification, the function does
154 (b) BOOL header_testname(header_line *hdr, uschar *name, int length,
157 This function tests whether the given header has the given name. It
158 is not just a string comparison, because whitespace is permitted
159 between the name and the colon. If the "notdel" argument is TRUE, a
160 FALSE return is forced for all "deleted" headers; otherwise they are
161 not treated specially. For example:
163 if (header_testname(h, US"X-Spam", 6, TRUE)) ...
165 (c) void header_add_at_position(BOOL after, uschar *name, BOOL topnot,
166 int type, char *format, ...)
168 This function adds a new header line at a specified point in the header
169 chain. If "name" is NULL, the new header is added at the end of the
170 chain if "after" is TRUE, or at the start if "after" is FALSE. If
171 "name" is not NULL, the headers are searched for the first non-deleted
172 header that matches the name. If one is found, the new header is added
173 before it if "after" is FALSE. If "after" is true, the new header is
174 added after the found header and any adjacent subsequent ones with the
175 same name (even if marked "deleted"). If no matching non-deleted header
176 is found, the "topnot" option controls where the header is added. If it
177 is TRUE, addition is at the top; otherwise at the bottom. Thus, to add
178 a header after all the Received: headers, or at the top if there are no
179 Received: headers, you could use
181 header_add_at_position(TRUE, US"Received", TRUE, ' ', "X-xxx: ...");
183 Normally, there is always at least one non-deleted Received: header,
184 but there may not be if received_header_text expands to an empty
187 (d) BOOL receive_remove_recipient(uschar *recipient)
189 This is a convenience function to remove a named recipient from the
190 list of recipients. It returns TRUE if a recipient was removed, and
191 FALSE if no matching recipient could be found. The argument must be a
192 complete email address.
194 4. When an ACL "warn" statement adds one or more header lines to a message,
195 they are added at the end of the existing header lines by default. It is
196 now possible to specify that any particular header line should be added
197 right at the start (before all the Received: lines) or immediately after
198 the first block of Received: lines in the message. This is done by
199 specifying :at_start: or :after_received: (or, for completeness, :at_end:)
200 before the text of the header line. (Header text cannot start with a colon,
201 as there has to be a header name first.) For example:
203 warn message = :after_received:X-My-Header: something or other...
205 If more than one header is supplied in a single warn statement, each one is
206 treated independently and can therefore be placed differently. If you add
207 more than one line at the start, or after the Received: block, they will
208 end up in reverse order.
210 Warning: This facility currently applies only to header lines that are
211 added in an ACL. It does NOT work for header lines that are added in a
212 system filter or in a router or transport.
214 5. There is now a new error code that can be used in retry rules. Its name is
215 "rcpt_4xx", and there are three forms. A literal "rcpt_4xx" matches any 4xx
216 error received for an outgoing SMTP RCPT command; alternatively, either the
217 first or both of the x's can be given as digits, for example: "rcpt_45x" or
218 "rcpt_436". If you want (say) to recognize 452 errors given to RCPT
219 commands by a particular host, and have only a one-hour retry for them, you
220 can set up a retry rule of this form:
222 the.host.name rcpt_452 F,1h,10m
224 Naturally, this rule must come before any others that would match.
226 These new errors apply to both outgoing SMTP (the smtp transport) and
227 outgoing LMTP (either the lmtp transport, or the smtp transport in LMTP
228 mode). Note, however, that they apply only to responses to RCPT commands.
230 6. The "postmaster" option of the callout feature of address verification has
231 been extended to make it possible to use a non-empty MAIL FROM address when
232 checking a postmaster address. The new suboption is called "postmaster_
233 mailfrom", and you use it like this:
235 require verify = sender/callout=postmaster_mailfrom=abc@x.y.z
237 Providing this suboption causes the postmaster check to be done using the
238 given address. The original "postmaster" option is equivalent to
240 require verify = sender/callout=postmaster_mailfrom=
242 If both suboptions are present, the rightmost one overrides.
246 (1) If you use a non-empty sender address for postmaster checking, there is
247 the likelihood that the remote host will itself initiate a callout
248 check back to your host to check that address. As this is a "normal"
249 callout check, the sender will most probably be empty, thus avoiding
250 possible callout loops. However, to be on the safe side it would be
251 best to set up your own ACLs so that they do not do sender verification
252 checks when the recipient is the address you use for postmaster callout
255 (2) The caching arrangements for postmaster checking do NOT take account of
256 the sender address. It is assumed that either the empty address, or a
257 fixed non-empty address will be used. All that Exim remembers is that
258 the postmaster check for the domain succeeded or failed.
260 7. When verifying addresses in header lines using the verify=header_sender
261 option, Exim behaves by default as if the addresses are envelope sender
262 addresses from a message. Callout verification therefore tests to see
263 whether a bounce message could be delivered, by using an empty address in
264 the MAIL FROM command. However, it is arguable that these addresses might
265 never be used as envelope senders, and could therefore justifiably reject
266 bounce messages (empty senders). There is now an additional callout option
267 for verify=header_sender that allows you to specify what address to use in
268 the MAIL FROM command. You use it as in this example:
270 require verify = header_sender/callout=mailfrom=abcd@x.y.z
274 (1) As in the case of postmaster_mailfrom (see above), you should think
275 about possible loops.
277 (2) In this case, as in the case of recipient callouts with non-empty
278 senders (the use_sender option), caching is done on the basis of a
279 recipient/sender pair.
281 8. If you build Exim with USE_READLINE=yes in Local/Makefile, it will try to
282 load libreadline dynamically whenever the -be (test expansion) option is
283 used without command line arguments. If successful, it will then use
284 readline() for reading the test data. A line history is supported. By the
285 time Exim does this, it is running as the calling user, so this should not
286 cause any security problems. Security is the reason why this is NOT
287 supported for -bt or -bv, when Exim is running as root or exim,
288 respectively. Note that this option adds to the size of the Exim binary,
289 because the dynamic loading library is not otherwise included. On my
290 desktop it adds about 2.5K. You may need to add -ldl to EXTRA_LIBS when you
291 set USE_READLINE=yes.
293 9. Added ${str2b64:<string>} to the expansion operators. This operator
294 converts an arbitrary string into one that is base64 encoded.
296 10. A new authenticator, called cyrus_sasl, has been added. This requires
297 the presence of the Cyrus SASL library; it authenticates by calling this
298 library, which supports a number of authentication mechanisms, including
299 PLAIN and LOGIN, but also several others that Exim does not support
300 directly. The code for this authenticator was provided by Matthew
301 Byng-Maddick of A L Digital Ltd (http://www.aldigital.co.uk). Here follows
304 xx. THE CYRUS_SASL AUTHENTICATOR
306 The cyrus_sasl authenticator provides server support for the Cyrus library
307 Implementation of the RFC 2222 "Simple Authentication and Security Layer".
308 It provides a gatewaying mechanism directly to the Cyrus interface, so if
309 your Cyrus library can do, for example, CRAM-MD5, then so can the
310 cyrus_sasl authenticator. By default it uses the public name of the driver
311 to determine which mechanism to support.
313 Where access to some kind of secret file is required, for example in GSSAPI
314 or CRAM-MD5, it is worth noting that the authenticator runs as the exim
315 user, and that the Cyrus SASL library has no way of escalating privileges
316 by default. You may also find you need to set environment variables,
317 depending on the driver you are using.
319 xx.1 Using cyrus_sasl as a server
321 The cyrus_sasl authenticator has four private options. It puts the username
322 (on a successful authentication) into $1.
324 server_hostname Type: string* Default: $primary_hostname
326 This option selects the hostname that is used when communicating with
327 the library. It is up to the underlying SASL plug-in what it does with
330 server_mech Type: string Default: public_name
332 This option selects the authentication mechanism this driver should
333 use. It allows you to use a different underlying mechanism from the
334 advertised name. For example:
338 public_name = X-ANYTHING
339 server_mech = CRAM-MD5
342 server_realm Type: string Default: unset
344 This is the SASL realm that the server is claiming to be in.
346 server_service Type: string Default: "smtp"
348 This is the SASL service that the server claims to implement.
350 For straigthforward cases, you do not need to set any of the
351 authenticator's private options. All you need to do is to specify an
352 appropriate mechanism as the public name. Thus, if you have a SASL library
353 that supports CRAM-MD5 and PLAIN, you might have two authenticators as
358 public_name = CRAM-MD5
366 11. There is a new global option called tls_on_connect_ports. Its value must be
367 a list of port numbers; the most common use is expected to be
369 tls_on_connect_ports = 465
371 Setting this option has the same effect as -tls-on-connect on the command
372 line, but only for the specified ports. It applies to all connections, both
373 via the daemon and via inetd. You still need to specify all the ports for
374 the daemon (using daemon_smtp_ports or local_interfaces or the -X command
375 line option) because this option does not add an extra port -- rather, it
376 specifies different behaviour on a port that is defined elsewhere. The
377 -tls-on-connect command line option overrides tls_on_connect_ports, and
378 forces tls-on-connect for all ports.
380 12. There is a new ACL that is run when a DATA command is received, before the
381 data itself is received. The ACL is defined by acl_smtp_predata. (Compare
382 acl_smtp_data, which is run after the data has been received.)
383 This new ACL allows a negative response to be given to the DATA command
384 itself. Header lines added by MAIL or RCPT ACLs are not visible at this
385 time, but any that are defined here are visible when the acl_smtp_data ACL
388 13. The "control=submission" ACL modifier has an option "/domain=xxx" which
389 specifies the domain to be used when creating From: or Sender: lines using
390 the authenticated id as a local part. If the option is supplied with an
391 empty domain, that is, just "/domain=", Exim assumes that the authenticated
392 id is a complete email address, and it uses it as is when creating From:
395 14. It is now possible to make retry rules that apply only when the failing
396 message has a specific sender. In particular, this can be used to define
397 retry rules that apply only to bounce messages. The syntax is to add a new
398 third item to a retry rule, of the form "senders=<address list>". The retry
399 timings themselves then become the fourth item. For example:
401 * * senders=: F,1h,30m
403 would match all bounce messages. If the address list contains white space,
404 it must be enclosed in quotes. For example:
406 a.domain timeout senders="x@b.dom : y@c.dom" G,8h,10m,1.5
408 When testing retry rules using -brt, you can supply a sender using the -f
409 command line option, like this:
411 exim -f "" -brt user@dom.ain
413 If you do not set -f with -brt, a retry rule that contains a senders list
414 will never be matched.
416 15. Two new control modifiers have been added to ACLs: "control = enforce_sync"
417 and "control = no_enforce_sync". This makes it possible to be selective
418 about when SMTP synchronization is enforced. The global option
419 smtp_enforce_sync now specifies the default state of the switch. These
420 controls can appear in any ACL, but the most obvious place to put them is
421 in the ACL defined by acl_smtp_connect, which is run at the start of an
422 incoming SMTP connection, before the first synchronization check.
424 16. Another two new control modifiers are "control = caseful_local_part" and
425 "control = caselower_local_part". These are permitted only in the ACL
426 specified by acl_smtp_rcpt (i.e. during RCPT processing). By default, the
427 contents of $local_part are lower cased before ACL processing.
428 After "control = caseful_local_part", any uppercase letters in the original
429 local part are restored in $local_part for the rest of the ACL, or until
430 "control = caselower_local_part" is encountered. However, this applies only
431 to local part handling that takes place directly in the ACL (for example,
432 as a key in lookups). If a "verify = recipient" test is obeyed, the
433 case-related handling of the local part during the verification is
434 controlled by the router configuration (see the caseful_local_part generic
437 This facility could be used, for example, to add a spam score to local
438 parts containing upper case letters. For example, using $acl_m4 to
439 accumulate the spam score:
441 warn control = caseful_local_part
442 set acl_m4 = ${eval:\
444 ${if match{$local_part}{[A-Z]}{1}{0}}\
446 control = caselower_local_part
448 Notice that we put back the lower cased version afterwards, assuming that
449 is what is wanted for subsequent tests.
451 17. The option hosts_connection_nolog is provided so that certain hosts can be
452 excepted from logging when the +smtp_connection log selector is set. For
453 example, you might want not to log SMTP connections from local processes,
454 or from 127.0.0.1, or from your local LAN. The option is a host list with
455 an unset default. Because it is consulted in the main loop of the daemon,
456 you should strive to restrict its value to a short inline list of IP
457 addresses and networks. To disable logging SMTP connections from local
458 processes, you must create a host list with an empty item. For example:
460 hosts_connection_nolog = :
462 If the +smtp_connection log selector is not set, this option has no effect.
464 18. There is now an acl called acl_smtp_quit, which is run for the QUIT
465 command. The outcome of the ACL does not affect the response code to QUIT,
466 which is always 221. Thus, the ACL does not in fact control any access.
467 For this reason, the only verbs that are permitted are "accept" and "warn".
469 The ACL can be used for tasks such as custom logging at the end of an SMTP
470 session. For example, you can use ACL variables in other ACLs to count
471 messages, recipients, etc., and log the totals at QUIT time using one or
472 more "logwrite" modifiers on a "warn" command.
474 You do not need to have a final "accept", but if you do, you can use a
475 "message" modifier to specify custom text that is sent as part of the 221
478 This ACL is run only for a "normal" QUIT. For certain kinds of disastrous
479 failure (for example, failure to open a log file, or when Exim is bombing
480 out because it has detected an unrecoverable error), all SMTP commands
481 from the client are given temporary error responses until QUIT is received
482 or the connection is closed. In these special cases, the ACL is not run.
484 19. The appendfile transport has two new options, mailbox_size and mailbox_
485 filecount. If either these options are set, it is expanded, and the result
486 is taken as the current size of the mailbox or the number of files in the
487 mailbox, respectively. This makes it possible to use some external means of
488 maintaining the data about the size of a mailbox for enforcing quota
489 limits. The result of expanding these option values must be a decimal
490 number, optionally followed by "K" or "M".
492 20. It seems that there are broken clients in use that cannot handle multiline
493 SMTP responses. Can't people who implement these braindead programs read?
494 RFC 821 mentions multiline responses, and it is over 20 years old. They
495 must handle multiline responses for EHLO, or do they still use HELO?
496 Anyway, here is YAWFAB (yet another workaround for asinine brokenness).
497 There's a new ACL switch that can be set by
499 control = no_multiline_responses
501 If this is set, it suppresses multiline SMTP responses from ACL rejections.
502 One way of doing this would have been just to put out these responses as
503 one long line. However, RFC 2821 specifies a maximum of 512 bytes per
504 response ("use multiline responses for more" it says), and some of the
505 responses might get close to that. So I have implemented this by doing two
508 (1) Extra information that is normally output as part of a rejection
509 caused by sender verification failure is omitted. Only the final line
510 (typically "sender verification failed") is now sent.
512 (2) If a "message" modifier supplies a multiline response, only the first
515 The setting of the switch can, of course, be made conditional on the
518 21. There is now support for the libradius library that comes with FreeBSD.
519 This is an alternative to the radiusclient library that Exim already
520 supports. To use the FreeBSD library, you need to set
522 RADIUS_LIB_TYPE=RADLIB
524 in Local/Makefile, in addition to RADIUS_CONFIGURE_FILE, and you probably
525 also need -libradius in EXTRALIBS.
531 1. The "personal" filter test is brought up-to-date with recommendations from
532 the Sieve specification: (a) The list of non-personal From: addresses now
533 includes "listserv", "majordomo", and "*-request"; (b) If the message
534 contains any header line starting with "List=-" it is treated as
537 2. The Sieve functionality has been extended to support the "copy" and
538 "vacation" extensions, and comparison tests.
540 3. There is now an overall timeout for performing a callout verification. It
541 defaults to 4 times the callout timeout, which applies to individual SMTP
542 commands during the callout. The overall timeout applies when there is more
543 than one host that can be tried. The timeout is checked before trying the
544 next host. This prevents very long delays if there are a large number of
545 hosts and all are timing out (e.g. when the network connections are timing
546 out). The value of the overall timeout can be changed by specifying an
547 additional sub-option for "callout", called "maxwait". For example:
549 verify = sender/callout=5s,maxwait=20s
551 4. Changes to the "personal" filter test:
553 (1) The list of non-personal local parts in From: addresses has been
554 extended to include "listserv", "majordomo", "*-request", and "owner-*",
555 taken from the Sieve specification recommendations.
557 (2) If the message contains any header line starting with "List-" it is
558 treated as non-personal.
560 (3) The test for "circular" in the Subject: header line has been removed
561 because it now seems ill-conceived.
563 5. The autoreply transport has a new option called never_mail. This is an
564 address list. If any run of the transport creates a message with a
565 recipient that matches any item in the list, that recipient is quietly
566 discarded. If all recipients are discarded, no message is created.
572 The documentation is up-to-date for the 4.40 release. What follows here is a
573 brief list of the new features that have been added since 4.30.
575 1. log_incoming_interface affects more log lines.
577 2. New ACL modifier "control = submission".
579 3. CONFIGURE_OWNER can be set at build time to define an alternative owner for
580 the configuration file, in addition to root and exim.
582 4. Added expansion variables $body_zerocount, $recipient_data, and
585 5. The time of last modification of the "new" subdirectory is now used as the
586 "mailbox time last read" when there is a quota error for a maildir
589 6. The special item "+ignore_unknown" may now appear in host lists.
591 7. The special domain-matching patterns @mx_any, @mx_primary, and
592 @mx_secondary can now be followed by "/ignore=<ip list>".
594 8. New expansion conditions: match_domain, match_address, match_local_part,
595 lt, lti, le, lei, gt, gti, ge, and new expansion operators time_interval,
598 9. New lookup type called "iplsearch".
600 10. New log selectors ident_timeout, tls_certificate_verified, queue_time,
601 deliver_time, outgoing_port, return_path_on_delivery.
603 11. New global options smtp_active_hostname and tls_require_ciphers.
605 12. Exinext has -C and -D options.
607 13. "domainlist_cache" forces caching of an apparently variable list.
609 14. For compatibility with Sendmail, the command line option -prval:sval
610 is equivalent to -oMr rval -oMs sval.
612 15. New callout options use_sender and use_postmaster for use when verifying
615 16. John Jetmore's "exipick" utility has been added to the distribution.
617 17. The TLS code now supports CRLs.
619 18. The dnslookup router and the dnsdb lookup type now support the use of SRV
622 19. The redirect router has a new option called qualify_domain.
624 20. exigrep's output now also includes lines that are not related to any
625 particular message, but which do match the pattern.
627 21. New global option write_rejectlog. If it is set false, Exim no longer
628 writes anything to the reject log.
git://git.exim.org / exim.git / blob