Named queues: support multiple queue-runners from single daemon

[exim.git] / doc / doc-docbook / spec.xfpt

diff --git a/doc/doc-docbook/spec.xfpt b/doc/doc-docbook/spec.xfpt
index af601665e160edf116f8bf1d4fc0c86f7745136c..6199b5d8946e0fc071b86523f6438b48025c9ee2 100644 (file)
--- a/doc/doc-docbook/spec.xfpt
+++ b/doc/doc-docbook/spec.xfpt
@@ -2810,6 +2810,14 @@ of Exim is installed. It is not necessary to do this when other files that are
 referenced from the configuration (for example, alias files) are changed,
 because these are reread each time they are used.
 
 referenced from the configuration (for example, alias files) are changed,
 because these are reread each time they are used.
 

+.new
+Either a SIGTERM or a SIGINT signal should be used to cause the daemon
+to cleanly shut down.
+Subprocesses handling recceiving or delivering messages,
+or for scanning the queue,
+will not be affected by the termination of the daemon process.
+.wen
+

 .cmdopt -bdf
 This option has the same effect as &%-bd%& except that it never disconnects
 from the controlling terminal, even when no debugging is specified.
 .cmdopt -bdf
 This option has the same effect as &%-bd%& except that it never disconnects
 from the controlling terminal, even when no debugging is specified.


@@ -4397,15 +4405,21 @@ It is only relevant when the &%-bd%& (start listening daemon) option is also
 given.
 Normally the daemon creates this socket, unless a &%-oX%& and &*no*& &%-oP%&
 option is also present.
 given.
 Normally the daemon creates this socket, unless a &%-oX%& and &*no*& &%-oP%&
 option is also present.

-If this option is given then the socket will not be created.  This could be
-required if the system is running multiple daemons.
+.new
+If this option is given then the socket will not be created.  This is required
+if the system is running multiple daemons, in which case it should
+be used on all.
+The features supported by the socket will not be available in such cases.

 
 The socket is currently used for
 .ilist
 fast ramp-up of queue runner processes
 .next
 
 The socket is currently used for
 .ilist
 fast ramp-up of queue runner processes
 .next

+caching compiled regexes
+.next

 obtaining a current queue size
 .endlist
 obtaining a current queue size
 .endlist

+.wen

 
 .cmdopt -pd
 .cindex "Perl" "starting the interpreter"
 
 .cmdopt -pd
 .cindex "Perl" "starting the interpreter"


@@ -4489,23 +4503,33 @@ every domain. Addresses are routed, local deliveries happen, but no remote
 transports are run.
 
 Performance will be best if the &%queue_run_in_order%& option is false.
 transports are run.
 
 Performance will be best if the &%queue_run_in_order%& option is false.

-If that is so and the &%queue_fast_ramp%& option is true then
-in the first phase of the run,
+If that is so and
+the &%queue_fast_ramp%& option is true
+and a daemon-notifier socket is available
+then in the first phase of the run,

 once a threshold number of messages are routed for a given host,
 a delivery process is forked in parallel with the rest of the scan.
 
 .cindex "hints database" "remembering routing"
 The hints database that remembers which messages are waiting for specific hosts
 once a threshold number of messages are routed for a given host,
 a delivery process is forked in parallel with the rest of the scan.
 
 .cindex "hints database" "remembering routing"
 The hints database that remembers which messages are waiting for specific hosts

-is updated, as if delivery to those hosts had been deferred. After this is
-complete, a second, normal queue scan happens, with routing and delivery taking
-place as normal. Messages that are routed to the same host should mostly be
+is updated, as if delivery to those hosts had been deferred.
+
+After the first queue scan complete,
+a second, normal queue scan is done, with routing and delivery taking
+place as normal.
+Messages that are routed to the same host should mostly be

 delivered down a single SMTP
 .cindex "SMTP" "passed connection"
 .cindex "SMTP" "multiple deliveries"
 .cindex "multiple SMTP deliveries"
 connection because of the hints that were set up during the first queue scan.
 delivered down a single SMTP
 .cindex "SMTP" "passed connection"
 .cindex "SMTP" "multiple deliveries"
 .cindex "multiple SMTP deliveries"
 connection because of the hints that were set up during the first queue scan.

-This option may be useful for hosts that are connected to the Internet
+
+.new
+Two-phase queue runs should be used on systems which, even intermittently,
+have a large queue (such as mailing-list operators).
+They may also be useful for hosts that are connected to the Internet

 intermittently.
 intermittently.

+.wen

 
 .vitem &%-q[q]i...%&
 .oindex "&%-qi%&"
 
 .vitem &%-q[q]i...%&
 .oindex "&%-qi%&"


@@ -4591,6 +4615,15 @@ combined daemon at system boot time is to use a command such as
 Such a daemon listens for incoming SMTP calls, and also starts a queue runner
 process every 30 minutes.
 
 Such a daemon listens for incoming SMTP calls, and also starts a queue runner
 process every 30 minutes.
 

+.new
+.cindex "named queues" "queue runners"
+It is possible to set up runners for multiple named queues within one daemon,
+For example:
+.code
+exim -qGhipri/2m -q10m -qqGmailinglist/1h
+.endd
+.wen
+

 When a daemon is started by &%-q%& with a time value, but without &%-bd%&, no
 pid file is written unless one is explicitly requested by the &%-oP%& option.
 
 When a daemon is started by &%-q%& with a time value, but without &%-bd%&, no
 pid file is written unless one is explicitly requested by the &%-oP%& option.
 


@@ -5103,6 +5136,10 @@ The following classes of macros are defined:
 &` _DRIVER_ROUTER_*           `&  router drivers
 &` _DRIVER_TRANSPORT_*        `&  transport drivers
 &` _DRIVER_AUTHENTICATOR_*    `&  authenticator drivers
 &` _DRIVER_ROUTER_*           `&  router drivers
 &` _DRIVER_TRANSPORT_*        `&  transport drivers
 &` _DRIVER_AUTHENTICATOR_*    `&  authenticator drivers

+&` _EXP_COND_*                `&  expansion conditions
+&` _EXP_ITEM_*                `&  expansion items
+&` _EXP_OP_*                  `&  expansion operators
+&` _EXP_VAR_*                 `&  expansion variables

 &` _LOG_*                     `&  log_selector values
 &` _OPT_MAIN_*                `&  main config options
 &` _OPT_ROUTERS_*             `&  generic router options
 &` _LOG_*                     `&  log_selector values
 &` _OPT_MAIN_*                `&  main config options
 &` _OPT_ROUTERS_*             `&  generic router options


@@ -6602,7 +6639,7 @@ file that is searched could contain lines like this:
 When the lookup succeeds, the result of the expansion is a list of domains (and
 possibly other types of item that are allowed in domain lists).
 .cindex "tainted data" "de-tainting"
 When the lookup succeeds, the result of the expansion is a list of domains (and
 possibly other types of item that are allowed in domain lists).
 .cindex "tainted data" "de-tainting"

-.cindex "de-tainting" "using a lookup expansion""
+.cindex "de-tainting" "using a lookup expansion"

 The result of the expansion is not tainted.
 
 .next
 The result of the expansion is not tainted.
 
 .next


@@ -7799,7 +7836,8 @@ connection timeout (the system timeout is used), no user or password, no limit
 on the number of entries returned, and no time limit on queries.
 
 When a DN is quoted in the USER= setting for LDAP authentication, Exim
 on the number of entries returned, and no time limit on queries.
 
 When a DN is quoted in the USER= setting for LDAP authentication, Exim

-removes any URL quoting that it may contain before passing it LDAP. Apparently
+removes any URL quoting that it may contain before passing it to the LDAP library.
+Apparently

 some libraries do this for themselves, but some do not. Removing the URL
 quoting has two advantages:
 
 some libraries do this for themselves, but some do not. Removing the URL
 quoting has two advantages:
 


@@ -9648,7 +9686,10 @@ Example use (as an ACL modifier):
 .code
       add_header = :at_start:${authresults {$primary_hostname}}
 .endd
 .code
       add_header = :at_start:${authresults {$primary_hostname}}
 .endd

-This is safe even if no authentication results are available.
+This is safe even if no authentication results are available
+.new
+and would generally be placed in the DATA ACL.
+.wen

 
 
 .vitem "&*${certextract{*&<&'field'&>&*}{*&<&'certificate'&>&*}&&&
 
 
 .vitem "&*${certextract{*&<&'field'&>&*}{*&<&'certificate'&>&*}&&&


@@ -11759,8 +11800,8 @@ Case and collation order are defined per the system C locale.
 SRS decode.  See SECT &<<SECTSRS>>& for details.
 
 
 SRS decode.  See SECT &<<SECTSRS>>& for details.
 
 

-.vitem &*inlist&~{*&<&'string1'&>&*}{*&<&'string2'&>&*}*& &&&
-       &*inlisti&~{*&<&'string1'&>&*}{*&<&'string2'&>&*}*&
+.vitem &*inlist&~{*&<&'subject'&>&*}{*&<&'list'&>&*}*& &&&
+       &*inlisti&~{*&<&'subject'&>&*}{*&<&'list'&>&*}*&

 .cindex "string" "comparison"
 .cindex "list" "iterative conditions"
 Both strings are expanded; the second string is treated as a list of simple
 .cindex "string" "comparison"
 .cindex "list" "iterative conditions"
 Both strings are expanded; the second string is treated as a list of simple


@@ -15613,7 +15654,12 @@ by a setting such as this:
 .code
 dns_again_means_nonexist = *.in-addr.arpa
 .endd
 .code
 dns_again_means_nonexist = *.in-addr.arpa
 .endd

-This option applies to all DNS lookups that Exim does. It also applies when the
+This option applies to all DNS lookups that Exim does,
+.new
+except for TLSA lookups (where knowing about such failures
+is security-relevant).
+.wen
+It also applies when the

 &[gethostbyname()]& or &[getipnodebyname()]& functions give temporary errors,
 since these are most likely to be caused by DNS lookup problems. The
 &(dnslookup)& router has some options of its own for controlling what happens
 &[gethostbyname()]& or &[getipnodebyname()]& functions give temporary errors,
 since these are most likely to be caused by DNS lookup problems. The
 &(dnslookup)& router has some options of its own for controlling what happens


@@ -16193,6 +16239,11 @@ This option is obsolete, and retained only for backward compatibility, because
 nowadays the ACL specified by &%acl_smtp_connect%& can also reject incoming
 connections immediately.
 
 nowadays the ACL specified by &%acl_smtp_connect%& can also reject incoming
 connections immediately.
 

+.new
+If the connection is on a TLS-on-connect port then the TCP connection is
+just dropped.  Otherwise, an SMTP error is sent first.
+.wen
+

 The ability to give an immediate rejection (either by this option or using an
 ACL) is provided for use in unusual cases. Many hosts will just try again,
 sometimes without much delay. Normally, it is better to use an ACL to reject
 The ability to give an immediate rejection (either by this option or using an
 ACL) is provided for use in unusual cases. Many hosts will just try again,
 sometimes without much delay. Normally, it is better to use an ACL to reject


@@ -17683,13 +17734,18 @@ This facility is only available on Linux.
 .cindex "banner for SMTP"
 .cindex "welcome banner for SMTP"
 .cindex "customizing" "SMTP banner"
 .cindex "banner for SMTP"
 .cindex "welcome banner for SMTP"
 .cindex "customizing" "SMTP banner"

-This string, which is expanded every time it is used, is output as the initial
+If a connect ACL does not supply a message,
+this string (which is expanded every time it is used) is output as the initial

 positive response to an SMTP connection. The default setting is:
 .code
 smtp_banner = $smtp_active_hostname ESMTP Exim \
   $version_number $tod_full
 .endd
 positive response to an SMTP connection. The default setting is:
 .code
 smtp_banner = $smtp_active_hostname ESMTP Exim \
   $version_number $tod_full
 .endd

-Failure to expand the string causes a panic error. If you want to create a
+.new
+Failure to expand the string causes a panic error;
+a forced fail just closes the connection.
+.wen
+If you want to create a

 multiline response to the initial SMTP connection, use &"\n"& in the string at
 appropriate points, but not at the end. Note that the 220 code is not included
 in this string. Exim adds it automatically (several times in the case of a
 multiline response to the initial SMTP connection, use &"\n"& in the string at
 appropriate points, but not at the end. Note that the 220 code is not included
 in this string. Exim adds it automatically (several times in the case of a


@@ -18431,20 +18487,27 @@ prior to the 4.80 release, as Debian used to patch Exim to raise the minimum
 acceptable bound from 1024 to 2048.
 
 
 acceptable bound from 1024 to 2048.
 
 

-.option tls_eccurve main string&!! &`auto`&
+.option tls_eccurve main string list&!! &`auto`&

 .cindex TLS "EC cryptography"
 .cindex TLS "EC cryptography"

-This option selects a EC curve for use by Exim when used with OpenSSL.
-It has no effect when Exim is used with GnuTLS.
+This option selects EC curves for use by Exim when used with OpenSSL.
+It has no effect when Exim is used with GnuTLS
+(the equivalent can be done using a priority string for the
+&%tls_require_ciphers%& option).

 
 

-After expansion it must contain a valid EC curve parameter, such as
-&`prime256v1`&, &`secp384r1`&, or &`P-512`&. Consult your OpenSSL manual
-for valid selections.
+After expansion it must contain
+.new
+one or (only for OpenSSL versiona 1.1.1 onwards) more
+.wen
+EC curve names, such as &`prime256v1`&, &`secp384r1`&, or &`P-521`&.
+Consult your OpenSSL manual for valid curve names.

 
 For OpenSSL versions before (and not including) 1.0.2, the string
 &`auto`& selects &`prime256v1`&. For more recent OpenSSL versions
 &`auto`& tells the library to choose.
 
 
 For OpenSSL versions before (and not including) 1.0.2, the string
 &`auto`& selects &`prime256v1`&. For more recent OpenSSL versions
 &`auto`& tells the library to choose.
 

-If the option expands to an empty string, no EC curves will be enabled.
+.new
+If the option expands to an empty string, the effect is undefined.
+.wen

 
 
 .option tls_ocsp_file main string&!! unset
 
 
 .option tls_ocsp_file main string&!! unset


@@ -22379,7 +22442,7 @@ its removal from incoming messages, so that delivered messages can safely be
 resent to other recipients.
 
 &*Note:*& If used on a transport handling multiple recipients
 resent to other recipients.
 
 &*Note:*& If used on a transport handling multiple recipients

-(the smtp transport unless &%rcpt_max%& is 1, the appendfile, pipe or lmtp
+(the smtp transport unless &%max_rcpt%& is 1, the appendfile, pipe or lmtp

 transport if &%batch_max%& is greater than 1)
 then information about Bcc recipients will be leaked.
 Doing so is generally not advised.
 transport if &%batch_max%& is greater than 1)
 then information about Bcc recipients will be leaked.
 Doing so is generally not advised.


@@ -25607,12 +25670,18 @@ hard failure if required. See also &%hosts_try_auth%&, and chapter
 &<<CHAPSMTPAUTH>>& for details of authentication.
 
 
 &<<CHAPSMTPAUTH>>& for details of authentication.
 
 

-.option hosts_request_ocsp smtp "host list&!!" *
+.option hosts_request_ocsp smtp "host list&!!" "see below"

 .cindex "TLS" "requiring for certain servers"
 Exim will request a Certificate Status on a
 TLS session for any host that matches this list.
 &%tls_verify_certificates%& should also be set for the transport.
 
 .cindex "TLS" "requiring for certain servers"
 Exim will request a Certificate Status on a
 TLS session for any host that matches this list.
 &%tls_verify_certificates%& should also be set for the transport.
 

+.new
+The default is &"**"& if DANE is not in use for the connection,
+or if DANE-TA us used.
+It is empty if DANE-EE is used.
+.wen
+

 .option hosts_require_alpn smtp "host list&!!" unset
 .cindex ALPN "require negotiation in client"
 .cindex TLS ALPN
 .option hosts_require_alpn smtp "host list&!!" unset
 .cindex ALPN "require negotiation in client"
 .cindex TLS ALPN


@@ -25754,12 +25823,22 @@ If this option is set true when the &%protocol%& option is set to &"lmtp"&, the
 string &`IGNOREQUOTA`& is added to RCPT commands, provided that the LMTP server
 has advertised support for IGNOREQUOTA in its response to the LHLO command.
 
 string &`IGNOREQUOTA`& is added to RCPT commands, provided that the LMTP server
 has advertised support for IGNOREQUOTA in its response to the LHLO command.
 

-.option max_rcpt smtp integer 100
+.option max_rcpt smtp integer&!! 100

 .cindex "RCPT" "maximum number of outgoing"
 .cindex "RCPT" "maximum number of outgoing"

-This option limits the number of RCPT commands that are sent in a single
-SMTP message transaction. Each set of addresses is treated independently, and
+This option,
+.new
+after expansion,
+.wen
+limits the number of RCPT commands that are sent in a single
+SMTP message transaction.
+A value setting of zero disables the limit.
+
+.new
+If a constant is given,
+.wen
+each set of addresses is treated independently, and

 so can cause parallel connections to the same host if &%remote_max_parallel%&
 so can cause parallel connections to the same host if &%remote_max_parallel%&

-permits this. A value setting of zero disables the limit.
+permits this.

 
 
 .option message_linelength_limit smtp integer 998
 
 
 .option message_linelength_limit smtp integer 998


@@ -26082,7 +26161,7 @@ If both this option and &%tls_try_verify_hosts%& are unset
 operation is as if this option selected all hosts.
 &*Warning*&: Including a host in &%tls_verify_hosts%& does not require
 that connections use TLS.
 operation is as if this option selected all hosts.
 &*Warning*&: Including a host in &%tls_verify_hosts%& does not require
 that connections use TLS.

-Fallback to in-clear communication will be done unless restricted by 
+Fallback to in-clear communication will be done unless restricted by

 the &%hosts_require_tls%& option.
 
 .option utf8_downconvert smtp integer&!! -1
 the &%hosts_require_tls%& option.
 
 .option utf8_downconvert smtp integer&!! -1


@@ -28223,7 +28302,7 @@ Dovecot 2 POP/IMAP server, which can support a number of authentication methods.
 Note that Dovecot must be configured to use auth-client not auth-userdb.
 If you are using Dovecot to authenticate POP/IMAP clients, it might be helpful
 to use the same mechanisms for SMTP authentication. This is a server
 Note that Dovecot must be configured to use auth-client not auth-userdb.
 If you are using Dovecot to authenticate POP/IMAP clients, it might be helpful
 to use the same mechanisms for SMTP authentication. This is a server

-authenticator only. There is only one option:
+authenticator only. There is only one non-generic option:

 
 .option server_socket dovecot string unset
 
 
 .option server_socket dovecot string unset
 


@@ -28235,6 +28314,7 @@ authenticators for different mechanisms. For example:
 dovecot_plain:
   driver = dovecot
   public_name = PLAIN
 dovecot_plain:
   driver = dovecot
   public_name = PLAIN

+  server_advertise_condition = ${if def:tls_in_cipher}

   server_socket = /var/run/dovecot/auth-client
   server_set_id = $auth1
 
   server_socket = /var/run/dovecot/auth-client
   server_set_id = $auth1
 


@@ -28244,6 +28324,13 @@ dovecot_ntlm:
   server_socket = /var/run/dovecot/auth-client
   server_set_id = $auth1
 .endd
   server_socket = /var/run/dovecot/auth-client
   server_set_id = $auth1
 .endd

+
+.new
+&*Note*&: plaintext authentication methods such as PLAIN and LOGIN
+should not be advertised on cleartext SMTP connections.
+See the discussion in section &<<SECTplain_TLS>>&.
+.wen
+

 If the SMTP connection is encrypted, or if &$sender_host_address$& is equal to
 &$received_ip_address$& (that is, the connection is local), the &"secured"&
 option is passed in the Dovecot authentication command. If, for a TLS
 If the SMTP connection is encrypted, or if &$sender_host_address$& is equal to
 &$received_ip_address$& (that is, the connection is local), the &"secured"&
 option is passed in the Dovecot authentication command. If, for a TLS


@@ -29772,7 +29859,7 @@ connection.
 The client for the connection proposes a set of protocol names, and
 the server responds with a selected one.
 It is not, as of 2021, commonly used for SMTP connections.
 The client for the connection proposes a set of protocol names, and
 the server responds with a selected one.
 It is not, as of 2021, commonly used for SMTP connections.

-However, to guard against misirected or malicious use of web clients
+However, to guard against misdirected or malicious use of web clients

 (which often do use ALPN) against MTA ports, Exim by default check that
 there is no incompatible ALPN specified by a client for a TLS connection.
 If there is, the connection is rejected.
 (which often do use ALPN) against MTA ports, Exim by default check that
 there is no incompatible ALPN specified by a client for a TLS connection.
 If there is, the connection is rejected.


@@ -29782,7 +29869,7 @@ The behaviour of both client and server can be configured using the options
 &%tls_alpn%& and &%hosts_require_alpn%&.
 There are no variables providing observability.
 Some feature-specific logging may appear on denied connections, but this
 &%tls_alpn%& and &%hosts_require_alpn%&.
 There are no variables providing observability.
 Some feature-specific logging may appear on denied connections, but this

-depends on the behavious of the peer
+depends on the behaviour of the peer

 (not all peers can send a feature-specific TLS Alert).
 
 This feature is available when Exim is built with
 (not all peers can send a feature-specific TLS Alert).
 
 This feature is available when Exim is built with


@@ -30474,8 +30561,11 @@ accepted by an &%accept%& verb that has a &%message%& modifier, the contents of
 the message override the banner message that is otherwise specified by the
 &%smtp_banner%& option.
 
 the message override the banner message that is otherwise specified by the
 &%smtp_banner%& option.
 

-For tls-on-connect connections, the ACL is run after the TLS connection
-is accepted (however, &%host_reject_connection%& is tested before).
+.new
+For tls-on-connect connections, the ACL is run before the TLS connection
+is accepted; if the ACL does not accept then the TCP connection is dropped without
+any TLS startup attempt and without any SMTP response being transmitted.
+.wen

 
 
 .subsection "The EHLO/HELO ACL" SECID192
 
 
 .subsection "The EHLO/HELO ACL" SECID192


@@ -31626,7 +31716,7 @@ pretrigger=<&'size'&>    This option specifies a memory buffuer to be used
                      immediate writes to file are done as normal.
 
 trigger=<&'reason'&>     This option selects cause for the pretrigger buffer
                      immediate writes to file are done as normal.
 
 trigger=<&'reason'&>     This option selects cause for the pretrigger buffer

-                     see above) to be copied to file.  A reason of $*now*
+                     see above) to be copied to file.  A reason of &*now*&

                      take effect immediately; one of &*paniclog*& triggers
                      on a write to the panic log.
 .endd
                      take effect immediately; one of &*paniclog*& triggers
                      on a write to the panic log.
 .endd


@@ -38880,7 +38970,7 @@ selection marked by asterisks:
 .irow &`etrn`&                         *       "ETRN commands"
 .irow &`host_lookup_failed`&           *       "as it says"
 .irow &`ident_timeout`&                        &nbsp; "timeout for ident connection"
 .irow &`etrn`&                         *       "ETRN commands"
 .irow &`host_lookup_failed`&           *       "as it says"
 .irow &`ident_timeout`&                        &nbsp; "timeout for ident connection"

-.irow &`incoming_interface`&           &nbsp; "local interface on <= and => lines"
+.irow &`incoming_interface`&           &nbsp; "local interface &amp; port on <= and => lines"

 .irow &`incoming_port`&                        &nbsp; "remote port on <= lines"
 .irow &`lost_incoming_connection`&     *       "as it says (includes timeouts)"
 .irow &`millisec`&                     &nbsp; "millisecond timestamps and RT,QT,DT,D times"
 .irow &`incoming_port`&                        &nbsp; "remote port on <= lines"
 .irow &`lost_incoming_connection`&     *       "as it says (includes timeouts)"
 .irow &`millisec`&                     &nbsp; "millisecond timestamps and RT,QT,DT,D times"


@@ -42216,7 +42306,7 @@ the DATA acl.
 .subsection ACL SSECDMARCACL
 .cindex DMARC "ACL condition"
 
 .subsection ACL SSECDMARCACL
 .cindex DMARC "ACL condition"
 

-DMARC checks cam be run on incoming SMTP  messages by using the
+DMARC checks can be run on incoming SMTP  messages by using the

 &"dmarc_status"& ACL condition in the DATA ACL.  You are required to
 call the &"spf"& condition first in the ACLs, then the &"dmarc_status"&
 condition.  Putting this condition in the ACLs is required in order
 &"dmarc_status"& ACL condition in the DATA ACL.  You are required to
 call the &"spf"& condition first in the ACLs, then the &"dmarc_status"&
 condition.  Putting this condition in the ACLs is required in order