Testsuite: bounce_message_file and warn_message_file

[exim.git] / doc / doc-docbook / spec.xfpt

diff --git a/doc/doc-docbook/spec.xfpt b/doc/doc-docbook/spec.xfpt
index bfe59fcccbcb746dcbcea25149294ba2f0d400f6..edd745174ab25a82a9df238f021e30a69bd3dbad 100644 (file)
--- a/doc/doc-docbook/spec.xfpt
+++ b/doc/doc-docbook/spec.xfpt
@@ -52,7 +52,7 @@
 .set I   "    "
 
 .macro copyyear
 .set I   "    "
 
 .macro copyyear

-2019
+2020

 .endmacro
 
 . /////////////////////////////////////////////////////////////////////////////
 .endmacro
 
 . /////////////////////////////////////////////////////////////////////////////


@@ -6777,13 +6777,34 @@ absolute
 directory path; this is searched for an entry
 whose name is the key by calling the &[lstat()]& function.
 The key may not
 directory path; this is searched for an entry
 whose name is the key by calling the &[lstat()]& function.
 The key may not

-contain any forward slash characters. If &[lstat()]& succeeds, the result of
-the lookup is the name of the entry, which may be a file, directory,
-symbolic link, or any other kind of directory entry.
+contain any forward slash characters.
+If &[lstat()]& succeeds then so does the lookup.

 .new
 .cindex "tainted data" "dsearch result"
 .new
 .cindex "tainted data" "dsearch result"

-It is regarded as untainted.
+The result is regarded as untainted.
+
+Options for the lookup can be given by appending them after the word "dsearch",
+separated by a comma.  Options, if present, are a comma-separated list having
+each element starting with a tag name and an equals.
+
+Two options are supported, for the return value and for filtering match
+candidates.
+The "ret" option requests an alternate result value of
+the entire path for the entry. Example:
+.code
+${lookup {passwd} dsearch,ret=full {/etc}}
+.endd
+The default result is just the requested entry.
+The "filter" option requests that only directory entries of a given type
+are matched. The match value is one of "file", "dir" or "subdir" (the latter
+not matching "." or ".."). Example:
+.code
+${lookup {passwd} dsearch,filter=file {/etc}}
+.endd
+The default matching is for any entry type, including directories
+and symlinks.

 .wen
 .wen

+

 An example of how this
 lookup can be used to support virtual domains is given in section
 &<<SECTvirtualdomains>>&.
 An example of how this
 lookup can be used to support virtual domains is given in section
 &<<SECTvirtualdomains>>&.


@@ -7995,12 +8016,14 @@ The &%quote_redis%& expansion operator
 escapes whitespace and backslash characters with a backslash.
 
 .section "Specifying the server in the query" "SECTspeserque"
 escapes whitespace and backslash characters with a backslash.
 
 .section "Specifying the server in the query" "SECTspeserque"

+.new

 For MySQL, PostgreSQL and Redis lookups (but not currently for Oracle and InterBase),
 it is possible to specify a list of servers with an individual query. This is
 For MySQL, PostgreSQL and Redis lookups (but not currently for Oracle and InterBase),
 it is possible to specify a list of servers with an individual query. This is

-done by starting the query with
+done by appending a comma-separated option to the query type:

 .display
 .display

-&`servers=`&&'server1:server2:server3:...'&&`;`&

 .endd
 .endd

+&`,servers=`&&'server1:server2:server3:...'&
+.wen

 Each item in the list may take one of two forms:
 .olist
 If it contains no slashes it is assumed to be just a host name. The appropriate
 Each item in the list may take one of two forms:
 .olist
 If it contains no slashes it is assumed to be just a host name. The appropriate


@@ -8025,14 +8048,25 @@ mysql_servers = slave1/db/name/pw:\
 .endd
 In an updating lookup, you could then write:
 .code
 .endd
 In an updating lookup, you could then write:
 .code

-${lookup mysql{servers=master; UPDATE ...} }
+${lookup mysql,servers=master {UPDATE ...} }

 .endd
 That query would then be sent only to the master server. If, on the other hand,
 the master is not to be used for reading, and so is not present in the global
 option, you can still update it by a query of this form:
 .code
 .endd
 That query would then be sent only to the master server. If, on the other hand,
 the master is not to be used for reading, and so is not present in the global
 option, you can still update it by a query of this form:
 .code

-${lookup pgsql{servers=master/db/name/pw; UPDATE ...} }
+${lookup pgsql,servers=master/db/name/pw {UPDATE ...} }
+.endd
+
+.new
+An older syntax places the servers speciification before the qury,
+semicolon separated:
+.code
+${lookup mysql{servers=master; UPDATE ...} }

 .endd
 .endd

+The new version avoids potential issues with tainted
+arguments in the query, for explicit expansion.
+&*Note*&: server specifications in list-style lookups are still problematic.
+.wen

 
 
 .section "Special MySQL features" "SECID73"
 
 
 .section "Special MySQL features" "SECID73"


@@ -8086,10 +8120,10 @@ daemon as in the other SQL databases.
 
 .new
 .oindex &%sqlite_dbfile%&
 
 .new
 .oindex &%sqlite_dbfile%&

-The preferred way of specifying the file is by using the 
+The preferred way of specifying the file is by using the

 &%sqlite_dbfile%& option, set to
 &%sqlite_dbfile%& option, set to

-.wen

 an absolute path.
 an absolute path.

+.wen

 A deprecated method is available, prefixing the query with the filename
 separated by white space.
 This means that the path name cannot contain white space.
 A deprecated method is available, prefixing the query with the filename
 separated by white space.
 This means that the path name cannot contain white space.


@@ -8098,6 +8132,7 @@ It also means that the query cannot use any tainted values, as that taints
 the entire query including the filename - resulting in a refusal to open
 the file.
 
 the entire query including the filename - resulting in a refusal to open
 the file.
 

+.new

 Here is a lookup expansion example:
 .code
 sqlite_dbfile = /some/thing/sqlitedb
 Here is a lookup expansion example:
 .code
 sqlite_dbfile = /some/thing/sqlitedb


@@ -8109,6 +8144,7 @@ In a list, the syntax is similar. For example:
 domainlist relay_to_domains = sqlite;\
    select * from relays where ip='$sender_host_address';
 .endd
 domainlist relay_to_domains = sqlite;\
    select * from relays where ip='$sender_host_address';
 .endd

+.wen

 The only character affected by the &%quote_sqlite%& operator is a single
 quote, which it doubles.
 
 The only character affected by the &%quote_sqlite%& operator is a single
 quote, which it doubles.
 


@@ -8596,6 +8632,14 @@ whether or not the query succeeds. However, when a lookup is used for the
 &%domains%& option on a router, the data is preserved in the &$domain_data$&
 variable and can be referred to in other options.
 .next
 &%domains%& option on a router, the data is preserved in the &$domain_data$&
 variable and can be referred to in other options.
 .next

+.new
+If the pattern starts with the name of a lookup type
+of either kind (single-key or query-style) it may be
+followed by a command and options,
+The options are lookup-type specific and consist of a comma-separated list.
+Each item starts with a tag and and equals "=".
+.wen
+.next

 .cindex "domain list" "matching literal domain name"
 If none of the above cases apply, a caseless textual comparison is made
 between the pattern and the domain.
 .cindex "domain list" "matching literal domain name"
 If none of the above cases apply, a caseless textual comparison is made
 between the pattern and the domain.


@@ -10241,21 +10285,37 @@ ${readsocket{/socket/name}{request string}{3s}}
 .endd
 
 The third argument is a list of options, of which the first element is the timeout
 .endd
 
 The third argument is a list of options, of which the first element is the timeout

-and must be present if the argument is given.
+and must be present if any options are given.

 Further elements are options of form &'name=value'&.
 Further elements are options of form &'name=value'&.

-Two option types is currently recognised: shutdown and tls.
-The first defines whether (the default)
-or not a shutdown is done on the connection after sending the request.
-Example, to not do so (preferred, eg. by some webservers):
+Example:

 .code
 ${readsocket{/socket/name}{request string}{3s:shutdown=no}}
 .endd
 .code
 ${readsocket{/socket/name}{request string}{3s:shutdown=no}}
 .endd

-The second, tls, controls the use of TLS on the connection.  Example:
-.code
-${readsocket{/socket/name}{request string}{3s:tls=yes}}
-.endd
-The default is to not use TLS.
+
+.new
+The following option names are recognised:
+.ilist
+&*cache*&
+Defines if the result data can be cached for use by a later identical
+request in the same process.
+Values are &"yes"& or &"no"& (the default).
+If not, all cached results for this connection specification
+will be invalidated.
+
+.next
+&*shutdown*&
+Defines whether or not a write-shutdown is done on the connection after
+sending the request. Values are &"yes"& (the default) or &"no"&
+(preferred, eg. by some webservers).
+
+.next
+&*tls*&
+Controls the use of TLS on the connection.
+Values are &"yes"& or &"no"& (the default).

 If it is enabled, a shutdown as descripbed above is never done.
 If it is enabled, a shutdown as descripbed above is never done.

+.endlist
+.wen
+

 
 A fourth argument allows you to change any newlines that are in the data
 that is read, in the same way as for &%readfile%& (see above). This example
 
 A fourth argument allows you to change any newlines that are in the data
 that is read, in the same way as for &%readfile%& (see above). This example


@@ -12281,7 +12341,9 @@ the complete argument of the ETRN command (see section &<<SECTETRN>>&).
 .cindex "tainted data"
 If the origin of the data is an incoming message,
 the result of expanding this variable is tainted.
 .cindex "tainted data"
 If the origin of the data is an incoming message,
 the result of expanding this variable is tainted.

-See also &$domain_verified$&.
+When un untainted version is needed, one should be obtained from
+looking up the value in a local (therefore trusted) database.
+See also &$domain_data$&.

 .wen
 
 
 .wen
 
 


@@ -13246,6 +13308,18 @@ library, by setting:
 dns_dnssec_ok = 1
 .endd
 
 dns_dnssec_ok = 1
 .endd
 

+.new
+In addition, on Linux with glibc 2.31 or newer the resolver library will
+default to stripping out a successful validation status.
+This will break a previously working Exim installation.
+Provided that you do trust the resolver (ie, is on localhost) you can tell
+glibc to pass through any successful validation with a new option in
+&_/etc/resolv.conf_&:
+.code
+options trust-ad
+.endd
+.wen
+

 Exim does not perform DNSSEC validation itself, instead leaving that to a
 validating resolver (e.g. unbound, or bind with suitable configuration).
 
 Exim does not perform DNSSEC validation itself, instead leaving that to a
 validating resolver (e.g. unbound, or bind with suitable configuration).
 


@@ -14518,6 +14592,7 @@ See also the &'Policy controls'& section above.
 .table2
 .row &%dkim_verify_hashes%&          "DKIM hash methods accepted for signatures"
 .row &%dkim_verify_keytypes%&        "DKIM key types accepted for signatures"
 .table2
 .row &%dkim_verify_hashes%&          "DKIM hash methods accepted for signatures"
 .row &%dkim_verify_keytypes%&        "DKIM key types accepted for signatures"

+.row &%dkim_verify_min_keysizes%&    "DKIM key sizes accepted for signatures"

 .row &%dkim_verify_signers%&         "DKIM domains for which DKIM ACL is run"
 .row &%host_lookup%&                 "host name looked up for these hosts"
 .row &%host_lookup_order%&           "order of DNS and local name lookups"
 .row &%dkim_verify_signers%&         "DKIM domains for which DKIM ACL is run"
 .row &%host_lookup%&                 "host name looked up for these hosts"
 .row &%host_lookup_order%&           "order of DNS and local name lookups"


@@ -15292,6 +15367,16 @@ This option gives a list of key types which are acceptable in signatures,
 and an order of processing.
 Signatures with algorithms not in the list will be ignored.
 
 and an order of processing.
 Signatures with algorithms not in the list will be ignored.
 

+
+.new
+.option dkim_verify_min_keysizes main "string list" "rsa=1024 ed25519=250"
+This option gives a list of key sizes which are acceptable in signatures.
+The list is keyed by the algorithm type for the key; the values are in bits.
+Signatures with keys smaller than given by this option will fail verification.
+
+The default enforces the RFC 8301 minimum key size for RSA signatures.
+.wen
+

 .option dkim_verify_minimal main boolean false
 If set to true, verification of signatures will terminate after the
 first success.
 .option dkim_verify_minimal main boolean false
 If set to true, verification of signatures will terminate after the
 first success.


@@ -15374,6 +15459,18 @@ default. A value of 0 coerces DNSSEC off, a value of 1 coerces DNSSEC on.
 
 If the resolver library does not support DNSSEC then this option has no effect.
 
 
 If the resolver library does not support DNSSEC then this option has no effect.
 

+.new
+On Linux with glibc 2.31 or newer this is insufficient, the resolver library
+will default to stripping out a successful validation status.
+This will break a previously working Exim installation.
+Provided that you do trust the resolver (ie, is on localhost) you can tell
+glibc to pass through any successful validation with a new option in
+&_/etc/resolv.conf_&:
+.code
+options trust-ad
+.endd
+.wen
+

 
 .option dns_ipv4_lookup main "domain list&!!" unset
 .cindex "IPv6" "DNS lookup for AAAA records"
 
 .option dns_ipv4_lookup main "domain list&!!" unset
 .cindex "IPv6" "DNS lookup for AAAA records"


@@ -16911,7 +17008,7 @@ received_header_text = Received: \
         ${if def:sender_helo_name {(helo=$sender_helo_name)\n\t}}}}\
   by $primary_hostname \
   ${if def:received_protocol {with $received_protocol }}\
         ${if def:sender_helo_name {(helo=$sender_helo_name)\n\t}}}}\
   by $primary_hostname \
   ${if def:received_protocol {with $received_protocol }}\

-  ${if def:tls_ver           { ($tls_ver)}}\
+  ${if def:tls_in_ver        { ($tls_in_ver)}}\

   ${if def:tls_in_cipher_std { tls $tls_in_cipher_std\n\t}}\
   (Exim $version_number)\n\t\
   ${if def:sender_address \
   ${if def:tls_in_cipher_std { tls $tls_in_cipher_std\n\t}}\
   (Exim $version_number)\n\t\
   ${if def:sender_address \


@@ -16920,7 +17017,8 @@ received_header_text = Received: \
   ${if def:received_for {\n\tfor $received_for}}
 .endd
 
   ${if def:received_for {\n\tfor $received_for}}
 .endd
 

-The reference to the TLS cipher is omitted when Exim is built without TLS
+The references to the TLS version and cipher are
+omitted when Exim is built without TLS

 support. The use of conditional expansions ensures that this works for both
 locally generated messages and messages received from remote hosts, giving
 header lines such as the following:
 support. The use of conditional expansions ensures that this works for both
 locally generated messages and messages received from remote hosts, giving
 header lines such as the following:


@@ -19086,7 +19184,7 @@ but the user is specified symbolically, the gid associated with the uid is
 used. For example:
 .code
 require_files = mail:/some/file
 used. For example:
 .code
 require_files = mail:/some/file

-require_files = $local_part:$home/.procmailrc
+require_files = $local_part_verified:$home/.procmailrc

 .endd
 If a user or group name in a &%require_files%& list does not exist, the
 &%require_files%& condition fails.
 .endd
 If a user or group name in a &%require_files%& list does not exist, the
 &%require_files%& condition fails.


@@ -21717,7 +21815,7 @@ local_users:
 # This transport overrides the group
 group_delivery:
   driver = appendfile
 # This transport overrides the group
 group_delivery:
   driver = appendfile

-  file = /var/spool/mail/$local_part
+  file = /var/spool/mail/$local_part_verified

   group = mail
 .endd
 If &%user%& is set for a transport, its value overrides what is set in the
   group = mail
 .endd
 If &%user%& is set for a transport, its value overrides what is set in the


@@ -22552,7 +22650,7 @@ is used as a result of a &"keep"& action in the filter. This example shows one
 way of handling this requirement:
 .code
 file = ${if eq{$address_file}{inbox} \
 way of handling this requirement:
 .code
 file = ${if eq{$address_file}{inbox} \

-            {/var/mail/$local_part} \
+            {/var/mail/$local_part_verified} \

             {${if eq{${substr_0_1:$address_file}}{/} \
                   {$address_file} \
                   {$home/mail/$address_file} \
             {${if eq{${substr_0_1:$address_file}}{/} \
                   {$address_file} \
                   {$home/mail/$address_file} \


@@ -22733,8 +22831,8 @@ The string value is expanded for each delivery, and must yield an absolute
 path. The most common settings of this option are variations on one of these
 examples:
 .code
 path. The most common settings of this option are variations on one of these
 examples:
 .code

-file = /var/spool/mail/$local_part
-file = /home/$local_part/inbox
+file = /var/spool/mail/$local_part_verified
+file = /home/$local_part_verified/inbox

 file = $home/inbox
 .endd
 .cindex "&""sticky""& bit"
 file = $home/inbox
 .endd
 .cindex "&""sticky""& bit"


@@ -23490,7 +23588,7 @@ and directories in a maildir mailbox, including subdirectories for maildir++
 folders. Consider this example:
 .code
 maildir_format = true
 folders. Consider this example:
 .code
 maildir_format = true

-directory = /var/mail/$local_part\
+directory = /var/mail/$local_part_verified\

            ${if eq{$local_part_suffix}{}{}\
            {/.${substr_1:$local_part_suffix}}}
 maildirfolder_create_regex = /\.[^/]+$
            ${if eq{$local_part_suffix}{}{}\
            {/.${substr_1:$local_part_suffix}}}
 maildirfolder_create_regex = /\.[^/]+$


@@ -35042,7 +35140,7 @@ central_filter:
   check_local_user
   driver = redirect
   domains = +local_domains
   check_local_user
   driver = redirect
   domains = +local_domains

-  file = /central/filters/$local_part
+  file = /central/filters/$local_part_verified

   no_verify
   allow_filter
   allow_freeze
   no_verify
   allow_filter
   allow_freeze


@@ -36557,10 +36655,10 @@ lists in a separate domain from normal mail. For example:
 lists:
   driver = redirect
   domains = lists.example
 lists:
   driver = redirect
   domains = lists.example

-  file = /usr/lists/$local_part
+  file = ${lookup {$local_part} dsearch,ret=full {/usr/lists}}

   forbid_pipe
   forbid_file
   forbid_pipe
   forbid_file

-  errors_to = $local_part-request@lists.example
+  errors_to = ${quote_local_part:$local_part-request}@lists.example

   no_more
 .endd
 This router is skipped for domains other than &'lists.example'&. For addresses
   no_more
 .endd
 This router is skipped for domains other than &'lists.example'&. For addresses


@@ -36648,7 +36746,8 @@ lists_request:
   driver = redirect
   domains = lists.example
   local_part_suffix = -request
   driver = redirect
   domains = lists.example
   local_part_suffix = -request

-  file = /usr/lists/$local_part$local_part_suffix
+  local_parts = ${lookup {$local_part} dsearch,filter=file {/usr/lists}}
+  file = /usr/lists/${local_part_data}-request

   no_more
 
 lists_post:
   no_more
 
 lists_post:


@@ -36656,10 +36755,10 @@ lists_post:
   domains = lists.example
   senders = ${if exists {/usr/lists/$local_part}\
              {lsearch;/usr/lists/$local_part}{*}}
   domains = lists.example
   senders = ${if exists {/usr/lists/$local_part}\
              {lsearch;/usr/lists/$local_part}{*}}

-  file = /usr/lists/$local_part
+  file = ${lookup {$local_part} dsearch,ret=full {/usr/lists}}

   forbid_pipe
   forbid_file
   forbid_pipe
   forbid_file

-  errors_to = $local_part-request@lists.example
+  errors_to = ${quote_local_part:$local_part-request}@lists.example

   no_more
 
 lists_closed:
   no_more
 
 lists_closed:


@@ -36717,7 +36816,7 @@ verp_smtp:
   max_rcpt = 1
   return_path = \
     ${if match {$return_path}{^(.+?)-request@your.dom.example\$}\
   max_rcpt = 1
   return_path = \
     ${if match {$return_path}{^(.+?)-request@your.dom.example\$}\

-      {$1-request+$local_part=$domain@your.dom.example}fail}
+      {${quote_local_part:$1-request+$local_part=$domain}@your.dom.example}fail}

 .endd
 This has the effect of rewriting the return path (envelope sender) on outgoing
 SMTP messages, if the local part of the original return path ends in
 .endd
 This has the effect of rewriting the return path (envelope sender) on outgoing
 SMTP messages, if the local part of the original return path ends in


@@ -36768,7 +36867,7 @@ verp_dnslookup:
   transport = remote_smtp
   errors_to = \
     ${if match {$return_path}{^(.+?)-request@your.dom.example\$}}
   transport = remote_smtp
   errors_to = \
     ${if match {$return_path}{^(.+?)-request@your.dom.example\$}}

-     {$1-request+$local_part=$domain@your.dom.example}fail}
+     {${quote_local_part:$1-request+$local_part=$domain}@your.dom.example}fail}

   no_more
 .endd
 Before you start sending out messages with VERPed return paths, you must also
   no_more
 .endd
 Before you start sending out messages with VERPed return paths, you must also


@@ -36856,7 +36955,7 @@ follows:
 .code
 my_mailboxes:
   driver = appendfile
 .code
 my_mailboxes:
   driver = appendfile

-  file = /var/mail/$domain/$local_part
+  file = /var/mail/$domain/$local_part_data

   user = mail
 .endd
 This uses a directory of mailboxes for each domain. The &%user%& setting is
   user = mail
 .endd
 This uses a directory of mailboxes for each domain. The &%user%& setting is


@@ -40382,7 +40481,7 @@ only supports signing with the same canonicalization method for both headers and
 .option dkim_strict smtp string&!! unset
 This  option  defines  how  Exim  behaves  when  signing a message that
 should be signed fails for some reason.  When the expansion evaluates to
 .option dkim_strict smtp string&!! unset
 This  option  defines  how  Exim  behaves  when  signing a message that
 should be signed fails for some reason.  When the expansion evaluates to

-either "1" or "true", Exim will defer. Otherwise Exim will send the message
+either &"1"& or &"true"&, Exim will defer. Otherwise Exim will send the message

 unsigned. You can use the &%$dkim_domain%& and &%$dkim_selector%& expansion
 variables here.
 
 unsigned. You can use the &%$dkim_domain%& and &%$dkim_selector%& expansion
 variables here.
 


@@ -40394,16 +40493,19 @@ in the message signature.
 When unspecified, the header names listed in RFC4871 will be used,
 whether or not each header is present in the message.
 The default list is available for the expansion in the macro
 When unspecified, the header names listed in RFC4871 will be used,
 whether or not each header is present in the message.
 The default list is available for the expansion in the macro

-"_DKIM_SIGN_HEADERS".
+&"_DKIM_SIGN_HEADERS"&
+.new
+and an oversigning variant is in &"_DKIM_OVERSIGN_HEADERS"&.
+.wen

 
 If a name is repeated, multiple headers by that name (or the absence thereof)
 will be signed.  The textually later headers in the headers part of the
 message are signed first, if there are multiples.
 
 
 If a name is repeated, multiple headers by that name (or the absence thereof)
 will be signed.  The textually later headers in the headers part of the
 message are signed first, if there are multiples.
 

-A name can be prefixed with either an '=' or a '+' character.
-If an '=' prefix is used, all headers that are present with this name
+A name can be prefixed with either an &"="& or a &"+"& character.
+If an &"="& prefix is used, all headers that are present with this name

 will be signed.
 will be signed.

-If a '+' prefix if used, all headers that are present with this name
+If a &"+"& prefix if used, all headers that are present with this name

 will be signed, and one signature added for a missing header with the
 name will be appended.
 
 will be signed, and one signature added for a missing header with the
 name will be appended.
 


@@ -40645,6 +40747,10 @@ Notes from the key record (tag n=).
 
 .vitem &%$dkim_key_length%&
 Number of bits in the key.
 
 .vitem &%$dkim_key_length%&
 Number of bits in the key.

+.new
+Valid only once the key is loaded, which is at the time the header signature
+is verified, which is after the body hash is.
+.wen

 
 Note that RFC 8301 says:
 .code
 
 Note that RFC 8301 says:
 .code


@@ -40652,9 +40758,8 @@ Verifiers MUST NOT consider signatures using RSA keys of
 less than 1024 bits as valid signatures.
 .endd
 
 less than 1024 bits as valid signatures.
 .endd
 

-To enforce this you must have a DKIM ACL which checks this variable
-and overwrites the &$dkim_verify_status$& variable as discussed above.
-As EC keys are much smaller, the check should only do this for RSA keys.
+This is enforced by the default setting for the &%dkim_verify_min_keysizes%&
+option.

 
 .endlist
 
 
 .endlist