Update ${utf8clean }. Bug 1401

[exim.git] / doc / doc-docbook / spec.xfpt

diff --git a/doc/doc-docbook/spec.xfpt b/doc/doc-docbook/spec.xfpt
index e4cd6ed300cbffb6e111711ffc3b2bbdadbee978..f7636f85a3f2b9be758fd0c93dd3aba0b5a39797 100644 (file)
--- a/doc/doc-docbook/spec.xfpt
+++ b/doc/doc-docbook/spec.xfpt
@@ -6840,7 +6840,7 @@ is used on its own as the result. If the lookup does not succeed, the
 &`fail`& keyword causes a &'forced expansion failure'& &-- see section
 &<<SECTforexpfai>>& for an explanation of what this means.
 
 &`fail`& keyword causes a &'forced expansion failure'& &-- see section
 &<<SECTforexpfai>>& for an explanation of what this means.
 

-The supported DNS record types are A, CNAME, MX, NS, PTR, SPF, SRV, and TXT,
+The supported DNS record types are A, CNAME, MX, NS, PTR, SPF, SRV, TLSA and TXT,

 and, when Exim is compiled with IPv6 support, AAAA (and A6 if that is also
 configured). If no type is given, TXT is assumed. When the type is PTR,
 the data can be an IP address, written as normal; inversion and the addition of
 and, when Exim is compiled with IPv6 support, AAAA (and A6 if that is also
 configured). If no type is given, TXT is assumed. When the type is PTR,
 the data can be an IP address, written as normal; inversion and the addition of


@@ -10172,10 +10172,10 @@ number of larger units and output in Exim's normal time format, for example,
 .cindex "&%uc%& expansion item"
 This forces the letters in the string into upper-case.
 
 .cindex "&%uc%& expansion item"
 This forces the letters in the string into upper-case.
 

-.vitem &*${utf8clean*&<&'utf-8 string'&>&*}*&
+.vitem &*${utf8clean:*&<&'string'&>&*}*&

 .cindex "correction of invalid utf-8 sequences in strings"
 .cindex "utf-8" "utf-8 sequences"
 .cindex "correction of invalid utf-8 sequences in strings"
 .cindex "utf-8" "utf-8 sequences"

-.cindex "wrong utf-8"
+.cindex "incorrect utf-8"

 .cindex "expansion" "utf-8 forcing"
 .cindex "&%utf8clean%& expansion item"
 This replaces any invalid utf-8 sequence in the string by the character &`?`&.
 .cindex "expansion" "utf-8 forcing"
 .cindex "&%utf8clean%& expansion item"
 This replaces any invalid utf-8 sequence in the string by the character &`?`&.


@@ -10247,7 +10247,7 @@ If the ACL returns defer the result is a forced-fail.
 .cindex "&%bool%& expansion condition"
 This condition turns a string holding a true or false representation into
 a boolean state.  It parses &"true"&, &"false"&, &"yes"& and &"no"&
 .cindex "&%bool%& expansion condition"
 This condition turns a string holding a true or false representation into
 a boolean state.  It parses &"true"&, &"false"&, &"yes"& and &"no"&

-(case-insensitively); also positive integer numbers map to true if non-zero,
+(case-insensitively); also integer numbers map to true if non-zero,

 false if zero.
 An empty string is treated as false.
 Leading and trailing whitespace is ignored;
 false if zero.
 An empty string is treated as false.
 Leading and trailing whitespace is ignored;


@@ -16747,11 +16747,12 @@ and the discussion in chapter &<<CHAPenvironment>>&.
 
 
 
 
 
 

-.option headers_add routers string&!! unset
+.option headers_add routers list&!! unset

 .cindex "header lines" "adding"
 .cindex "router" "adding header lines"
 .cindex "header lines" "adding"
 .cindex "router" "adding header lines"

-This option specifies a string of text that is expanded at routing time, and
-associated with any addresses that are accepted by the router. However, this
+This option specifies a list of text headers, newline-separated,
+that is associated with any addresses that are accepted by the router.
+Each item is separately expanded, at routing time.  However, this

 option has no effect when an address is just being verified. The way in which
 the text is used to add header lines at transport time is described in section
 &<<SECTheadersaddrem>>&. New header lines are not actually added until the
 option has no effect when an address is just being verified. The way in which
 the text is used to add header lines at transport time is described in section
 &<<SECTheadersaddrem>>&. New header lines are not actually added until the


@@ -16760,8 +16761,8 @@ header lines in string expansions in the transport's configuration do not
 &"see"& the added header lines.
 
 The &%headers_add%& option is expanded after &%errors_to%&, but before
 &"see"& the added header lines.
 
 The &%headers_add%& option is expanded after &%errors_to%&, but before

-&%headers_remove%& and &%transport%&. If the expanded string is empty, or if
-the expansion is forced to fail, the option has no effect. Other expansion
+&%headers_remove%& and &%transport%&. If an item is empty, or if
+an item expansion is forced to fail, the item has no effect. Other expansion

 failures are treated as configuration errors.
 
 Unlike most options, &%headers_add%& can be specified multiple times
 failures are treated as configuration errors.
 
 Unlike most options, &%headers_add%& can be specified multiple times


@@ -16783,11 +16784,12 @@ avoided. The &%repeat_use%& option of the &%redirect%& router may be of help.
 
 
 
 
 
 

-.option headers_remove routers string&!! unset
+.option headers_remove routers list&!! unset

 .cindex "header lines" "removing"
 .cindex "router" "removing header lines"
 .cindex "header lines" "removing"
 .cindex "router" "removing header lines"

-This option specifies a string of text that is expanded at routing time, and
-associated with any addresses that are accepted by the router. However, this
+This option specifies a list of text headers, colon-separated,
+that is associated with any addresses that are accepted by the router.
+Each item is separately expanded, at routing time.  However, this

 option has no effect when an address is just being verified. The way in which
 the text is used to remove header lines at transport time is described in
 section &<<SECTheadersaddrem>>&. Header lines are not actually removed until
 option has no effect when an address is just being verified. The way in which
 the text is used to remove header lines at transport time is described in
 section &<<SECTheadersaddrem>>&. Header lines are not actually removed until


@@ -16796,8 +16798,8 @@ to header lines in string expansions in the transport's configuration still
 &"see"& the original header lines.
 
 The &%headers_remove%& option is expanded after &%errors_to%& and
 &"see"& the original header lines.
 
 The &%headers_remove%& option is expanded after &%errors_to%& and

-&%headers_add%&, but before &%transport%&. If the expansion is forced to fail,
-the option has no effect. Other expansion failures are treated as configuration
+&%headers_add%&, but before &%transport%&. If an item expansion is forced to fail,
+the item has no effect. Other expansion failures are treated as configuration

 errors.
 
 Unlike most options, &%headers_remove%& can be specified multiple times
 errors.
 
 Unlike most options, &%headers_remove%& can be specified multiple times


@@ -19797,10 +19799,11 @@ value that the router supplies, and also overriding any value associated with
 &%user%& (see below).
 
 
 &%user%& (see below).
 
 

-.option headers_add transports string&!! unset
+.option headers_add transports list&!! unset

 .cindex "header lines" "adding in transport"
 .cindex "transport" "header lines; adding"
 .cindex "header lines" "adding in transport"
 .cindex "transport" "header lines; adding"

-This option specifies a string of text that is expanded and added to the header
+This option specifies a list of text headers, newline-separated,
+which are (separately) expanded and added to the header

 portion of a message as it is transported, as described in section
 &<<SECTheadersaddrem>>&. Additional header lines can also be specified by
 routers. If the result of the expansion is an empty string, or if the expansion
 portion of a message as it is transported, as described in section
 &<<SECTheadersaddrem>>&. Additional header lines can also be specified by
 routers. If the result of the expansion is an empty string, or if the expansion


@@ -19821,18 +19824,20 @@ transports, the settings of &%message_prefix%& and &%message_suffix%& should be
 checked, since this option does not automatically suppress them.
 
 
 checked, since this option does not automatically suppress them.
 
 

-.option headers_remove transports string&!! unset
+.option headers_remove transports list&!! unset

 .cindex "header lines" "removing"
 .cindex "transport" "header lines; removing"
 .cindex "header lines" "removing"
 .cindex "transport" "header lines; removing"

-This option specifies a string that is expanded into a list of header names;
+This option specifies a list of header names, colon-separated;

 these headers are omitted from the message as it is transported, as described
 in section &<<SECTheadersaddrem>>&. Header removal can also be specified by
 these headers are omitted from the message as it is transported, as described
 in section &<<SECTheadersaddrem>>&. Header removal can also be specified by

-routers. If the result of the expansion is an empty string, or if the expansion
+routers.
+Each list item is separately expanded.
+If the result of the expansion is an empty string, or if the expansion

 is forced to fail, no action is taken. Other expansion failures are treated as
 errors and cause the delivery to be deferred.
 
 Unlike most options, &%headers_remove%& can be specified multiple times
 is forced to fail, no action is taken. Other expansion failures are treated as
 errors and cause the delivery to be deferred.
 
 Unlike most options, &%headers_remove%& can be specified multiple times

-for a router; all listed headers are added.
+for a router; all listed headers are removed.

 
 
 
 
 
 


@@ -23027,6 +23032,14 @@ unknown state), opens a new one to the same host, and then tries the delivery
 in clear.
 
 
 in clear.
 
 

+.option tls_try_verify_hosts smtp "host list&!! unset
+.cindex "TLS" "server certificate verification"
+.cindex "certificate" "verification of server"
+This option gives a list of hosts for which, on encrypted connections,
+certificate verification will be tried but need not succeed.
+The &%tls_verify_certificates%& option must also be set.
+
+

 .option tls_verify_certificates smtp string&!! unset
 .cindex "TLS" "server certificate verification"
 .cindex "certificate" "verification of server"
 .option tls_verify_certificates smtp string&!! unset
 .cindex "TLS" "server certificate verification"
 .cindex "certificate" "verification of server"


@@ -23041,6 +23054,20 @@ single file if you are using GnuTLS. The values of &$host$& and
 &$host_address$& are set to the name and address of the server during the
 expansion of this option. See chapter &<<CHAPTLS>>& for details of TLS.
 
 &$host_address$& are set to the name and address of the server during the
 expansion of this option. See chapter &<<CHAPTLS>>& for details of TLS.
 

+For back-compatability,
+if neither tls_verify_hosts nor tls_try_verify_hosts are set
+and certificate verification fails the TLS connection is closed.
+
+
+.option tls_verify_hosts smtp "host list&!! unset
+.cindex "TLS" "server certificate verification"
+.cindex "certificate" "verification of server"
+This option gives a list of hosts for which. on encrypted connections,
+certificate verification must succeed.
+The &%tls_verify_certificates%& option must also be set.
+If both this option and &%tls_try_verify_hosts%& are unset
+operation is as if this option selected all hosts.
+

 
 
 
 
 
 


@@ -25942,6 +25969,12 @@ for OpenSSL only (not GnuTLS), a directory, that contains a collection of
 expected server certificates. The client verifies the server's certificate
 against this collection, taking into account any revoked certificates that are
 in the list defined by &%tls_crl%&.
 expected server certificates. The client verifies the server's certificate
 against this collection, taking into account any revoked certificates that are
 in the list defined by &%tls_crl%&.

+Failure to verify fails the TLS connection unless either of the
+&%tls_verify_hosts%& or &%tls_try_verify_hosts%& options are set.
+
+The &%tls_verify_hosts%& and &%tls_try_verify_hosts%& options restrict
+certificate verification to the listed servers.  Verification either must
+or need not succeed respectively.

 
 If
 &%tls_require_ciphers%& is set on the &(smtp)& transport, it must contain a
 
 If
 &%tls_require_ciphers%& is set on the &(smtp)& transport, it must contain a


@@ -27302,8 +27335,12 @@ from one SMTP connection to another.  If a recipient-verify callout connection i
 requested in the same ACL it is held open and used for the data, otherwise one is made
 after the ACL completes.
 
 requested in the same ACL it is held open and used for the data, otherwise one is made
 after the ACL completes.
 

-Note that routers are used in verify mode.  Note also that headers cannot be
+Note that routers are used in verify mode,
+and cannot depend on content of received headers.
+Note also that headers cannot be

 modified by any of the post-data ACLs (DATA, MIME and DKIM).
 modified by any of the post-data ACLs (DATA, MIME and DKIM).

+Headers may be modified by routers (subject to the above) and transports.
+

 Cutthrough delivery is not supported via transport-filters or when DKIM signing
 of outgoing messages is done, because it sends data to the ultimate destination
 before the entire message has been received from the source.
 Cutthrough delivery is not supported via transport-filters or when DKIM signing
 of outgoing messages is done, because it sends data to the ultimate destination
 before the entire message has been received from the source.


@@ -29007,6 +29044,7 @@ router that does not set up hosts routes to an &(smtp)& transport with a
 &%hosts%& setting, the transport's hosts are used. If an &(smtp)& transport has
 &%hosts_override%& set, its hosts are always used, whether or not the router
 supplies a host list.
 &%hosts%& setting, the transport's hosts are used. If an &(smtp)& transport has
 &%hosts_override%& set, its hosts are always used, whether or not the router
 supplies a host list.

+Callouts are only supported on &(smtp)& transports.

 
 The port that is used is taken from the transport, if it is specified and is a
 remote transport. (For routers that do verification only, no transport need be
 
 The port that is used is taken from the transport, if it is specified and is a
 remote transport. (For routers that do verification only, no transport need be


@@ -29810,6 +29848,24 @@ av_scanner = mksd:2
 .endd
 You can safely omit this option (the default value is 1).
 
 .endd
 You can safely omit this option (the default value is 1).
 

+.vitem &%sock%&
+.cindex "virus scanners" "simple socket-connected"
+This is a general-purpose way of talking to simple scanner daemons
+running on the local machine.
+There are four options:
+an address (which may be an IP addres and port, or the path of a Unix socket),
+a commandline to send (may include a single %s which will be replaced with
+the path to the mail file to be scanned),
+an RE to trigger on from the returned data,
+an RE to extract malware_name from the returned data.
+For example:
+.code
+av_scanner = sock:127.0.0.1 6001:%s:(SPAM|VIRUS):(.*)\$
+.endd
+Default for the socket specifier is &_/tmp/malware.sock_&.
+Default for the commandline is &_%s\n_&.
+Both regular-expressions are required.
+

 .vitem &%sophie%&
 .cindex "virus scanners" "Sophos and Sophie"
 Sophie is a daemon that uses Sophos' &%libsavi%& library to scan for viruses.
 .vitem &%sophie%&
 .cindex "virus scanners" "Sophos and Sophie"
 Sophie is a daemon that uses Sophos' &%libsavi%& library to scan for viruses.


@@ -31932,7 +31988,7 @@ they do not affect the values of the variables that refer to header lines.
 the transport cannot refer to the modified header lines, because such
 expansions all occur before the message is actually transported.
 
 the transport cannot refer to the modified header lines, because such
 expansions all occur before the message is actually transported.
 

-For both routers and transports, the result of expanding a &%headers_add%&
+For both routers and transports, the argument of a &%headers_add%&

 option must be in the form of one or more RFC 2822 header lines, separated by
 newlines (coded as &"\n"&). For example:
 .code
 option must be in the form of one or more RFC 2822 header lines, separated by
 newlines (coded as &"\n"&). For example:
 .code


@@ -31942,10 +31998,10 @@ headers_add = X-added-header: added by $primary_hostname\n\
 Exim does not check the syntax of these added header lines.
 
 Multiple &%headers_add%& options for a single router or transport can be
 Exim does not check the syntax of these added header lines.
 
 Multiple &%headers_add%& options for a single router or transport can be

-specified; the values will be concatenated (with a separating newline
-added) before expansion.
+specified; the values will append to a single list of header lines.
+Each header-line is separately expanded.

 
 

-The result of expanding &%headers_remove%& must consist of a colon-separated
+The argument of a &%headers_remove%& option must consist of a colon-separated

 list of header names. This is confusing, because header names themselves are
 often terminated by colons. In this case, the colons are the list separators,
 not part of the names. For example:
 list of header names. This is confusing, because header names themselves are
 often terminated by colons. In this case, the colons are the list separators,
 not part of the names. For example:


@@ -31954,11 +32010,12 @@ headers_remove = return-receipt-to:acknowledge-to
 .endd
 
 Multiple &%headers_remove%& options for a single router or transport can be
 .endd
 
 Multiple &%headers_remove%& options for a single router or transport can be

-specified; the values will be concatenated (with a separating colon
-added) before expansion.
+specified; the arguments will append to a single header-names list.
+Each item is separately expanded.

 
 

-When &%headers_add%& or &%headers_remove%& is specified on a router, its value
-is expanded at routing time, and then associated with all addresses that are
+When &%headers_add%& or &%headers_remove%& is specified on a router,
+items are expanded at routing time,
+and then associated with all addresses that are

 accepted by that router, and also with any new addresses that it generates. If
 an address passes through several routers as a result of aliasing or
 forwarding, the changes are cumulative.
 accepted by that router, and also with any new addresses that it generates. If
 an address passes through several routers as a result of aliasing or
 forwarding, the changes are cumulative.


@@ -34056,6 +34113,7 @@ the following table:
 &`R   `&        on &`<=`& lines: reference for local bounce
 &`    `&        on &`=>`&  &`**`& and &`==`& lines: router name
 &`S   `&        size of message
 &`R   `&        on &`<=`& lines: reference for local bounce
 &`    `&        on &`=>`&  &`**`& and &`==`& lines: router name
 &`S   `&        size of message

+&`SNI `&        server name indication from TLS client hello

 &`ST  `&        shadow transport name
 &`T   `&        on &`<=`& lines: message subject (topic)
 &`    `&        on &`=>`& &`**`& and &`==`& lines: transport name
 &`ST  `&        shadow transport name
 &`T   `&        on &`<=`& lines: message subject (topic)
 &`    `&        on &`=>`& &`**`& and &`==`& lines: transport name