readsocket expansion: response caching

[users/heiko/exim.git] / doc / doc-docbook / spec.xfpt

diff --git a/doc/doc-docbook/spec.xfpt b/doc/doc-docbook/spec.xfpt
index cf15942dbfa8d02cfd4623a275260ca83dbff3bc..8702485703c279d78f21cf6e86aa02bd77dcd4d5 100644 (file)
--- a/doc/doc-docbook/spec.xfpt
+++ b/doc/doc-docbook/spec.xfpt
@@ -3871,6 +3871,14 @@ This option is not intended for use by external callers. It is used internally
 by Exim in conjunction with the &%-MC%& option. It signifies that the
 remote host supports the ESMTP &_DSN_& extension.
 
 by Exim in conjunction with the &%-MC%& option. It signifies that the
 remote host supports the ESMTP &_DSN_& extension.
 

+.new
+.vitem &%-MCd%&
+.oindex "&%-MCd%&"
+This option is not intended for use by external callers. It is used internally
+by Exim in conjunction with the &%-d%& option
+to pass on an information string on the purpose of the process.
+.wen
+

 .vitem &%-MCG%&&~<&'queue&~name'&>
 .oindex "&%-MCG%&"
 This option is not intended for use by external callers. It is used internally
 .vitem &%-MCG%&&~<&'queue&~name'&>
 .oindex "&%-MCG%&"
 This option is not intended for use by external callers. It is used internally


@@ -6667,6 +6675,10 @@ Two different types of data lookup are implemented:
 The &'single-key'& type requires the specification of a file in which to look,
 and a single key to search for. The key must be a non-empty string for the
 lookup to succeed. The lookup type determines how the file is searched.
 The &'single-key'& type requires the specification of a file in which to look,
 and a single key to search for. The key must be a non-empty string for the
 lookup to succeed. The lookup type determines how the file is searched.

+.new
+.cindex "tainted data" "single-key lookups"
+The file string may not be tainted
+.wen

 .next
 .cindex "query-style lookup" "definition of"
 The &'query-style'& type accepts a generalized database query. No particular
 .next
 .cindex "query-style lookup" "definition of"
 The &'query-style'& type accepts a generalized database query. No particular


@@ -6758,15 +6770,41 @@ by default, but has an option to omit them (see section &<<SECTdbmbuild>>&).
 .next
 .cindex "lookup" "dsearch"
 .cindex "dsearch lookup type"
 .next
 .cindex "lookup" "dsearch"
 .cindex "dsearch lookup type"

-&(dsearch)&: The given file must be a directory; this is searched for an entry
-whose name is the key by calling the &[lstat()]& function. The key may not
-contain any forward slash characters. If &[lstat()]& succeeds, the result of
-the lookup is the name of the entry, which may be a file, directory,
-symbolic link, or any other kind of directory entry.
+&(dsearch)&: The given file must be an
+.new
+absolute
+.wen
+directory path; this is searched for an entry
+whose name is the key by calling the &[lstat()]& function.
+The key may not
+contain any forward slash characters.
+If &[lstat()]& succeeds then so does the lookup.

 .new
 .cindex "tainted data" "dsearch result"
 .new
 .cindex "tainted data" "dsearch result"

-It is regarded as untainted.
+The result is regarded as untainted.
+
+Options for the lookup can be given by appending them after the word "dsearch",
+separated by a comma.  Options, if present, are a comma-separated list having
+each element starting with a tag name and an equals.
+
+Two options are supported, for the return value and for filtering match
+candidates.
+The "ret" option requests an alternate result value of
+the entire path for the entry. Example:
+.code
+${lookup {passwd} dsearch,ret=full {/etc}}
+.endd
+The default result is just the requested entry.
+The "filter" option requests that only directory entries of a given type
+are matched. The match value is one of "file", "dir" or "subdir" (the latter
+not matching "." or ".."). Example:
+.code
+${lookup {passwd} dsearch,filter=file {/etc}}
+.endd
+The default matching is for any entry type, including directories
+and symlinks.

 .wen
 .wen

+

 An example of how this
 lookup can be used to support virtual domains is given in section
 &<<SECTvirtualdomains>>&.
 An example of how this
 lookup can be used to support virtual domains is given in section
 &<<SECTvirtualdomains>>&.


@@ -7012,7 +7050,11 @@ passed to a Redis database. See section &<<SECTsql>>&.
 .next
 .cindex "sqlite lookup type"
 .cindex "lookup" "sqlite"
 .next
 .cindex "sqlite lookup type"
 .cindex "lookup" "sqlite"

-&(sqlite)&: The format of the query is a filename followed by an SQL statement
+&(sqlite)&: The format of the query is
+new
+an optional filename
+.wen
+followed by an SQL statement

 that is passed to an SQLite database. See section &<<SECTsqlite>>&.
 
 .next
 that is passed to an SQLite database. See section &<<SECTsqlite>>&.
 
 .next


@@ -7919,6 +7961,11 @@ If any MySQL, PostgreSQL, Oracle, InterBase or Redis lookups are used, the
 or &%redis_servers%&
 option (as appropriate) must be set to a colon-separated list of server
 information.
 or &%redis_servers%&
 option (as appropriate) must be set to a colon-separated list of server
 information.

+.oindex &%mysql_servers%&
+.oindex &%pgsql_servers%&
+.oindex &%oracle_servers%&
+.oindex &%ibase_servers%&
+.oindex &%redis_servers%&

 (For MySQL and PostgreSQL, the global option need not be set if all
 queries contain their own server information &-- see section
 &<<SECTspeserque>>&.)
 (For MySQL and PostgreSQL, the global option need not be set if all
 queries contain their own server information &-- see section
 &<<SECTspeserque>>&.)


@@ -7969,12 +8016,14 @@ The &%quote_redis%& expansion operator
 escapes whitespace and backslash characters with a backslash.
 
 .section "Specifying the server in the query" "SECTspeserque"
 escapes whitespace and backslash characters with a backslash.
 
 .section "Specifying the server in the query" "SECTspeserque"

+.new

 For MySQL, PostgreSQL and Redis lookups (but not currently for Oracle and InterBase),
 it is possible to specify a list of servers with an individual query. This is
 For MySQL, PostgreSQL and Redis lookups (but not currently for Oracle and InterBase),
 it is possible to specify a list of servers with an individual query. This is

-done by starting the query with
+done by appending a comma-separated option to the query type:

 .display
 .display

-&`servers=`&&'server1:server2:server3:...'&&`;`&

 .endd
 .endd

+&`,servers=`&&'server1:server2:server3:...'&
+.wen

 Each item in the list may take one of two forms:
 .olist
 If it contains no slashes it is assumed to be just a host name. The appropriate
 Each item in the list may take one of two forms:
 .olist
 If it contains no slashes it is assumed to be just a host name. The appropriate


@@ -7999,15 +8048,26 @@ mysql_servers = slave1/db/name/pw:\
 .endd
 In an updating lookup, you could then write:
 .code
 .endd
 In an updating lookup, you could then write:
 .code

-${lookup mysql{servers=master; UPDATE ...} }
+${lookup mysql,servers=master {UPDATE ...} }

 .endd
 That query would then be sent only to the master server. If, on the other hand,
 the master is not to be used for reading, and so is not present in the global
 option, you can still update it by a query of this form:
 .code
 .endd
 That query would then be sent only to the master server. If, on the other hand,
 the master is not to be used for reading, and so is not present in the global
 option, you can still update it by a query of this form:
 .code

-${lookup pgsql{servers=master/db/name/pw; UPDATE ...} }
+${lookup pgsql,servers=master/db/name/pw {UPDATE ...} }

 .endd
 
 .endd
 

+.new
+An older syntax places the servers speciification before the qury,
+semicolon separated:
+.code
+${lookup mysql{servers=master; UPDATE ...} }
+.endd
+The new version avoids potential issues with tainted
+arguments in the query, for explicit expansion.
+&*Note*&: server specifications in list-style lookups are still problematic.
+.wen
+

 
 .section "Special MySQL features" "SECID73"
 For MySQL, an empty host name or the use of &"localhost"& in &%mysql_servers%&
 
 .section "Special MySQL features" "SECID73"
 For MySQL, an empty host name or the use of &"localhost"& in &%mysql_servers%&


@@ -8056,19 +8116,35 @@ affected.
 .cindex "sqlite lookup type"
 SQLite is different to the other SQL lookups because a filename is required in
 addition to the SQL query. An SQLite database is a single file, and there is no
 .cindex "sqlite lookup type"
 SQLite is different to the other SQL lookups because a filename is required in
 addition to the SQL query. An SQLite database is a single file, and there is no

-daemon as in the other SQL databases. The interface to Exim requires the name
-of the file, as an absolute path, to be given at the start of the query. It is
-separated from the query by white space. This means that the path name cannot
-contain white space. Here is a lookup expansion example:
+daemon as in the other SQL databases.
+
+.new
+.oindex &%sqlite_dbfile%&
+The preferred way of specifying the file is by using the
+&%sqlite_dbfile%& option, set to
+an absolute path.
+.wen
+A deprecated method is available, prefixing the query with the filename
+separated by white space.
+This means that the path name cannot contain white space.
+.cindex "tainted data" "sqlite file"
+It also means that the query cannot use any tainted values, as that taints
+the entire query including the filename - resulting in a refusal to open
+the file.
+
+.new
+Here is a lookup expansion example:

 .code
 .code

-${lookup sqlite {/some/thing/sqlitedb \
-  select name from aliases where id='userx';}}
+sqlite_dbfile = /some/thing/sqlitedb
+...
+${lookup sqlite {select name from aliases where id='userx';}}

 .endd
 In a list, the syntax is similar. For example:
 .code
 .endd
 In a list, the syntax is similar. For example:
 .code

-domainlist relay_to_domains = sqlite;/some/thing/sqlitedb \
+domainlist relay_to_domains = sqlite;\

    select * from relays where ip='$sender_host_address';
 .endd
    select * from relays where ip='$sender_host_address';
 .endd

+.wen

 The only character affected by the &%quote_sqlite%& operator is a single
 quote, which it doubles.
 
 The only character affected by the &%quote_sqlite%& operator is a single
 quote, which it doubles.
 


@@ -8556,6 +8632,14 @@ whether or not the query succeeds. However, when a lookup is used for the
 &%domains%& option on a router, the data is preserved in the &$domain_data$&
 variable and can be referred to in other options.
 .next
 &%domains%& option on a router, the data is preserved in the &$domain_data$&
 variable and can be referred to in other options.
 .next

+.new
+If the pattern starts with the name of a lookup type
+of either kind (single-key or query-style) it may be
+followed by a command and options,
+The options are lookup-type specific and consist of a comma-separated list.
+Each item starts with a tag and and equals "=".
+.wen
+.next

 .cindex "domain list" "matching literal domain name"
 If none of the above cases apply, a caseless textual comparison is made
 between the pattern and the domain.
 .cindex "domain list" "matching literal domain name"
 If none of the above cases apply, a caseless textual comparison is made
 between the pattern and the domain.


@@ -10201,21 +10285,37 @@ ${readsocket{/socket/name}{request string}{3s}}
 .endd
 
 The third argument is a list of options, of which the first element is the timeout
 .endd
 
 The third argument is a list of options, of which the first element is the timeout

-and must be present if the argument is given.
+and must be present if any options are given.

 Further elements are options of form &'name=value'&.
 Further elements are options of form &'name=value'&.

-Two option types is currently recognised: shutdown and tls.
-The first defines whether (the default)
-or not a shutdown is done on the connection after sending the request.
-Example, to not do so (preferred, eg. by some webservers):
+Example:

 .code
 ${readsocket{/socket/name}{request string}{3s:shutdown=no}}
 .endd
 .code
 ${readsocket{/socket/name}{request string}{3s:shutdown=no}}
 .endd

-The second, tls, controls the use of TLS on the connection.  Example:
-.code
-${readsocket{/socket/name}{request string}{3s:tls=yes}}
-.endd
-The default is to not use TLS.
+
+.new
+The following option names are recognised:
+.ilist
+&*cache*&
+Defines if the result data can be cached for use by a later identical
+request in the same process.
+Values are &"yes"& or &"no"& (the default).
+If not, all cached results for this connection specification
+will be invalidated.
+
+.next
+&*shutdown*&
+Defines whether or not a write-shutdown is done on the connection after
+sending the request. Values are &"yes"& (the default) or &"no"&
+(preferred, eg. by some webservers).
+
+.next
+&*tls*&
+Controls the use of TLS on the connection.
+Values are &"yes"& or &"no"& (the default).

 If it is enabled, a shutdown as descripbed above is never done.
 If it is enabled, a shutdown as descripbed above is never done.

+.endlist
+.wen
+

 
 A fourth argument allows you to change any newlines that are in the data
 that is read, in the same way as for &%readfile%& (see above). This example
 
 A fourth argument allows you to change any newlines that are in the data
 that is read, in the same way as for &%readfile%& (see above). This example


@@ -13206,6 +13306,18 @@ library, by setting:
 dns_dnssec_ok = 1
 .endd
 
 dns_dnssec_ok = 1
 .endd
 

+.new
+In addition, on Linux with glibc 2.31 or newer the resolver library will
+default to stripping out a successful validation status.
+This will break a previously working Exim installation.
+Provided that you do trust the resolver (ie, is on localhost) you can tell
+glibc to pass through any successful validation with a new option in
+&_/etc/resolv.conf_&:
+.code
+options trust-ad
+.endd
+.wen
+

 Exim does not perform DNSSEC validation itself, instead leaving that to a
 validating resolver (e.g. unbound, or bind with suitable configuration).
 
 Exim does not perform DNSSEC validation itself, instead leaving that to a
 validating resolver (e.g. unbound, or bind with suitable configuration).
 


@@ -15334,6 +15446,18 @@ default. A value of 0 coerces DNSSEC off, a value of 1 coerces DNSSEC on.
 
 If the resolver library does not support DNSSEC then this option has no effect.
 
 
 If the resolver library does not support DNSSEC then this option has no effect.
 

+.new
+On Linux with glibc 2.31 or newer this is insufficient, the resolver library
+will default to stripping out a successful validation status.
+This will break a previously working Exim installation.
+Provided that you do trust the resolver (ie, is on localhost) you can tell
+glibc to pass through any successful validation with a new option in
+&_/etc/resolv.conf_&:
+.code
+options trust-ad
+.endd
+.wen
+

 
 .option dns_ipv4_lookup main "domain list&!!" unset
 .cindex "IPv6" "DNS lookup for AAAA records"
 
 .option dns_ipv4_lookup main "domain list&!!" unset
 .cindex "IPv6" "DNS lookup for AAAA records"


@@ -16871,7 +16995,7 @@ received_header_text = Received: \
         ${if def:sender_helo_name {(helo=$sender_helo_name)\n\t}}}}\
   by $primary_hostname \
   ${if def:received_protocol {with $received_protocol }}\
         ${if def:sender_helo_name {(helo=$sender_helo_name)\n\t}}}}\
   by $primary_hostname \
   ${if def:received_protocol {with $received_protocol }}\

-  ${if def:tls_ver           { ($tls_ver)}}\
+  ${if def:tls_in_ver        { ($tls_in_ver)}}\

   ${if def:tls_in_cipher_std { tls $tls_in_cipher_std\n\t}}\
   (Exim $version_number)\n\t\
   ${if def:sender_address \
   ${if def:tls_in_cipher_std { tls $tls_in_cipher_std\n\t}}\
   (Exim $version_number)\n\t\
   ${if def:sender_address \


@@ -16880,7 +17004,8 @@ received_header_text = Received: \
   ${if def:received_for {\n\tfor $received_for}}
 .endd
 
   ${if def:received_for {\n\tfor $received_for}}
 .endd
 

-The reference to the TLS cipher is omitted when Exim is built without TLS
+The references to the TLS version and cipher are
+omitted when Exim is built without TLS

 support. The use of conditional expansions ensures that this works for both
 locally generated messages and messages received from remote hosts, giving
 header lines such as the following:
 support. The use of conditional expansions ensures that this works for both
 locally generated messages and messages received from remote hosts, giving
 header lines such as the following:


@@ -40342,7 +40467,7 @@ only supports signing with the same canonicalization method for both headers and
 .option dkim_strict smtp string&!! unset
 This  option  defines  how  Exim  behaves  when  signing a message that
 should be signed fails for some reason.  When the expansion evaluates to
 .option dkim_strict smtp string&!! unset
 This  option  defines  how  Exim  behaves  when  signing a message that
 should be signed fails for some reason.  When the expansion evaluates to

-either "1" or "true", Exim will defer. Otherwise Exim will send the message
+either &"1"& or &"true"&, Exim will defer. Otherwise Exim will send the message

 unsigned. You can use the &%$dkim_domain%& and &%$dkim_selector%& expansion
 variables here.
 
 unsigned. You can use the &%$dkim_domain%& and &%$dkim_selector%& expansion
 variables here.
 


@@ -40354,16 +40479,19 @@ in the message signature.
 When unspecified, the header names listed in RFC4871 will be used,
 whether or not each header is present in the message.
 The default list is available for the expansion in the macro
 When unspecified, the header names listed in RFC4871 will be used,
 whether or not each header is present in the message.
 The default list is available for the expansion in the macro

-"_DKIM_SIGN_HEADERS".
+&"_DKIM_SIGN_HEADERS"&
+.new
+and an oversigning variant is in &"_DKIM_OVERSIGN_HEADERS"&.
+.wen

 
 If a name is repeated, multiple headers by that name (or the absence thereof)
 will be signed.  The textually later headers in the headers part of the
 message are signed first, if there are multiples.
 
 
 If a name is repeated, multiple headers by that name (or the absence thereof)
 will be signed.  The textually later headers in the headers part of the
 message are signed first, if there are multiples.
 

-A name can be prefixed with either an '=' or a '+' character.
-If an '=' prefix is used, all headers that are present with this name
+A name can be prefixed with either an &"="& or a &"+"& character.
+If an &"="& prefix is used, all headers that are present with this name

 will be signed.
 will be signed.

-If a '+' prefix if used, all headers that are present with this name
+If a &"+"& prefix if used, all headers that are present with this name

 will be signed, and one signature added for a missing header with the
 name will be appended.
 
 will be signed, and one signature added for a missing header with the
 name will be appended.