X-Git-Url: https://git.exim.org/users/heiko/exim.git/blobdiff_plain/77b0ec2d0a1f29612f5af6843a92856a45dbaf98..7f69e814219268610c9d5c9b724f64a17a78b2cb:/doc/doc-docbook/spec.xfpt?ds=sidebyside diff --git a/doc/doc-docbook/spec.xfpt b/doc/doc-docbook/spec.xfpt index c53a392a4..22f06e3c1 100644 --- a/doc/doc-docbook/spec.xfpt +++ b/doc/doc-docbook/spec.xfpt @@ -45,7 +45,7 @@ . Update the Copyright year (only) when changing content. . ///////////////////////////////////////////////////////////////////////////// -.set previousversion "4.91" +.set previousversion "4.92" .include ./local_params .set ACL "access control lists (ACLs)" @@ -371,13 +371,11 @@ contributors. .section "Exim documentation" "SECID1" . Keep this example change bar when updating the documentation! -.new .cindex "documentation" This edition of the Exim specification applies to version &version() of Exim. Substantive changes from the &previousversion; edition are marked in some renditions of this document; this paragraph is so marked if the rendition is capable of showing a change indicator. -.wen This document is very much a reference manual; it is not a tutorial. The reader is expected to have some familiarity with the SMTP mail transfer protocol and @@ -3661,14 +3659,12 @@ The &`timestamp`& selector causes the current time to be inserted at the start of all debug output lines. This can be useful when trying to track down delays in processing. -.new .cindex debugging "UTF-8 in" .cindex UTF-8 "in debug output" The &`noutf8`& selector disables the use of UTF-8 line-drawing characters to group related information. When disabled. ascii-art is used instead. Using the &`+all`& option does not set this modifier, -.wen If the &%debug_print%& option is set in any driver, it produces output whenever any debugging is selected, or if &%-v%& is used. @@ -6746,6 +6742,25 @@ lookup types support only literal keys. the implicit key is the host's IP address rather than its name (see section &<>&). .next +.new +.cindex lookup json +.cindex json "lookup type" +.cindex JSON expansions +&(json)&: The given file is a text file with a JSON structure. +An element of the structure is extracted, defined by the search key. +The key is a list of subelement selectors +(colon-separated by default but changeable in the usual way) +which are applied in turn to select smaller and smaller portions +of the JSON structure. +If a selector is numeric, it must apply to a JSON array; the (zero-based) +nunbered array element is selected. +Otherwise it must apply to a JSON object; the named element is selected. +The final resulting element can be a simple JSON type or a JSON object +or array; for the latter two a string-representation os the JSON +is returned. +For elements of type string, the returned value is de-quoted. +.wen +.next .cindex "linear search" .cindex "lookup" "lsearch" .cindex "lsearch lookup type" @@ -9521,8 +9536,9 @@ ${extract{Z}{A=... B=...}{$value} fail } This forces an expansion failure (see section &<>&); {<&'string2'&>} must be present for &"fail"& to be recognized. -.new .vitem "&*${extract json{*&<&'key'&>&*}{*&<&'string1'&>&*}{*&<&'string2'&>&*}&&& + {*&<&'string3'&>&*}}*&" &&& + "&*${extract jsons{*&<&'key'&>&*}{*&<&'string1'&>&*}{*&<&'string2'&>&*}&&& {*&<&'string3'&>&*}}*&" .cindex "expansion" "extracting from JSON object" .cindex JSON expansions @@ -9537,10 +9553,16 @@ The expanded <&'string1'&> must be of the form: The braces, commas and colons, and the quoting of the member name are required; the spaces are optional. Matching of the key against the member names is done case-sensitively. +For the &"json"& variant, +if a returned value is a JSON string, it retains its leading and +trailing quotes. +.new +For the &"jsons"& variant, which is intended for use with JSON strings, the +leading and trailing quotes are removed from the returned value. +.wen . XXX should be a UTF-8 compare The results of matching are handled as above. -.wen .vitem "&*${extract{*&<&'number'&>&*}{*&<&'separators'&>&*}&&& @@ -9574,8 +9596,9 @@ yields &"99"&. Two successive separators mean that the field between them is empty (for example, the fifth field above). -.new -.vitem "&*${extract json{*&<&'number'&>&*}}&&& +.vitem "&*${extract json {*&<&'number'&>&*}}&&& + {*&<&'string1'&>&*}{*&<&'string2'&>&*}{*&<&'string3'&>&*}}*&" &&& + "&*${extract jsons{*&<&'number'&>&*}}&&& {*&<&'string1'&>&*}{*&<&'string2'&>&*}{*&<&'string3'&>&*}}*&" .cindex "expansion" "extracting from JSON array" .cindex JSON expansions @@ -9584,6 +9607,12 @@ apart from leading and trailing white space, which is ignored. Field selection and result handling is as above; there is no choice of field separator. +For the &"json"& variant, +if a returned value is a JSON string, it retains its leading and +trailing quotes. +.new +For the &"jsons"& variant, which is intended for use with JSON strings, the +leading and trailing quotes are removed from the returned value. .wen @@ -10097,14 +10126,12 @@ Example, to not do so (preferred, eg. by some webservers): .code ${readsocket{/socket/name}{request string}{3s:shutdown=no}} .endd -.new The second, tls, controls the use of TLS on the connection. Example: .code ${readsocket{/socket/name}{request string}{3s:tls=yes}} .endd The default is to not use TLS. If it is enabled, a shutdown as descripbed above is never done. -.wen A fourth argument allows you to change any newlines that are in the data that is read, in the same way as for &%readfile%& (see above). This example @@ -11026,7 +11053,6 @@ Case is defined per the system C locale. .cindex "expansion" "utf-8 forcing" .cindex "&%utf8clean%& expansion item" This replaces any invalid utf-8 sequence in the string by the character &`?`&. -.new In versions of Exim before 4.92, this did not correctly do so for a truncated final codepoint's encoding, and the character would be silently dropped. If you must handle detection of this scenario across both sets of Exim behavior, @@ -11039,7 +11065,6 @@ condition = ${if inlist{${utf8clean:${length_1:$local_part}}}{:?}{yes}{no}} .endd (which will false-positive if the first character of the local part is a literal question mark). -.wen .vitem "&*${utf8_domain_to_alabel:*&<&'string'&>&*}*&" &&& "&*${utf8_domain_from_alabel:*&<&'string'&>&*}*&" &&& @@ -11316,6 +11341,25 @@ being processed, to enable these expansion items to be nested. To scan a named list, expand it with the &*listnamed*& operator. +.new +.vitem "&*forall_json{*&<&'a JSON array'&>&*}{*&<&'a condition'&>&*}*&" &&& + "&*forany_json{*&<&'a JSON array'&>&*}{*&<&'a condition'&>&*}*&" &&& + "&*forall_jsons{*&<&'a JSON array'&>&*}{*&<&'a condition'&>&*}*&" &&& + "&*forany_jsons{*&<&'a JSON array'&>&*}{*&<&'a condition'&>&*}*&" +.cindex JSON "iterative conditions" +.cindex JSON expansions +.cindex expansion "&*forall_json*& condition" +.cindex expansion "&*forany_json*& condition" +.cindex expansion "&*forall_jsons*& condition" +.cindex expansion "&*forany_jsons*& condition" +As for the above, except that the first argument must, after expansion, +be a JSON array. +The array separator is not changeable. +For the &"jsons"& variants the elements are expected to be JSON strings +and have their quotes removed before the evaluation of the condition. +.wen + + .vitem &*ge&~{*&<&'string1'&>&*}{*&<&'string2'&>&*}*& &&& &*gei&~{*&<&'string1'&>&*}{*&<&'string2'&>&*}*& @@ -15040,7 +15084,6 @@ This option controls whether or not an IP address, given as a CSA domain, is reversed and looked up in the reverse DNS, as described in more detail in section &<>&. -.new .option dns_cname_loops main integer 1 .cindex DNS "CNAME following" This option controls the following of CNAME chains, needed if the resolver does @@ -15051,7 +15094,6 @@ If you have an ancient one, a value of 10 is likely needed. The default value of one CNAME-follow is needed thanks to the observed return for an MX request, given no MX presence but a CNAME to an A, of the CNAME. -.wen .option dns_dnssec_ok main integer -1 @@ -23581,7 +23623,8 @@ command = /bin/sh -c ${lookup{$local_part}lsearch{/some/file}} .cindex "filter" "transport filter" .vindex "&$pipe_addresses$&" Special handling takes place when an argument consists of precisely the text -&`$pipe_addresses`&. This is not a general expansion variable; the only +&`$pipe_addresses`& (no quotes). +This is not a general expansion variable; the only place this string is recognized is when it appears as an argument for a pipe or transport filter command. It causes each address that is being handled to be inserted in the argument list at that point &'as a separate argument'&. This @@ -24724,10 +24767,8 @@ and if &%protocol%& is set to &"smtps"& the default is &"smtps"&. If the expansion fails, or if a port number cannot be found, delivery is deferred. -.new Note that at least one Linux distribution has been seen failing to put &"smtps"& in its &"/etc/services"& file, resulting is such deferrals. -.wen @@ -24745,11 +24786,9 @@ over a pipe to a local process &-- see chapter &<>&. If this option is set to &"smtps"&, the default value for the &%port%& option changes to &"smtps"&, and the transport initiates TLS immediately after connecting, as an outbound SSL-on-connect, instead of using STARTTLS to upgrade. -.new The Internet standards bodies used to strongly discourage use of this mode, but as of RFC 8314 it is perferred over STARTTLS for message submission (as distinct from MTA-MTA communication). -.wen .option retry_include_ip_address smtp boolean&!! true @@ -24978,7 +25017,6 @@ The &%tls_verify_certificates%& option must also be set. If both this option and &%tls_try_verify_hosts%& are unset operation is as if this option selected all hosts. -.new .option utf8_downconvert smtp integer!! unset .cindex utf8 "address downconversion" .cindex i18n "utf8 address downconversion" @@ -24986,7 +25024,6 @@ If built with internationalization support, this option controls conversion of UTF-8 in message addresses to a-label form. For details see section &<>&. -.wen @@ -26147,6 +26184,7 @@ included by setting AUTH_CRAM_MD5=yes AUTH_CYRUS_SASL=yes AUTH_DOVECOT=yes +AUTH_EXTERNAL=yes AUTH_GSASL=yes AUTH_HEIMDAL_GSSAPI=yes AUTH_PLAINTEXT=yes @@ -26158,15 +26196,20 @@ authentication mechanism (RFC 2195), and the second provides an interface to the Cyrus SASL authentication library. The third is an interface to Dovecot's authentication system, delegating the work via a socket interface. -The fourth provides an interface to the GNU SASL authentication library, which +.new +The fourth provides for negotiation of authentication done via non-SMTP means, +as defined by RFC 4422 Appendix A. +.wen +The fifth provides an interface to the GNU SASL authentication library, which provides mechanisms but typically not data sources. -The fifth provides direct access to Heimdal GSSAPI, geared for Kerberos, but +The sixth provides direct access to Heimdal GSSAPI, geared for Kerberos, but supporting setting a server keytab. -The sixth can be configured to support +The seventh can be configured to support the PLAIN authentication mechanism (RFC 2595) or the LOGIN mechanism, which is -not formally documented, but used by several MUAs. The seventh authenticator +not formally documented, but used by several MUAs. +The eighth authenticator supports Microsoft's &'Secure Password Authentication'& mechanism. -The eighth is an Exim authenticator but not an SMTP one; +The last is an Exim authenticator but not an SMTP one; instead it can use information from a TLS negotiation. The authenticators are configured using the same syntax as other drivers (see @@ -27401,6 +27444,143 @@ msn: +. //////////////////////////////////////////////////////////////////////////// +. //////////////////////////////////////////////////////////////////////////// + +.chapter "The external authenticator" "CHAPexternauth" +.scindex IIDexternauth1 "&(external)& authenticator" +.scindex IIDexternauth2 "authenticators" "&(external)&" +.cindex "authentication" "Client Certificate" +.cindex "authentication" "X509" +.cindex "Certificate-based authentication" +The &(external)& authenticator provides support for +authentication based on non-SMTP information. +The specification is in RFC 4422 Appendix A +(&url(https://tools.ietf.org/html/rfc4422)). +It is only a transport and negotiation mechanism; +the process of authentication is entirely controlled +by the server configuration. + +The client presents an identity in-clear. +It is probably wise for a server to only advertise, +and for clients to only attempt, +this authentication method on a secure (eg. under TLS) connection. + +One possible use, compatible with the +K-9 Mail Andoid client (&url(https://k9mail.github.io/)), +is for using X509 client certificates. + +It thus overlaps in function with the TLS authenticator +(see &<>&) +but is a full SMTP SASL authenticator +rather than being implicit for TLS-connection carried +client certificates only. + +The examples and discussion in this chapter assume that +client-certificate authentication is being done. + +The client must present a certificate, +for which it must have been requested via the +&%tls_verify_hosts%& or &%tls_try_verify_hosts%& main options +(see &<>&). +For authentication to be effective the certificate should be +verifiable against a trust-anchor certificate known to the server. + +.section "External options" "SECTexternsoptions" +.cindex "options" "&(external)& authenticator (server)" +The &(external)& authenticator has two server options: + +.option server_param2 external string&!! unset +.option server_param3 external string&!! unset +.cindex "variables (&$auth1$& &$auth2$& etc)" "in &(external)& authenticator" +These options are expanded before the &%server_condition%& option +and the result are placed in &$auth2$& and &$auth3$& resectively. +If the expansion is forced to fail, authentication fails. Any other expansion +failure causes a temporary error code to be returned. + +They can be used to clarify the coding of a complex &%server_condition%&. + +.section "Using external in a server" "SECTexternserver" +.cindex "AUTH" "in &(external)& authenticator" +.cindex "numerical variables (&$1$& &$2$& etc)" &&& + "in &(external)& authenticator" +.vindex "&$auth1$&, &$auth2$&, etc" +.cindex "base64 encoding" "in &(external)& authenticator" + +When running as a server, &(external)& performs the authentication test by +expanding a string. The data sent by the client with the AUTH command, or in +response to subsequent prompts, is base64 encoded, and so may contain any byte +values when decoded. The decoded value is treated as +an identity for authentication and +placed in the expansion variable &$auth1$&. + +For compatibility with previous releases of Exim, the value is also placed in +the expansion variable &$1$&. However, the use of this +variable for this purpose is now deprecated, as it can lead to confusion in +string expansions that also use them for other things. + +.vindex "&$authenticated_id$&" +Once an identity has been received, +&%server_condition%& is expanded. If the expansion is forced to fail, +authentication fails. Any other expansion failure causes a temporary error code +to be returned. If the result of a successful expansion is an empty string, +&"0"&, &"no"&, or &"false"&, authentication fails. If the result of the +expansion is &"1"&, &"yes"&, or &"true"&, authentication succeeds and the +generic &%server_set_id%& option is expanded and saved in &$authenticated_id$&. +For any other result, a temporary error code is returned, with the expanded +string as the error text. + +Example: +.code +ext_ccert_san_mail: + driver = external + public_name = EXTERNAL + + server_advertise_condition = $tls_in_certificate_verified + server_param2 = ${certextract {subj_altname,mail,>:} \ + {$tls_in_peercert}} + server_condition = ${if forany {$auth2} \ + {eq {$item}{$auth1}}} + server_set_id = $auth1 +.endd +This accepts a client certificate that is verifiable against any +of your configured trust-anchors +(which usually means the full set of public CAs) +and which has a mail-SAN matching the claimed identity sent by the client. + +Note that, up to TLS1.2, the client cert is on the wire in-clear, including the SAN, +The account name is therefore guessable by an opponent. +TLS 1.3 protects both server and client certificates, and is not vulnerable +in this way. +Likewise, a traditional plaintext SMTP AUTH done inside TLS is not. + + +.section "Using external in a client" "SECTexternclient" +.cindex "options" "&(external)& authenticator (client)" +The &(external)& authenticator has one client option: + +.option client_send external string&!! unset +This option is expanded and sent with the AUTH command as the +identity being asserted. + +Example: +.code +ext_ccert: + driver = external + public_name = EXTERNAL + + client_condition = ${if !eq{$tls_out_cipher}{}} + client_send = myaccount@smarthost.example.net +.endd + + +.ecindex IIDexternauth1 +.ecindex IIDexternauth2 + + + + + . //////////////////////////////////////////////////////////////////////////// . //////////////////////////////////////////////////////////////////////////// @@ -27729,9 +27909,7 @@ the size of the generated prime, so it might still be too large. .oindex "&%tls_require_ciphers%&" "OpenSSL" There is a function in the OpenSSL library that can be passed a list of cipher suites before the cipher negotiation takes place. This specifies which ciphers -.new are acceptable for TLS versions prior to 1.3. -.wen The list is colon separated and may contain names like DES-CBC3-SHA. Exim passes the expanded value of &%tls_require_ciphers%& directly to this function call. @@ -27797,7 +27975,6 @@ This example will prefer ECDSA-authenticated ciphers over RSA ones: tls_require_ciphers = ECDSA:RSA:!COMPLEMENTOFDEFAULT .endd -.new For TLS version 1.3 the control available is less fine-grained and Exim does not provide access to it at present. The value of the &%tls_require_ciphers%& option is ignored when @@ -27807,7 +27984,6 @@ As of writing the library default cipher suite list for TLSv1.3 is .code TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256 .endd -.wen .section "Requiring specific ciphers or other parameters in GnuTLS" &&& @@ -28490,7 +28666,6 @@ DANE-TA is commonly used for several services and/or servers, each having a TLSA all of which point to a single TLSA record. DANE-TA and DANE-EE can both be used together. -.new Our recommendation is to use DANE with a certificate from a public CA, because this enables a variety of strategies for remote clients to verify your certificate. @@ -28515,7 +28690,6 @@ The list of requirements is subject to change as best practices evolve. If you're not already using a private CA, or it doesn't meet these requirements, then we encourage you to avoid all these issues and use a public CA such as &url(https://letsencrypt.org/,Let's Encrypt) instead. -.wen The TLSA record should have a Selector field of SPKI(1) and a Matching Type field of SHA2-512(2). @@ -28533,7 +28707,6 @@ are workable for 4th-field hashes. For use with the DANE-TA model, server certificates must have a correct name (SubjectName or SubjectAltName). -.new The Certificate issued by the CA published in the DANE-TA model should be issued using a strong hash algorithm. Exim, and importantly various other MTAs sending to you, will not @@ -28541,7 +28714,6 @@ re-enable hash algorithms which have been disabled by default in TLS libraries. This means no MD5 and no SHA-1. SHA2-256 is the minimum for reliable interoperability (and probably the maximum too, in 2018). -.wen The use of OCSP-stapling should be considered, allowing for fast revocation of certificates (which would otherwise be limited by the DNS TTL on the TLSA records). However, this is likely to only be usable with DANE-TA. NOTE: the @@ -31382,14 +31554,12 @@ rest of the ACL. The &%leaky%& (default) option means that the client's recorded rate is not updated if it is above the limit. The effect of this is that Exim measures the client's average rate of successfully sent email, -.new up to the given limit. This is appropriate if the countermeasure when the condition is true consists of refusing the message, and is generally the better choice if you have clients that retry automatically. If the action when true is anything more complex then this option is likely not what is wanted. -.wen The &%strict%& option means that the client's recorded rate is always updated. The effect of this is that Exim measures the client's average rate @@ -31560,14 +31730,12 @@ connection, HELO, or MAIL). The main use of these variables is expected to be to distinguish between rejections of MAIL and rejections of RCPT in callouts. -.new The above variables may also be set after a &*successful*& address verification to: .ilist &%random%&: A random local-part callout succeeded .endlist -.wen @@ -33165,9 +33333,7 @@ code. The incident is logged on the main and reject logs. .cindex "&[local_scan()]& function" "building Exim to use" To make use of the local scan function feature, you must tell Exim where your function is before building Exim, by setting -.new both HAVE_LOCAL_SCAN and -.wen LOCAL_SCAN_SOURCE in your &_Local/Makefile_&. A recommended place to put it is in the &_Local_& directory, so you might set @@ -37136,7 +37302,6 @@ local port is a random ephemeral port. &%pid%&: The current process id is added to every log line, in square brackets, immediately after the time and date. .next -.new .cindex log pipelining .cindex pipelining "logging outgoing" &%pipelining%&: A field is added to delivery and accept @@ -38965,11 +39130,9 @@ two files contains the final component of its own name as its first line. This is insurance against disk crashes where the directory is lost but the files themselves are recoverable. -.new The file formats may be changed, or new formats added, at any release. Spool files are not intended as an interface to other programs and should not be used as such. -.wen Some people are tempted into editing -D files in order to modify messages. You need to be extremely careful if you do this; it is not recommended and you are @@ -39512,7 +39675,6 @@ If a '+' prefix if used, all headers that are present with this name will be signed, and one signature added for a missing header with the name will be appended. -.new .option dkim_timestamps smtp integer&!! unset This option controls the inclusion of timestamp information in the signature. If not set, no such information will be included. @@ -39522,22 +39684,19 @@ for the expiry tag both creation (t=) and expiry (x=) tags will be included. RFC 6376 lists these tags as RECOMMENDED. -.wen .section "Verifying DKIM signatures in incoming mail" "SECDKIMVFY" .cindex "DKIM" "verification" -.new Verification of DKIM signatures in SMTP incoming email is done for all messages for which an ACL control &%dkim_disable_verify%& has not been set. .cindex authentication "expansion item" Performing verification sets up information used by the &$authresults$& expansion item. -.wen -.new The results of that verification are then made available to the -&%acl_smtp_dkim%& ACL, &new(which can examine and modify them). +The results of that verification are then made available to the +&%acl_smtp_dkim%& ACL, which can examine and modify them. By default, this ACL is called once for each syntactically(!) correct signature in the incoming message. A missing ACL definition defaults to accept. @@ -39545,7 +39704,7 @@ If any ACL call does not accept, the message is not accepted. If a cutthrough delivery was in progress for the message, that is summarily dropped (having wasted the transmission effort). -To evaluate the &new(verification result) in the ACL +To evaluate the verification result in the ACL a large number of expansion variables containing the signature status and its details are set up during the runtime of the ACL. @@ -39699,12 +39858,10 @@ strict enforcement should code the check explicitly. The number of signed body bytes. If zero ("0"), the body is unsigned. If no limit was set by the signer, "9999999999999" is returned. This makes sure that this variable always expands to an integer value. -.new &*Note:*& The presence of the signature tag specifying a signing body length is one possible route to spoofing of valid DKIM signatures. A paranoid implementation might wish to regard signature where this variable shows less than the "no limit" return as being invalid. -.wen .vitem &%$dkim_created%& UNIX timestamp reflecting the date and time when the signature was created. @@ -40192,11 +40349,9 @@ If a value is appended it may be: If mua_wrapper is set, the utf8_downconvert control is initially set to -1. -.new The smtp transport has an option &%utf8_downconvert%&. If set it must expand to one of the three values described above, and it overrides any previously set value. -.wen There is no explicit support for VRFY and EXPN.