select() -> poll(). Bug 2831

[exim.git] / doc / doc-docbook / spec.xfpt

diff --git a/doc/doc-docbook/spec.xfpt b/doc/doc-docbook/spec.xfpt
index 529861208b2480409878e304b0447a635041444a..a8cd63b192a5c38fe7ebac7be55cbf11f997c694 100644 (file)
--- a/doc/doc-docbook/spec.xfpt
+++ b/doc/doc-docbook/spec.xfpt
@@ -766,17 +766,17 @@ the Exim documentation, &"spool"& is always used in the first sense.
 .chapter "Incorporated code" "CHID2"
 .cindex "incorporated code"
 .cindex "regular expressions" "library"
 .chapter "Incorporated code" "CHID2"
 .cindex "incorporated code"
 .cindex "regular expressions" "library"

-.cindex "PCRE"
+.cindex "PCRE2"

 .cindex "OpenDMARC"
 A number of pieces of external code are included in the Exim distribution.
 
 .ilist
 Regular expressions are supported in the main Exim program and in the
 .cindex "OpenDMARC"
 A number of pieces of external code are included in the Exim distribution.
 
 .ilist
 Regular expressions are supported in the main Exim program and in the

-Exim monitor using the freely-distributable PCRE library, copyright
-© University of Cambridge. The source to PCRE is no longer shipped with
-Exim, so you will need to use the version of PCRE shipped with your system,
+Exim monitor using the freely-distributable PCRE2 library, copyright
+© University of Cambridge. The source to PCRE2 is not longer shipped with
+Exim, so you will need to use the version of PCRE2 shipped with your system,

 or obtain and install the full version of the library from
 or obtain and install the full version of the library from

-&url(ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre).
+&url(https://github.com/PhilipHazel/pcre2/releases).

 .next
 .cindex "cdb" "acknowledgment"
 Support for the cdb (Constant DataBase) lookup method is provided by code
 .next
 .cindex "cdb" "acknowledgment"
 Support for the cdb (Constant DataBase) lookup method is provided by code


@@ -1745,20 +1745,20 @@ overridden if necessary.
 A C99-capable compiler will be required for the build.
 
 
 A C99-capable compiler will be required for the build.
 
 

-.section "PCRE library" "SECTpcre"
-.cindex "PCRE library"
-Exim no longer has an embedded PCRE library as the vast majority of
-modern systems include PCRE as a system library, although you may need to
-install the PCRE package or the PCRE development package for your operating
-system. If your system has a normal PCRE installation the Exim build
+.section "PCRE2 library" "SECTpcre"
+.cindex "PCRE2 library"
+Exim no longer has an embedded regular-expression library as the vast majority of
+modern systems include PCRE2 as a system library, although you may need to
+install the PCRE2 package or the PCRE2 development package for your operating
+system. If your system has a normal PCRE2 installation the Exim build

 process will need no further configuration. If the library or the
 process will need no further configuration. If the library or the

-headers are in an unusual location you will need to either set the PCRE_LIBS
+headers are in an unusual location you will need to either set the PCRE2_LIBS

 and INCLUDE directives appropriately,
 and INCLUDE directives appropriately,

-or set PCRE_CONFIG=yes to use the installed &(pcre-config)& command.
+or set PCRE2_CONFIG=yes to use the installed &(pcre-config)& command.

 If your operating system has no
 If your operating system has no

-PCRE support then you will need to obtain and build the current PCRE
-from &url(ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/).
-More information on PCRE is available at &url(https://www.pcre.org/).
+PCRE2 support then you will need to obtain and build the current PCRE
+from &url(https://github.com/PhilipHazel/pcre2/releases).
+More information on PCRE2 is available at &url(https://www.pcre.org/).

 
 .section "DBM libraries" "SECTdb"
 .cindex "DBM libraries" "discussion of"
 
 .section "DBM libraries" "SECTdb"
 .cindex "DBM libraries" "discussion of"


@@ -2587,6 +2587,25 @@ use of Exim's filtering capabilities, you should make the document entitled
 
 
 
 
 
 

+.section "Running the daemon" SECTdaemonLaunch
+The most common command line for launching the Exim daemon looks like
+.code
+exim -bd -q5m
+.endd
+This starts a daemon which
+.ilist
+listens for incoming smtp connections, launching handler processes for
+each new one
+.next
+starts a queue-runner process every five minutes, to inspect queued messages
+and run delivery attempts on any that have arrived at their retry time
+.endlist
+Should a queue run take longer than the time between queue-runner starts,
+they will run in parallel.
+Numbers of jobs of the various types are subject to policy controls
+defined in the configuration.
+
+

 .section "Upgrading Exim" "SECID36"
 .cindex "upgrading Exim"
 If you are already running Exim on your host, building and installing a new
 .section "Upgrading Exim" "SECID36"
 .cindex "upgrading Exim"
 If you are already running Exim on your host, building and installing a new


@@ -6635,9 +6654,9 @@ Chapter &<<CHAPplaintext>>& covers both.
 .chapter "Regular expressions" "CHAPregexp"
 
 .cindex "regular expressions" "library"
 .chapter "Regular expressions" "CHAPregexp"
 
 .cindex "regular expressions" "library"

-.cindex "PCRE"
+.cindex "PCRE2"

 Exim supports the use of regular expressions in many of its options. It
 Exim supports the use of regular expressions in many of its options. It

-uses the PCRE regular expression library; this provides regular expression
+uses the PCRE2 regular expression library; this provides regular expression

 matching that is compatible with Perl 5. The syntax and semantics of
 regular expressions is discussed in
 online Perl manpages, in
 matching that is compatible with Perl 5. The syntax and semantics of
 regular expressions is discussed in
 online Perl manpages, in


@@ -6649,10 +6668,10 @@ O'Reilly (see &url(http://www.oreilly.com/catalog/regex2/)).
 . --- to the old URL for now.  2018-09-07.
 
 The documentation for the syntax and semantics of the regular expressions that
 . --- to the old URL for now.  2018-09-07.
 
 The documentation for the syntax and semantics of the regular expressions that

-are supported by PCRE is included in the PCRE distribution, and no further
-description is included here. The PCRE functions are called from Exim using
-the default option settings (that is, with no PCRE options set), except that
-the PCRE_CASELESS option is set when the matching is required to be
+are supported by PCRE2 is included in the PCRE2 distribution, and no further
+description is included here. The PCRE2 functions are called from Exim using
+the default option settings (that is, with no PCRE2 options set), except that
+the PCRE2_CASELESS option is set when the matching is required to be

 case-insensitive.
 
 In most cases, when a regular expression is required in an Exim configuration,
 case-insensitive.
 
 In most cases, when a regular expression is required in an Exim configuration,


@@ -11194,7 +11213,8 @@ empty.
 The parsing correctly handles SMTPUTF8 Unicode in the string.
 
 
 The parsing correctly handles SMTPUTF8 Unicode in the string.
 
 

-.vitem &*${mask:*&<&'IP&~address'&>&*/*&<&'bit&~count'&>&*}*&
+.vitem &*${mask:*&<&'IP&~address'&>&*/*&<&'bit&~count'&>&*}*& &&&
+       &*${mask_n:*&<&'IP&~address'&>&*/*&<&'bit&~count'&>&*}*&

 .cindex "masked IP address"
 .cindex "IP address" "masking"
 .cindex "CIDR notation"
 .cindex "masked IP address"
 .cindex "IP address" "masking"
 .cindex "CIDR notation"


@@ -11208,8 +11228,14 @@ the result back to text, with mask appended. For example,
 .code
 ${mask:10.111.131.206/28}
 .endd
 .code
 ${mask:10.111.131.206/28}
 .endd

-returns the string &"10.111.131.192/28"&. Since this operation is expected to
-be mostly used for looking up masked addresses in files, the result for an IPv6
+returns the string &"10.111.131.192/28"&.
+
+Since this operation is expected to
+be mostly used for looking up masked addresses in files, the
+.new
+normal
+.wen
+result for an IPv6

 address uses dots to separate components instead of colons, because colon
 terminates a key string in lsearch files. So, for example,
 .code
 address uses dots to separate components instead of colons, because colon
 terminates a key string in lsearch files. So, for example,
 .code


@@ -11219,6 +11245,10 @@ returns the string
 .code
 3ffe.ffff.836f.0a00.000a.0800.2000.0000/99
 .endd
 .code
 3ffe.ffff.836f.0a00.000a.0800.2000.0000/99
 .endd

+.new
+If the optional form &*mask_n*& is used, IPv6 address result are instead
+returned in normailsed form, using colons and with zero-compression.
+.wen

 Letters in IPv6 addresses are always output in lower case.
 
 
 Letters in IPv6 addresses are always output in lower case.
 
 


@@ -18399,12 +18429,7 @@ larger prime than requested.
 The value of this option is expanded and indicates the source of DH parameters
 to be used by Exim.
 
 The value of this option is expanded and indicates the source of DH parameters
 to be used by Exim.
 

-This option is ignored for GnuTLS version 3.6.0 and later.
-The library manages parameter negotiation internally.
-
-&*Note: The Exim Maintainers strongly recommend,
-for other TLS library versions,
-using a filename with site-generated
+&*Note: The Exim Maintainers strongly recommend using a filename with site-generated

 local DH parameters*&, which has been supported across all versions of Exim.  The
 other specific constants available are a fallback so that even when
 "unconfigured", Exim can offer Perfect Forward Secrecy in older ciphersuites in TLS.
 local DH parameters*&, which has been supported across all versions of Exim.  The
 other specific constants available are a fallback so that even when
 "unconfigured", Exim can offer Perfect Forward Secrecy in older ciphersuites in TLS.


@@ -18629,7 +18654,8 @@ either &%tls_verify_hosts%& or &%tls_try_verify_hosts%& is set and
 Any client that matches &%tls_verify_hosts%& is constrained by
 &%tls_verify_certificates%&. When the client initiates a TLS session, it must
 present one of the listed certificates. If it does not, the connection is
 Any client that matches &%tls_verify_hosts%& is constrained by
 &%tls_verify_certificates%&. When the client initiates a TLS session, it must
 present one of the listed certificates. If it does not, the connection is

-aborted. &*Warning*&: Including a host in &%tls_verify_hosts%& does not require
+aborted.
+&*Warning*&: Including a host in &%tls_verify_hosts%& does not require

 the host to use TLS. It can still send SMTP commands through unencrypted
 connections. Forcing a client to use TLS has to be done separately using an
 ACL to reject inappropriate commands when the connection is not encrypted.
 the host to use TLS. It can still send SMTP commands through unencrypted
 connections. Forcing a client to use TLS has to be done separately using an
 ACL to reject inappropriate commands when the connection is not encrypted.


@@ -26084,6 +26110,10 @@ certificate verification must succeed.
 The &%tls_verify_certificates%& option must also be set.
 If both this option and &%tls_try_verify_hosts%& are unset
 operation is as if this option selected all hosts.
 The &%tls_verify_certificates%& option must also be set.
 If both this option and &%tls_try_verify_hosts%& are unset
 operation is as if this option selected all hosts.

+&*Warning*&: Including a host in &%tls_verify_hosts%& does not require
+that connections use TLS.
+Fallback to in-clear communication will be done unless restricted by 
+the &%hosts_require_tls%& option.

 
 .option utf8_downconvert smtp integer&!! -1
 .cindex utf8 "address downconversion"
 
 .option utf8_downconvert smtp integer&!! -1
 .cindex utf8 "address downconversion"


@@ -32190,6 +32220,14 @@ content-scanning extension, and is available only in the DATA, MIME, and
 non-SMTP ACLs. It causes the incoming message to be scanned for a match with
 any of the regular expressions. For details, see chapter &<<CHAPexiscan>>&.
 
 non-SMTP ACLs. It causes the incoming message to be scanned for a match with
 any of the regular expressions. For details, see chapter &<<CHAPexiscan>>&.
 

+.new
+.vitem &*seen&~=&~*&<&'parameters'&>
+.cindex "&%sseen%& ACL condition"
+This condition can be used to test if a situation has been previously met,
+for example for greylisting.
+Details are given in section &<<SECTseen>>&.
+.wen
+

 .vitem &*sender_domains&~=&~*&<&'domain&~list'&>
 .cindex "&%sender_domains%& ACL condition"
 .cindex "sender" "ACL checking"
 .vitem &*sender_domains&~=&~*&<&'domain&~list'&>
 .cindex "&%sender_domains%& ACL condition"
 .cindex "sender" "ACL checking"


@@ -32912,6 +32950,59 @@ address you should specify alternate list separators for both the outer
        dnslists = <; dnsbl.example.com/<|$acl_m_addrslist
 .endd
 
        dnslists = <; dnsbl.example.com/<|$acl_m_addrslist
 .endd
 

+
+.new
+.section "Previously seen user and hosts" "SECTseen"
+.cindex "&%sseen%& ACL condition"
+.cindex greylisting
+The &%seen%& ACL condition can be used to test whether a
+situation has been previously met.
+It uses a hints database to record a timestamp against a key.
+host. The syntax of the condition is:
+.display
+&`seen =`& <&'time interval'&> &`/`& <&'options'&>
+.endd
+
+For example,
+.code
+defer  seen = -5m / key=${sender_host_address}_$local_part@$domain
+.endd
+in a RCPT ACL will implement simple greylisting.
+
+The parameters for the condition
+are an interval followed, slash-separated, by a list of options.
+The interval is taken as an offset before the current time,
+and used for the test.
+If the interval is preceded by a minus sign then the condition returns
+whether a record is found which is before the test time.
+Otherwise, the condition returns whether one is found which is since the
+test time.
+
+Options are read in order with later ones overriding earlier ones.
+
+The default key is &$sender_host_address$&.
+An explicit key can be set using a &%key=value%& option.
+
+If a &%readonly%& option is given then
+no record create or update is done.
+If a &%write%& option is given then
+a record create or update is always done.
+An update is done if the test is for &"since"&.
+
+Creates and updates are marked with the current time.
+
+Finally, a &"before"& test which succeeds, and for which the record
+is old enough, will be refreshed with a timestamp of the test time.
+This can prevent tidying of the database from removing the entry.
+The interval for this is, by default, 10 days.
+An explicit interval can be set using a
+&%refresh=value%& option.
+
+Note that &"seen"& should be added to the list of hints databases
+for maintenance if this ACL condition is used.
+.wen
+
+

 .section "Rate limiting incoming messages" "SECTratelimiting"
 .cindex "rate limiting" "client sending"
 .cindex "limiting client sending rates"
 .section "Rate limiting incoming messages" "SECTratelimiting"
 .cindex "rate limiting" "client sending"
 .cindex "limiting client sending rates"


@@ -35180,8 +35271,10 @@ discussed below.
 .vitem &*header_line&~*header_last*&
 A pointer to the last of the header lines.
 
 .vitem &*header_line&~*header_last*&
 A pointer to the last of the header lines.
 

-.vitem &*uschar&~*headers_charset*&
+.new
+.vitem &*const&~uschar&~*headers_charset*&

 The value of the &%headers_charset%& configuration option.
 The value of the &%headers_charset%& configuration option.

+.wen

 
 .vitem &*BOOL&~host_checking*&
 This variable is TRUE during a host checking session that is initiated by the
 
 .vitem &*BOOL&~host_checking*&
 This variable is TRUE during a host checking session that is initiated by the


@@ -39724,8 +39817,13 @@ in a transport)
 .section "exim_dumpdb" "SECTdumpdb"
 .cindex "&'exim_dumpdb'&"
 The entire contents of a database are written to the standard output by the
 .section "exim_dumpdb" "SECTdumpdb"
 .cindex "&'exim_dumpdb'&"
 The entire contents of a database are written to the standard output by the

-&'exim_dumpdb'& program, which has no options or arguments other than the
-spool and database names. For example, to dump the retry database:
+&'exim_dumpdb'& program,
+.new
+taking as arguments the spool and database names.
+An option &'-z'& may be given to regest times in UTC;
+otherwise times are in the local timezone.
+.wen
+For example, to dump the retry database:

 .code
 exim_dumpdb /var/spool/exim retry
 .endd
 .code
 exim_dumpdb /var/spool/exim retry
 .endd


@@ -39811,7 +39909,7 @@ databases is likely to keep on increasing.
 .cindex "&'exim_fixdb'&"
 The &'exim_fixdb'& program is a utility for interactively modifying databases.
 Its main use is for testing Exim, but it might also be occasionally useful for
 .cindex "&'exim_fixdb'&"
 The &'exim_fixdb'& program is a utility for interactively modifying databases.
 Its main use is for testing Exim, but it might also be occasionally useful for

-getting round problems in a live system. It has no options, and its interface
+getting round problems in a live system. Its interface

 is somewhat crude. On entry, it prompts for input with a right angle-bracket. A
 key of a database record can then be entered, and the data for that record is
 displayed.
 is somewhat crude. On entry, it prompts for input with a right angle-bracket. A
 key of a database record can then be entered, and the data for that record is
 displayed.


@@ -39828,6 +39926,12 @@ resets the time of the next delivery attempt. Time values are given as a
 sequence of digit pairs for year, month, day, hour, and minute. Colons can be
 used as optional separators.
 
 sequence of digit pairs for year, month, day, hour, and minute. Colons can be
 used as optional separators.
 

+.new
+Both displayed and input times are in the local timezone by default.
+If an option &'-z'& is used on the command line, displayed times
+are in UTC.
+.wen
+

 
 
 
 
 
 


@@ -40598,7 +40702,7 @@ Consider the use of the &%inlisti%& expansion condition instead.
 .cindex "security" "data sources"
 .cindex "security" "regular expressions"
 .cindex "regular expressions" "security"
 .cindex "security" "data sources"
 .cindex "security" "regular expressions"
 .cindex "regular expressions" "security"

-.cindex "PCRE" "security"
+.cindex "PCRE2" "security"

 If configuration data for Exim can come from untrustworthy sources, there
 are some issues to be aware of:
 
 If configuration data for Exim can come from untrustworthy sources, there
 are some issues to be aware of:
 


@@ -40608,7 +40712,7 @@ Use of &%${expand...}%& may provide a path for shell injection attacks.
 Letting untrusted data provide a regular expression is unwise.
 .next
 Using &%${match...}%& to apply a fixed regular expression against untrusted
 Letting untrusted data provide a regular expression is unwise.
 .next
 Using &%${match...}%& to apply a fixed regular expression against untrusted

-data may result in pathological behaviour within PCRE.  Be aware of what
+data may result in pathological behaviour within PCRE2.  Be aware of what

 "backtracking" means and consider options for being more strict with a regular
 expression. Avenues to explore include limiting what can match (avoiding &`.`&
 when &`[a-z0-9]`& or other character class will do), use of atomic grouping and
 "backtracking" means and consider options for being more strict with a regular
 expression. Avenues to explore include limiting what can match (avoiding &`.`&
 when &`[a-z0-9]`& or other character class will do), use of atomic grouping and