TLS: PIPELINING under OpenSSL

[users/heiko/exim.git] / doc / doc-docbook / spec.xfpt

diff --git a/doc/doc-docbook/spec.xfpt b/doc/doc-docbook/spec.xfpt
index c9f841ad0d70aba943c27bfe21fc46cce0306a01..f33b193eb781f3abcbbe2d99fac7ea030b214bda 100644 (file)
--- a/doc/doc-docbook/spec.xfpt
+++ b/doc/doc-docbook/spec.xfpt
@@ -12067,6 +12067,9 @@ when the ACL &%malware%& condition is true (see section &<<SECTscanvirus>>&).
 This variable contains the number of bytes in the longest line that was
 received as part of the message, not counting the line termination
 character(s).
 This variable contains the number of bytes in the longest line that was
 received as part of the message, not counting the line termination
 character(s).

+.new
+It is not valid if the &%spool_files_wireformat%& option is used.
+.wen

 
 .vitem &$message_age$&
 .cindex "message" "age of"
 
 .vitem &$message_age$&
 .cindex "message" "age of"


@@ -12109,6 +12112,12 @@ in bytes. The count starts from the character after the blank line that
 separates the body from the header. Newlines are included in the count. See
 also &$message_size$&, &$body_linecount$&, and &$body_zerocount$&.
 
 separates the body from the header. Newlines are included in the count. See
 also &$message_size$&, &$body_linecount$&, and &$body_zerocount$&.
 

+.new
+If the spool file is wireformat
+(see the &%spool_files_wireformat%& main option)
+the CRLF line-terminators are included in the count.
+.wen
+

 .vitem &$message_exim_id$&
 .vindex "&$message_exim_id$&"
 When a message is being received or delivered, this variable contains the
 .vitem &$message_exim_id$&
 .vindex "&$message_exim_id$&"
 When a message is being received or delivered, this variable contains the


@@ -12159,6 +12168,10 @@ deny message   = Too many lines in message header
 In the MAIL and RCPT ACLs, the value is zero because at that stage the
 message has not yet been received.
 
 In the MAIL and RCPT ACLs, the value is zero because at that stage the
 message has not yet been received.
 

+.new
+This variable is not valid if the &%spool_files_wireformat%& option is used.
+.wen
+

 .vitem &$message_size$&
 .cindex "size" "of message"
 .cindex "message" "size"
 .vitem &$message_size$&
 .cindex "size" "of message"
 .cindex "message" "size"


@@ -13566,6 +13579,7 @@ listed in more than one group.
 .row &%message_body_visible%&        "how much to show in &$message_body$&"
 .row &%mua_wrapper%&                 "run in &""MUA wrapper""& mode"
 .row &%print_topbitchars%&           "top-bit characters are printing"
 .row &%message_body_visible%&        "how much to show in &$message_body$&"
 .row &%mua_wrapper%&                 "run in &""MUA wrapper""& mode"
 .row &%print_topbitchars%&           "top-bit characters are printing"

+.row &%spool_wireformat%&            "use wire-format spool data files when possible"

 .row &%timezone%&                    "force time zone"
 .endtable
 
 .row &%timezone%&                    "force time zone"
 .endtable
 


@@ -13585,6 +13599,7 @@ listed in more than one group.
 .section "Privilege controls" "SECID98"
 .table2
 .row &%admin_groups%&                "groups that are Exim admin users"
 .section "Privilege controls" "SECID98"
 .table2
 .row &%admin_groups%&                "groups that are Exim admin users"

+.row &%commandline_checks_require_admin%& "require admin for various checks"

 .row &%deliver_drop_privilege%&      "drop root for delivery processes"
 .row &%local_from_check%&            "insert &'Sender:'& if necessary"
 .row &%local_from_prefix%&           "for testing &'From:'& for local sender"
 .row &%deliver_drop_privilege%&      "drop root for delivery processes"
 .row &%local_from_check%&            "insert &'Sender:'& if necessary"
 .row &%local_from_prefix%&           "for testing &'From:'& for local sender"


@@ -14447,6 +14462,14 @@ The CHUNKING extension (RFC3030) will be advertised in the EHLO message to
 these hosts.
 Hosts may use the BDAT command as an alternate to DATA.
 
 these hosts.
 Hosts may use the BDAT command as an alternate to DATA.
 

+.new
+.option commandline_checks_require_admin main boolean &`false`&
+.cindex "restricting access to features"
+This option restricts various basic checking features to require an
+administrative user.
+This affects most of the &%-b*%& options, such as &%-be%&.
+.wen
+

 .option debug_store main boolean &`false`&
 .cindex debugging "memory corruption"
 .cindex memory debugging
 .option debug_store main boolean &`false`&
 .cindex debugging "memory corruption"
 .cindex memory debugging


@@ -15928,12 +15951,13 @@ different spool directories.
 
 
 .option prod_requires_admin main boolean true
 
 
 .option prod_requires_admin main boolean true

+.cindex "restricting access to features"

 .oindex "&%-M%&"
 .oindex "&%-R%&"
 .oindex "&%-q%&"
 The &%-M%&, &%-R%&, and &%-q%& command-line options require the caller to be an
 admin user unless &%prod_requires_admin%& is set false. See also
 .oindex "&%-M%&"
 .oindex "&%-R%&"
 .oindex "&%-q%&"
 The &%-M%&, &%-R%&, and &%-q%& command-line options require the caller to be an
 admin user unless &%prod_requires_admin%& is set false. See also

-&%queue_list_requires_admin%&.
+&%queue_list_requires_admin%& and &%commandline_checks_require_admin%&.

 
 
 .option qualify_domain main string "see below"
 
 
 .option qualify_domain main string "see below"


@@ -15972,10 +15996,12 @@ next queue run. See also &%hold_domains%& and &%queue_smtp_domains%&.
 
 
 .option queue_list_requires_admin main boolean true
 
 
 .option queue_list_requires_admin main boolean true

+.cindex "restricting access to features"

 .oindex "&%-bp%&"
 The &%-bp%& command-line option, which lists the messages that are on the
 queue, requires the caller to be an admin user unless
 .oindex "&%-bp%&"
 The &%-bp%& command-line option, which lists the messages that are on the
 queue, requires the caller to be an admin user unless

-&%queue_list_requires_admin%& is set false. See also &%prod_requires_admin%&.
+&%queue_list_requires_admin%& is set false.
+See also &%prod_requires_admin%& and &%commandline_checks_require_admin%&.

 
 
 .option queue_only main boolean false
 
 
 .option queue_only main boolean false


@@ -16821,6 +16847,32 @@ as failures in the configuration file.
 By using this option to override the compiled-in path, it is possible to run
 tests of Exim without using the standard spool.
 
 By using this option to override the compiled-in path, it is possible to run
 tests of Exim without using the standard spool.
 

+.new
+.option spool_wireformat main boolean false
+.cindex "spool directory" "file formats"
+If this option is set, Exim may for some messages use an alternate format
+for data-files in the spool which matches the wire format.
+Doing this permits more efficient message reception and transmission.
+Currently it is only done for messages received using the EMSTP CHUNKING
+option.
+
+The following variables will not have useful values:
+.code
+$max_received_linelength
+$body_linecount
+$body_zerocount
+.endd
+
+Users of the local_scan() API (see &<<CHAPlocalscan>>&),
+and any external programs which are passed a reference to a message data file
+(except via the &"regex"&, &"malware"& or &"spam"&) ACL conditions)
+will need to be aware of the potential different format.
+
+Using any of the ACL conditions noted will negate the reception benefit
+(as a Unix-mbox-format file is contructed for them).
+The transimssion benefit is maintained.
+.wen
+

 .option sqlite_lock_timeout main time 5s
 .cindex "sqlite lookup type" "lock timeout"
 This option controls the timeout that the &(sqlite)& lookup uses when trying to
 .option sqlite_lock_timeout main time 5s
 .cindex "sqlite lookup type" "lock timeout"
 This option controls the timeout that the &(sqlite)& lookup uses when trying to


@@ -20293,11 +20345,15 @@ relative path is then passed to the transport unmodified.
 
 
 .option forbid_blackhole redirect boolean false
 
 
 .option forbid_blackhole redirect boolean false

+.cindex "restricting access to features"
+.cindex "filter" "locking out certain features"

 If this option is true, the &':blackhole:'& item may not appear in a
 redirection list.
 
 
 .option forbid_exim_filter redirect boolean false
 If this option is true, the &':blackhole:'& item may not appear in a
 redirection list.
 
 
 .option forbid_exim_filter redirect boolean false

+.cindex "restricting access to features"
+.cindex "filter" "locking out certain features"

 If this option is set true, only Sieve filters are permitted when
 &%allow_filter%& is true.
 
 If this option is set true, only Sieve filters are permitted when
 &%allow_filter%& is true.
 


@@ -20305,7 +20361,9 @@ If this option is set true, only Sieve filters are permitted when
 
 
 .option forbid_file redirect boolean false
 
 
 .option forbid_file redirect boolean false

+.cindex "restricting access to features"

 .cindex "delivery" "to file; forbidding"
 .cindex "delivery" "to file; forbidding"

+.cindex "filter" "locking out certain features"

 .cindex "Sieve filter" "forbidding delivery to a file"
 .cindex "Sieve filter" "&""keep""& facility; disabling"
 If this option is true, this router may not generate a new address that
 .cindex "Sieve filter" "forbidding delivery to a file"
 .cindex "Sieve filter" "&""keep""& facility; disabling"
 If this option is true, this router may not generate a new address that


@@ -20316,17 +20374,22 @@ locks out the Sieve's &"keep"& facility.
 
 
 .option forbid_filter_dlfunc redirect boolean false
 
 
 .option forbid_filter_dlfunc redirect boolean false

+.cindex "restricting access to features"

 .cindex "filter" "locking out certain features"
 If this option is true, string expansions in Exim filters are not allowed to
 make use of the &%dlfunc%& expansion facility to run dynamically loaded
 functions.
 
 .option forbid_filter_existstest redirect boolean false
 .cindex "filter" "locking out certain features"
 If this option is true, string expansions in Exim filters are not allowed to
 make use of the &%dlfunc%& expansion facility to run dynamically loaded
 functions.
 
 .option forbid_filter_existstest redirect boolean false

+.cindex "restricting access to features"
+.cindex "filter" "locking out certain features"

 .cindex "expansion" "statting a file"
 If this option is true, string expansions in Exim filters are not allowed to
 make use of the &%exists%& condition or the &%stat%& expansion item.
 
 .option forbid_filter_logwrite redirect boolean false
 .cindex "expansion" "statting a file"
 If this option is true, string expansions in Exim filters are not allowed to
 make use of the &%exists%& condition or the &%stat%& expansion item.
 
 .option forbid_filter_logwrite redirect boolean false

+.cindex "restricting access to features"
+.cindex "filter" "locking out certain features"

 If this option is true, use of the logging facility in Exim filters is not
 permitted. Logging is in any case available only if the filter is being run
 under some unprivileged uid (which is normally the case for ordinary users'
 If this option is true, use of the logging facility in Exim filters is not
 permitted. Logging is in any case available only if the filter is being run
 under some unprivileged uid (which is normally the case for ordinary users'


@@ -20334,27 +20397,37 @@ under some unprivileged uid (which is normally the case for ordinary users'
 
 
 .option forbid_filter_lookup redirect boolean false
 
 
 .option forbid_filter_lookup redirect boolean false

+.cindex "restricting access to features"
+.cindex "filter" "locking out certain features"

 If this option is true, string expansions in Exim filter files are not allowed
 to make use of &%lookup%& items.
 
 
 .option forbid_filter_perl redirect boolean false
 If this option is true, string expansions in Exim filter files are not allowed
 to make use of &%lookup%& items.
 
 
 .option forbid_filter_perl redirect boolean false

+.cindex "restricting access to features"
+.cindex "filter" "locking out certain features"

 This option has an effect only if Exim is built with embedded Perl support. If
 it is true, string expansions in Exim filter files are not allowed to make use
 of the embedded Perl support.
 
 
 .option forbid_filter_readfile redirect boolean false
 This option has an effect only if Exim is built with embedded Perl support. If
 it is true, string expansions in Exim filter files are not allowed to make use
 of the embedded Perl support.
 
 
 .option forbid_filter_readfile redirect boolean false

+.cindex "restricting access to features"
+.cindex "filter" "locking out certain features"

 If this option is true, string expansions in Exim filter files are not allowed
 to make use of &%readfile%& items.
 
 
 .option forbid_filter_readsocket redirect boolean false
 If this option is true, string expansions in Exim filter files are not allowed
 to make use of &%readfile%& items.
 
 
 .option forbid_filter_readsocket redirect boolean false

+.cindex "restricting access to features"
+.cindex "filter" "locking out certain features"

 If this option is true, string expansions in Exim filter files are not allowed
 to make use of &%readsocket%& items.
 
 
 .option forbid_filter_reply redirect boolean false
 If this option is true, string expansions in Exim filter files are not allowed
 to make use of &%readsocket%& items.
 
 
 .option forbid_filter_reply redirect boolean false

+.cindex "restricting access to features"
+.cindex "filter" "locking out certain features"

 If this option is true, this router may not generate an automatic reply
 message. Automatic replies can be generated only from Exim or Sieve filter
 files, not from traditional forward files. This option is forced to be true if
 If this option is true, this router may not generate an automatic reply
 message. Automatic replies can be generated only from Exim or Sieve filter
 files, not from traditional forward files. This option is forced to be true if


@@ -20362,11 +20435,15 @@ files, not from traditional forward files. This option is forced to be true if
 
 
 .option forbid_filter_run redirect boolean false
 
 
 .option forbid_filter_run redirect boolean false

+.cindex "restricting access to features"
+.cindex "filter" "locking out certain features"

 If this option is true, string expansions in Exim filter files are not allowed
 to make use of &%run%& items.
 
 
 .option forbid_include redirect boolean false
 If this option is true, string expansions in Exim filter files are not allowed
 to make use of &%run%& items.
 
 
 .option forbid_include redirect boolean false

+.cindex "restricting access to features"
+.cindex "filter" "locking out certain features"

 If this option is true, items of the form
 .code
 :include:<path name>
 If this option is true, items of the form
 .code
 :include:<path name>


@@ -20375,6 +20452,8 @@ are not permitted in non-filter redirection lists.
 
 
 .option forbid_pipe redirect boolean false
 
 
 .option forbid_pipe redirect boolean false

+.cindex "restricting access to features"
+.cindex "filter" "locking out certain features"

 .cindex "delivery" "to pipe; forbidding"
 If this option is true, this router may not generate a new address which
 specifies delivery to a pipe, either from an Exim filter or from a conventional
 .cindex "delivery" "to pipe; forbidding"
 If this option is true, this router may not generate a new address which
 specifies delivery to a pipe, either from an Exim filter or from a conventional


@@ -20382,6 +20461,8 @@ forward file. This option is forced to be true if &%one_time%& is set.
 
 
 .option forbid_sieve_filter redirect boolean false
 
 
 .option forbid_sieve_filter redirect boolean false

+.cindex "restricting access to features"
+.cindex "filter" "locking out certain features"

 If this option is set true, only Exim filters are permitted when
 &%allow_filter%& is true.
 
 If this option is set true, only Exim filters are permitted when
 &%allow_filter%& is true.
 


@@ -27631,13 +27712,22 @@ built, then you have SNI support).
          "SECTmulmessam"
 .cindex "multiple SMTP deliveries with TLS"
 .cindex "TLS" "multiple message deliveries"
          "SECTmulmessam"
 .cindex "multiple SMTP deliveries with TLS"
 .cindex "TLS" "multiple message deliveries"

+.new

 Exim sends multiple messages down the same TCP/IP connection by starting up
 an entirely new delivery process for each message, passing the socket from
 one process to the next. This implementation does not fit well with the use
 of TLS, because there is quite a lot of state information associated with a TLS
 connection, not just a socket identification. Passing all the state information
 Exim sends multiple messages down the same TCP/IP connection by starting up
 an entirely new delivery process for each message, passing the socket from
 one process to the next. This implementation does not fit well with the use
 of TLS, because there is quite a lot of state information associated with a TLS
 connection, not just a socket identification. Passing all the state information

-to a new process is not feasible. Consequently, Exim shuts down an existing TLS
-session before passing the socket to a new process. The new process may then
+to a new process is not feasible. Consequently, for sending using TLS Exim
+starts an additional proxy process for handling the encryption, piping the
+unencrypted data stream from and to the delivery processes.
+
+An older mode of operation can be enabled on a per-host basis by the
+&%hosts_noproxy_tls%& option on the &(smtp)& transport.  If the host matches
+this list the proxy process descibed above is not used; instead Exim
+.wen
+shuts down an existing TLS session being run by the delivery process
+before passing the socket to a new process. The new process may then

 try to start a new TLS session, and if successful, may try to re-authenticate
 if AUTH is in use, before sending the next message.
 
 try to start a new TLS session, and if successful, may try to re-authenticate
 if AUTH is in use, before sending the next message.
 


@@ -28965,6 +29055,11 @@ and cannot depend on content of received headers.
 Note also that headers cannot be
 modified by any of the post-data ACLs (DATA, MIME and DKIM).
 Headers may be modified by routers (subject to the above) and transports.
 Note also that headers cannot be
 modified by any of the post-data ACLs (DATA, MIME and DKIM).
 Headers may be modified by routers (subject to the above) and transports.

+.new
+The Received-By: header is generated as soon as the body reception starts,
+rather than the traditional time after the full message is received;
+this will affect the timestamp.
+.wen

 
 All the usual ACLs are called; if one results in the message being
 rejected, all effort spent in delivery (including the costs on
 
 All the usual ACLs are called; if one results in the message being
 rejected, all effort spent in delivery (including the costs on


@@ -32454,9 +32549,15 @@ C variables are as follows:
 .vlist
 .vitem &*int&~body_linecount*&
 This variable contains the number of lines in the message's body.
 .vlist
 .vitem &*int&~body_linecount*&
 This variable contains the number of lines in the message's body.

+.new
+It is not valid if the &%spool_files_wireformat%& option is used.
+.wen

 
 .vitem &*int&~body_zerocount*&
 This variable contains the number of binary zero bytes in the message's body.
 
 .vitem &*int&~body_zerocount*&
 This variable contains the number of binary zero bytes in the message's body.

+.new
+It is not valid if the &%spool_files_wireformat%& option is used.
+.wen

 
 .vitem &*unsigned&~int&~debug_selector*&
 This variable is set to zero when no debugging is taking place. Otherwise, it
 
 .vitem &*unsigned&~int&~debug_selector*&
 This variable is set to zero when no debugging is taking place. Otherwise, it


@@ -35898,6 +35999,7 @@ the following table:
 &`I   `&        local interface used
 &`K   `&        CHUNKING extension used
 &`id  `&        message id for incoming message
 &`I   `&        local interface used
 &`K   `&        CHUNKING extension used
 &`id  `&        message id for incoming message

+&`M8S `&        8BITMIME status for incoming message

 &`P   `&        on &`<=`& lines: protocol used
 &`    `&        on &`=>`& and &`**`& lines: return path
 &`PRDR`&        PRDR extension used
 &`P   `&        on &`<=`& lines: protocol used
 &`    `&        on &`=>`& and &`**`& lines: return path
 &`PRDR`&        PRDR extension used


@@ -37896,6 +37998,13 @@ the contents of files on the spool via the Exim monitor (which runs
 unprivileged), Exim must be built to allow group read access to its spool
 files.
 
 unprivileged), Exim must be built to allow group read access to its spool
 files.
 

+.new
+By default, regular users are trusted to perform basic testing and
+introspection commands, as themselves.  This setting can be tightened by
+setting the &%commandline_checks_require_admin%& option.
+This affects most of the checking options,
+such as &%-be%& and anything else &%-b*%&.
+.wen

 
 
 .section "Spool files" "SECID275"
 
 
 .section "Spool files" "SECID275"


@@ -38016,6 +38125,13 @@ file remains in existence. When Exim next processes the message, it notices the
 -J file and uses it to update the -H file before starting the next delivery
 attempt.
 
 -J file and uses it to update the -H file before starting the next delivery
 attempt.
 

+.new
+Files whose names end with -K or .eml may also be seen in the spool.
+These are temporaries used for DKIM or malware processing, when that is used.
+They should be tidied up by normal operations; any old ones are probably
+relics of crashes and can be removed.
+.wen
+

 .section "Format of the -H file" "SECID282"
 .cindex "uid (user id)" "in spool file"
 .cindex "gid (group id)" "in spool file"
 .section "Format of the -H file" "SECID282"
 .cindex "uid (user id)" "in spool file"
 .cindex "gid (group id)" "in spool file"


@@ -38093,8 +38209,8 @@ The address of an authenticated sender &-- the value of the
 &$authenticated_sender$& variable.
 
 .vitem "&%-body_linecount%&&~<&'number'&>"
 &$authenticated_sender$& variable.
 
 .vitem "&%-body_linecount%&&~<&'number'&>"

-This records the number of lines in the body of the message, and is always
-present.
+This records the number of lines in the body of the message, and is
+present unless &%-spool_file_wireformat%& is.

 
 .vitem "&%-body_zerocount%&&~<&'number'&>"
 This records the number of binary zero bytes in the body of the message, and is
 
 .vitem "&%-body_zerocount%&&~<&'number'&>"
 This records the number of binary zero bytes in the body of the message, and is


@@ -38176,6 +38292,14 @@ to ensure that the caller is displayed in queue listings).
 If a message was scanned by SpamAssassin, this is present. It records the value
 of &$spam_score_int$&.
 
 If a message was scanned by SpamAssassin, this is present. It records the value
 of &$spam_score_int$&.
 

+.new
+.vitem &%-spool_file_wireformat%&
+The -D file for this message is in wire-format (for ESMTP CHUNKING)
+rather than Unix-format.
+The line-ending is CRLF rather than newline.
+There is still, however, no leading-dot-stuffing.
+.wen
+

 .vitem &%-tls_certificate_verified%&
 A TLS certificate was received from the client that sent this message, and the
 certificate was verified by the server.
 .vitem &%-tls_certificate_verified%&
 A TLS certificate was received from the client that sent this message, and the
 certificate was verified by the server.


@@ -38283,6 +38407,20 @@ unqualified domain &'foundation'&.
 .ecindex IIDforspo2
 .ecindex IIDforspo3
 
 .ecindex IIDforspo2
 .ecindex IIDforspo3
 

+.new
+.section "Format of the -D file" "SECID282a"
+The data file is traditionally in Unix-standard format: lines are ended with
+an ASCII newline character.
+However, when the &%spool_wireformat%& main option is used some -D files
+can have an alternate format.
+This is flagged by a &%-spool_file_wireformat%& line in the corresponding -H file.
+The -D file lines (not including the first name-component line) are
+suitable for direct copying to the wire when transmitting using the
+ESMTP CHUNKING option, meaning lower processing overhead.
+Lines are terminated with an ASCII CRLF pair.
+There is no dot-stuffing (and no dot-termination).
+.wen
+

 . ////////////////////////////////////////////////////////////////////////////
 . ////////////////////////////////////////////////////////////////////////////
 
 . ////////////////////////////////////////////////////////////////////////////
 . ////////////////////////////////////////////////////////////////////////////