From: Jeremy Harris Date: Sun, 30 Jun 2013 16:02:05 +0000 (+0100) Subject: Merge branch 'master' into transp_logging_1031 X-Git-Url: https://git.exim.org/users/jgh/exim.git/commitdiff_plain/35e4802b4bdf92823c4813e61d9e6ddf5162f8ca Merge branch 'master' into transp_logging_1031 Conflicts: doc/doc-txt/experimental-spec.txt src/src/EDITME src/src/deliver.c src/src/exim.c --- 35e4802b4bdf92823c4813e61d9e6ddf5162f8ca diff --cc doc/doc-txt/experimental-spec.txt index 047440611,7fd2bd8ec..565c01851 --- a/doc/doc-txt/experimental-spec.txt +++ b/doc/doc-txt/experimental-spec.txt @@@ -574,85 -622,228 +622,307 @@@ to tell the DCC routines to add more in this to some results from ClamAV. Be careful. Header syntax is not checked and is added "as is". + In case you've troubles with sites sending the same queue items from several + hosts and fail to get through greylisting you can use + $acl_m_dcc_override_client_ip + + Setting $acl_m_dcc_override_client_ip to an IP address overrides the default + of $sender_host_address. eg. use the following ACL in DATA stage: + + warn set acl_m_dcc_override_client_ip = \ + ${lookup{$sender_helo_name}nwildlsearch{/etc/mail/multipleip_sites}{$value}{}} + condition = ${if def:acl_m_dcc_override_client_ip} + log_message = dbg: acl_m_dcc_override_client_ip set to \ + $acl_m_dcc_override_client_ip + + Then set something like + # cat /etc/mail/multipleip_sites + mout-xforward.gmx.net 82.165.159.12 + mout.gmx.net 212.227.15.16 + + Use a reasonable IP. eg. one the sending cluster acutally uses. + + DMARC Support + -------------------------------------------------------------- + + DMARC combines feedback from SPF, DKIM, and header From: in order + to attempt to provide better indicators of the authenticity of an + email. This document does not explain the fundamentals, you + should read and understand how it works by visiting the website at + http://www.dmarc.org/. + + DMARC support is added via the libopendmarc library. Visit: + + http://sourceforge.net/projects/opendmarc/ + + to obtain a copy, or find it in your favorite rpm package + repository. If building from source, this description assumes + that headers will be in /usr/local/include, and that the libraries + are in /usr/local/lib. + + 1. To compile Exim with DMARC support, you must first enable SPF. + Please read the above section on enabling the EXPERIMENTAL_SPF + feature. You must also have DKIM support, so you cannot set the + DISABLE_DKIM feature. Once both of those conditions have been met + you can enable DMARC in Local/Makefile: + + EXPERIMENTAL_DMARC=yes + LDFLAGS += -lopendmarc + # CFLAGS += -I/usr/local/include + # LDFLAGS += -L/usr/local/lib + + The first line sets the feature to include the correct code, and + the second line says to link the libopendmarc libraries into the + exim binary. The commented out lines should be uncommented if you + built opendmarc from source and installed in the default location. + Adjust the paths if you installed them elsewhere, but you do not + need to uncomment them if an rpm (or you) installed them in the + package controlled locations (/usr/include and /usr/lib). + + + 2. Use the following global settings to configure DMARC: + + Required: + dmarc_tld_file Defines the location of a text file of valid + top level domains the opendmarc library uses + during domain parsing. Maintained by Mozilla, + the most current version can be downloaded + from a link at http://publicsuffix.org/list/. + + Optional: + dmarc_history_file Defines the location of a file to log results + of dmarc verification on inbound emails. The + contents are importable by the opendmarc tools + which will manage the data, send out DMARC + reports, and expire the data. Make sure the + directory of this file is writable by the user + exim runs as. + + dmarc_forensic_sender The email address to use when sending a + forensic report detailing alignment failures + if a sender domain's dmarc record specifies it + and you have configured Exim to send them. + Default: do-not-reply@$default_hostname + + + 3. By default, the DMARC processing will run for any remote, + non-authenticated user. It makes sense to only verify DMARC + status of messages coming from remote, untrusted sources. You can + use standard conditions such as hosts, senders, etc, to decide that + DMARC verification should *not* be performed for them and disable + DMARC with a control setting: + + control = dmarc_verify_disable + + A DMARC record can also specify a "forensic address", which gives + exim an email address to submit reports about failed alignment. + Exim does not do this by default because in certain conditions it + results in unintended information leakage (what lists a user might + be subscribed to, etc). You must configure exim to submit forensic + reports to the owner of the domain. If the DMARC record contains a + forensic address and you specify the control statement below, then + exim will send these forensic emails. It's also advised that you + configure a dmarc_forensic_sender because the default sender address + construction might be inadequate. + + control = dmarc_forensic_enable + + (AGAIN: You can choose not to send these forensic reports by simply + not putting the dmarc_forensic_enable control line at any point in + your exim config. If you don't tell it to send them, it will not + send them.) + + There are no options to either control. Both must appear before + the DATA acl. + + + 4. You can now run DMARC checks in incoming SMTP by using the + "dmarc_status" ACL condition in the DATA ACL. You are required to + call the spf condition first in the ACLs, then the "dmarc_status" + condition. Putting this condition in the ACLs is required in order + for a DMARC check to actually occur. All of the variables are set + up before the DATA ACL, but there is no actual DMARC check that + occurs until a "dmarc_status" condition is encountered in the ACLs. + + The dmarc_status condition takes a list of strings on its + right-hand side. These strings describe recommended action based + on the DMARC check. To understand what the policy recommendations + mean, refer to the DMARC website above. Valid strings are: + + o accept The DMARC check passed and the library recommends + accepting the email. + o reject The DMARC check failed and the library recommends + rejecting the email. + o quarantine The DMARC check failed and the library recommends + keeping it for further inspection. + o norecord No policy section in the DMARC record for this + sender domain. + o nofrom Unable to determine the domain of the sender. + o none There is no DMARC record for this sender domain. + o error Library error or dns error. + + You can prefix each string with an exclamation mark to invert its + meaning, for example "!accept" will match all results but + "accept". The string list is evaluated left-to-right in a + short-circuit fashion. When a string matches the outcome of the + DMARC check, the condition succeeds. If none of the listed + strings matches the outcome of the DMARC check, the condition + fails. + + Of course, you can also use any other lookup method that Exim + supports, including LDAP, Postgres, MySQL, etc, as long as the + result is a list of colon-separated strings; + + Several expansion variables are set before the DATA ACL is + processed, and you can use them in this ACL. The following + expansion variables are available: + + o $dmarc_status + This is a one word status indicating what the DMARC library + thinks of the email. + + o $dmarc_status_text + This is a slightly longer, human readable status. + + o $dmarc_used_domain + This is the domain which DMARC used to look up the DMARC + policy record. + + o $dmarc_ar_header + This is the entire Authentication-Results header which you can + add using an add_header modifier. + + + 5. How to enable DMARC advanced operation: + By default, Exim's DMARC configuration is intended to be + non-intrusive and conservative. To facilitate this, Exim will not + create any type of logging files without explicit configuration by + you, the admin. Nor will Exim send out any emails/reports about + DMARC issues without explicit configuration by you, the admin (other + than typical bounce messages that may come about due to ACL + processing or failure delivery issues). + + In order to log statistics suitable to be imported by the opendmarc + tools, you need to: + a. Configure the global setting dmarc_history_file. + b. Configure cron jobs to call the appropriate opendmarc history + import scripts and truncating the dmarc_history_file. + + In order to send forensic reports, you need to: + a. Configure the global setting dmarc_forensic_sender. + b. Configure, somewhere before the DATA ACL, the control option to + enable sending DMARC forensic reports. + + + 6. Example usage: + (RCPT ACL) + warn domains = +local_domains + hosts = +local_hosts + control = dmarc_verify_disable + + warn !domains = +screwed_up_dmarc_records + control = dmarc_enable_forensic + + (DATA ACL) + warn dmarc_status = accept : none : off + !authenticated = * + log_message = DMARC DEBUG: $dmarc_status $dmarc_used_domain + add_header = $dmarc_ar_header + + warn dmarc_status = !accept + !authenticated = * + log_message = DMARC DEBUG: '$dmarc_status' for $dmarc_used_domain + + warn dmarc_status = quarantine + !authenticated = * + set $acl_m_quarantine = 1 + # Do something in a transport with this flag variable + + deny dmarc_status = reject + !authenticated = * + message = Message from $domain_used_domain failed sender's DMARC policy, REJECT + + + +DBL (Database Logging) +-------------------------------------------------------------- + +This feature allows to write exim internal log information +(not available otherwise) into a database. +Initially implemented is logging of details about successfully +completed remote deliveries, which are needed for reputation +systems, and deferrals caused by a host error. + +In order to use DBL, you must set + +EXPERIMENTAL_DBL=yes + +in your Local/Makefile + +and define the database queries in the runtime config file, to +be executed at end of delivery. + +Additionally, there are 8 more variables, available at end of +delivery: + +dbl_delivery_ip IP of host, which has accepted delivery +dbl_delivery_port Port of remote host which has accepted delivery +dbl_delivery_fqdn FQDN of host, which has accepted delivery +dbl_delivery_local_part local part of address being delivered +dbl_delivery_domain domain part of address being delivered +dbl_delivery_confirmation SMTP confirmation message + +In case of a deferral caused by a host-error: +dbl_defer_errno Error number +dbl_defer_errstr Error string possibly containing more details + + +To log successful deliveries, set the following option in the main +option part of runtime config. + +dbl_delivery_query + +An example might look like: + +dbl_delivery_query = \ +${lookup pgsql {SELECT * FROM record_Delivery( \ + '${quote_pgsql:$sender_address_domain}',\ + '${quote_pgsql:${lc:$sender_address_local_part}}', \ + '${quote_pgsql:$dbl_delivery_domain}', \ + '${quote_pgsql:${lc:$dbl_delivery_local_part}}', \ + '${quote_pgsql:$dbl_delivery_ip}', \ + '${quote_pgsql:${lc:$dbl_delivery_fqdn}}', \ + '${quote_pgsql:$message_exim_id}')}} + + +In order to log host deferrals, add the following option to an SMTP +transport: + +dbl_host_defer_query + +This is a private option of the SMTP transport. It is intended to +log failures of remote hosts. It is executed only when exim has +attempted to deliver a message to a remote host and failed due to +an error which doesn't seem to be related to the individual +message, sender, or recipient address. +See section 45.2 of the exim documentation for more details on how +this is determined. + +Example: + +dbl_host_defer_query = \ +${lookup mysql {insert into delivlog set \ + msgid = '${quote_mysql:$message_exim_id}', \ + senderlp = '${quote_mysql:${lc:$sender_address_local_part}}', \ + senderdom = '${quote_mysql:$sender_address_domain}', \ + delivlp = '${quote_mysql:${lc:$dbl_delivery_local_part}}', \ + delivdom = '${quote_mysql:$dbl_delivery_domain}', \ + delivip = '${quote_mysql:$dbl_delivery_ip}', \ + delivport = '${quote_mysql:$dbl_delivery_port}', \ + delivfqdn = '${quote_mysql:$dbl_delivery_fqdn}', \ + deliverrno = '${quote_mysql:$dbl_defer_errno}', \ + deliverrstr = '${quote_mysql:$dbl_defer_errstr}' \ + }} -------------------------------------------------------------- End of file diff --cc src/src/EDITME index 1387b980a,e29c1eb25..637256c7d --- a/src/src/EDITME +++ b/src/src/EDITME @@@ -452,10 -460,16 +460,20 @@@ EXIM_MONITOR=eximon.bi # EXPERIMENTAL_OCSP=yes + # Uncomment the following line to add DMARC checking capability, implemented + # using libopendmarc libraries. + # EXPERIMENTAL_DMARC=yes + # CFLAGS += -I/usr/local/include + # LDFLAGS += -lopendmarc + + # Uncomment the following line to add Per-Recipient-Data-Response support. + # EXPERIMENTAL_PRDR=yes + +# This feature allows to write log information about a completed +# delivery into a database +# EXPERIMENTAL_DBL=yes + + ############################################################################### # THESE ARE THINGS YOU MIGHT WANT TO SPECIFY # ############################################################################### diff --cc src/src/config.h.defaults index 1652c7842,1594f6858..bcbc70b38 --- a/src/src/config.h.defaults +++ b/src/src/config.h.defaults @@@ -163,10 -165,11 +165,12 @@@ it's a default value. * /* EXPERIMENTAL features */ #define EXPERIMENTAL_BRIGHTMAIL #define EXPERIMENTAL_DCC + #define EXPERIMENTAL_DMARC #define EXPERIMENTAL_OCSP + #define EXPERIMENTAL_PRDR #define EXPERIMENTAL_SPF #define EXPERIMENTAL_SRS +#define EXPERIMENTAL_DBL /* For developers */ #define WANT_DEEPER_PRINTF_CHECKS diff --cc src/src/deliver.c index 1c3c4bafd,23e63d553..7ff25f22a --- a/src/src/deliver.c +++ b/src/src/deliver.c @@@ -673,6 -673,166 +673,177 @@@ while (addr->parent != NULL + /* If msg is NULL this is a delivery log and logchar is used. Otherwise + this is a nonstandard call; no two-characher delivery flag is written + but sender-host and sender are prefixed and "msg" is inserted in the log line. + + Arguments: + flags passed to log_write() + */ + void + delivery_log(int flags, address_item * addr, int logchar, uschar * msg) + { + uschar *log_address; + int size = 256; /* Used for a temporary, */ + int ptr = 0; /* expanding buffer, for */ + uschar *s; /* building log lines; */ + void *reset_point; /* released afterwards. */ + + + /* Log the delivery on the main log. We use an extensible string to build up + the log line, and reset the store afterwards. Remote deliveries should always + have a pointer to the host item that succeeded; local deliveries can have a + pointer to a single host item in their host list, for use by the transport. */ + + s = reset_point = store_get(size); + + log_address = string_log_address(addr, (log_write_selector & L_all_parents) != 0, TRUE); + if (msg) + s = string_append(s, &size, &ptr, 3, host_and_ident(TRUE), US" ", log_address); + else + { + s[ptr++] = logchar; + s = string_append(s, &size, &ptr, 2, US"> ", log_address); + } + + if ((log_extra_selector & LX_sender_on_delivery) != 0 || msg) + s = string_append(s, &size, &ptr, 3, US" F=<", sender_address, US">"); + + #ifdef EXPERIMENTAL_SRS + if(addr->p.srs_sender) + s = string_append(s, &size, &ptr, 3, US" SRS=<", addr->p.srs_sender, US">"); + #endif ++#ifdef EXPERIMENTAL_DBL ++ dbl_delivery_ip = NULL; /* presume no successful remote delivery */ ++#endif + + /* You might think that the return path must always be set for a successful + delivery; indeed, I did for some time, until this statement crashed. The case + when it is not set is for a delivery to /dev/null which is optimised by not + being run at all. */ + + if (used_return_path != NULL && + (log_extra_selector & LX_return_path_on_delivery) != 0) + s = string_append(s, &size, &ptr, 3, US" P=<", used_return_path, US">"); + + if (msg) + s = string_append(s, &size, &ptr, 2, US" ", msg); + + /* For a delivery from a system filter, there may not be a router */ + if (addr->router != NULL) + s = string_append(s, &size, &ptr, 2, US" R=", addr->router->name); + + s = string_append(s, &size, &ptr, 2, US" T=", addr->transport->name); + + if ((log_extra_selector & LX_delivery_size) != 0) + s = string_append(s, &size, &ptr, 2, US" S=", + string_sprintf("%d", transport_count)); + + /* Local delivery */ + + if (addr->transport->info->local) + { + if (addr->host_list != NULL) + s = string_append(s, &size, &ptr, 2, US" H=", addr->host_list->name); + if (addr->shadow_message != NULL) + s = string_cat(s, &size, &ptr, addr->shadow_message, + Ustrlen(addr->shadow_message)); + } + + /* Remote delivery */ + + else + { + if (addr->host_used != NULL) + { + s = string_append(s, &size, &ptr, 5, US" H=", addr->host_used->name, + US" [", addr->host_used->address, US"]"); + if ((log_extra_selector & LX_outgoing_port) != 0) + s = string_append(s, &size, &ptr, 2, US":", string_sprintf("%d", + addr->host_used->port)); + if (continue_sequence > 1) + s = string_cat(s, &size, &ptr, US"*", 1); + } + + #ifdef SUPPORT_TLS + if ((log_extra_selector & LX_tls_cipher) != 0 && addr->cipher != NULL) + s = string_append(s, &size, &ptr, 2, US" X=", addr->cipher); + if ((log_extra_selector & LX_tls_certificate_verified) != 0 && + addr->cipher != NULL) + s = string_append(s, &size, &ptr, 2, US" CV=", + testflag(addr, af_cert_verified)? "yes":"no"); + if ((log_extra_selector & LX_tls_peerdn) != 0 && addr->peerdn != NULL) + s = string_append(s, &size, &ptr, 3, US" DN=\"", + string_printing(addr->peerdn), US"\""); + #endif + + if (addr->authenticator) + { + s = string_append(s, &size, &ptr, 2, US" A=", addr->authenticator); + if (addr->auth_id) + { + s = string_append(s, &size, &ptr, 2, US":", addr->auth_id); + if (log_extra_selector & LX_smtp_mailauth && addr->auth_sndr) + s = string_append(s, &size, &ptr, 2, US":", addr->auth_sndr); + } + } + + #ifdef EXPERIMENTAL_PRDR + if (addr->flags & af_prdr_used) + s = string_append(s, &size, &ptr, 1, US" PRDR"); + #endif + + if ((log_extra_selector & LX_smtp_confirmation) != 0 && + addr->message != NULL) + { + int i; + uschar *p = big_buffer; + uschar *ss = addr->message; + *p++ = '\"'; + for (i = 0; i < 100 && ss[i] != 0; i++) + { + if (ss[i] == '\"' || ss[i] == '\\') *p++ = '\\'; + *p++ = ss[i]; + } + *p++ = '\"'; + *p = 0; + s = string_append(s, &size, &ptr, 2, US" C=", big_buffer); + } + } + + /* Time on queue and actual time taken to deliver */ + + if ((log_extra_selector & LX_queue_time) != 0) + { + s = string_append(s, &size, &ptr, 2, US" QT=", + readconf_printtime(time(NULL) - received_time)); + } + + if ((log_extra_selector & LX_deliver_time) != 0) + { + s = string_append(s, &size, &ptr, 2, US" DT=", + readconf_printtime(addr->more_errno)); + } + + /* string_cat() always leaves room for the terminator. Release the + store we used to build the line after writing it. */ + + s[ptr] = 0; + log_write(0, flags, "%s", s); ++#ifdef EXPERIMENTAL_DBL ++DEBUG(D_deliver) ++ { ++ debug_printf(" DBL(Delivery): dbl_delivery_query=|%s| dbl_delivery_IP=%s\n", dbl_delivery_query, dbl_delivery_ip); ++ } ++if (dbl_delivery_ip != NULL && dbl_delivery_query != NULL) ++ expand_string(dbl_delivery_query); ++#endif + store_reset(reset_point); + return; + } + + + /************************************************* * Actions at the end of handling an address * *************************************************/ diff --cc src/src/exim.c index 587e13685,a27e391d1..1e4d6d646 --- a/src/src/exim.c +++ b/src/src/exim.c @@@ -806,9 -816,15 +816,18 @@@ fprintf(f, "Support for:") #ifdef EXPERIMENTAL_DCC fprintf(f, " Experimental_DCC"); #endif + #ifdef EXPERIMENTAL_DMARC + fprintf(f, " Experimental_DMARC"); + #endif + #ifdef EXPERIMENTAL_OCSP + fprintf(f, " Experimental_OCSP"); + #endif + #ifdef EXPERIMENTAL_PRDR + fprintf(f, " Experimental_PRDR"); + #endif +#ifdef EXPERIMENTAL_DBL + fprintf(f, " EXPERIMENTAL_DBL"); +#endif fprintf(f, "\n"); fprintf(f, "Lookups (built-in):");