-$Cambridge: exim/doc/doc-txt/ChangeLog,v 1.416 2006/10/24 15:01:25 ph10 Exp $
+$Cambridge: exim/doc/doc-txt/ChangeLog,v 1.417 2006/10/30 16:41:04 ph10 Exp $
Change log file for Exim from version 4.21
-------------------------------------------
headers of the same name were present, leading whitespace was being
removed from all of them, but trailing whitespace was being removed only
from the last one. Now trailing whitespace is removed from each header
- before concatenation.
+ before concatenation. Completely empty headers in a concatenation (as
+ before) are ignored.
PH/28 Fixed bug in backwards-compatibility feature of PH/09 (thanks to John
Jetmore). It would have mis-read ACL variables from pre-4.61 spool files.
+PH/29 After an address error (typically a 4xx response from a server), Exim
+ always tries the failing address if it appears in a new message, but
+ respects the retry time otherwise. This was implemented by checking for
+ being in a queue run, which isn't quite right. Now it checks the
+ "first_delivery" flag instead.
+
+PH/30 Exim was sometimes attempting to deliver messages that had suffered
+ address errors (4xx response to RCPT) over the same connection as other
+ messages routed to the same hosts. Such deliveries are always "forced",
+ so retry times are not inspected. This resulted in far too many retries
+ for the affected addresses. The effect occurred only when there were more
+ hosts than the hosts_max_try setting in the smtp transport when it had
+ the 4xx errors. Those hosts that it had tried were not added to the list
+ of hosts for which the message was waiting, so if all were tried, there
+ was no problem. Two fixes have been applied:
+
+ (i) If there are any address or message errors in an SMTP delivery, none
+ of the hosts (tried or untried) are now added to the list of hosts
+ for which the message is waiting, so the message should not be a
+ candidate for sending over the same connection that was used for a
+ successful delivery of some other message. This seems entirely
+ reasonable: after all the message is NOT "waiting for some host".
+ This is so "obvious" that I'm not sure why it wasn't done
+ previously. Hope I haven't missed anything, but it can't do any
+ harm, as the worst effect is to miss an optimization.
+
+ (ii) If, despite (i), such a delivery is accidentally attempted, the
+ routing retry time is respected, so at least it doesn't keep
+ hammering the server.
+
Exim version 4.63
-----------------
-/* $Cambridge: exim/src/src/deliver.c,v 1.36 2006/09/18 14:49:23 ph10 Exp $ */
+/* $Cambridge: exim/src/src/deliver.c,v 1.37 2006/10/30 16:41:04 ph10 Exp $ */
/*************************************************
* Exim - an Internet mail transport agent *
(void)post_process_one(addr, DEFER, LOG_MAIN, DTYPE_ROUTER, 0);
}
- /* If queue_running, defer routing unless no retry data or we've
- passed the next retry time, or this message is forced. However,
- if the retry time has expired, allow the routing attempt.
- If it fails again, the address will be failed. This ensures that
+ /* If we are in a queue run, defer routing unless there is no retry data or
+ we've passed the next retry time, or this message is forced. In other
+ words, ignore retry data when not in a queue run.
+
+ However, if the domain retry time has expired, always allow the routing
+ attempt. If it fails again, the address will be failed. This ensures that
each address is routed at least once, even after long-term routing
failures.
If there is an address retry, check that too; just wait for the next
retry time. This helps with the case when the temporary error on the
address was really message-specific rather than address specific, since
- it allows other messages through. */
+ it allows other messages through.
+
+ We also wait for the next retry time if this is a message sent down an
+ existing SMTP connection (even though that will be forced). Otherwise there
+ will be far too many attempts for an address that gets a 4xx error. In
+ fact, after such an error, we should not get here because, the host should
+ not be remembered as one this message needs. However, there was a bug that
+ used to cause this to happen, so it is best to be on the safe side. */
- else if (!deliver_force && queue_running &&
+ else if (((queue_running && !deliver_force) || continue_hostname != NULL)
+ &&
((domain_retry_record != NULL &&
now < domain_retry_record->next_try &&
!domain_retry_record->expired)
-/* $Cambridge: exim/src/src/structs.h,v 1.13 2006/10/16 15:44:36 ph10 Exp $ */
+/* $Cambridge: exim/src/src/structs.h,v 1.14 2006/10/30 16:41:04 ph10 Exp $ */
/*************************************************
* Exim - an Internet mail transport agent *
int status; /* Usable, unusable, or unknown */
int why; /* Why host is unusable */
int last_try; /* Time of last try if known */
- BOOL update_waiting; /* Turned off if wait db not to be updated */
} host_item;
/* Chain of rewrite rules, read from the rewrite config, or parsed from the
-/* $Cambridge: exim/src/src/transport.c,v 1.15 2006/02/07 11:19:00 ph10 Exp $ */
+/* $Cambridge: exim/src/src/transport.c,v 1.16 2006/10/30 16:41:04 ph10 Exp $ */
/*************************************************
* Exim - an Internet mail transport agent *
Old records should eventually get swept up by the exim_tidydb utility.
Arguments:
- hostlist list of hosts that this message could be sent to;
- the update_waiting flag is set if a host is to be noted
+ hostlist list of hosts that this message could be sent to
tpname name of the transport
Returns: nothing
if (dbm_file == NULL) return;
/* Scan the list of hosts for which this message is waiting, and ensure
-that the message id is in each host record for those that have the
-update_waiting flag set. */
+that the message id is in each host record. */
for (host = hostlist; host!= NULL; host = host->next)
{
uschar *s;
int i, host_length;
- /* Skip if the update_waiting flag is not set. */
-
- if (!host->update_waiting) continue;
-
/* Skip if this is the same host as we just processed; otherwise remember
the name for next time. */
-/* $Cambridge: exim/src/src/transports/smtp.c,v 1.27 2006/10/09 14:36:25 ph10 Exp $ */
+/* $Cambridge: exim/src/src/transports/smtp.c,v 1.28 2006/10/30 16:41:04 ph10 Exp $ */
/*************************************************
* Exim - an Internet mail transport agent *
static uschar *smtp_command; /* Points to last cmd for error messages */
static uschar *mail_command; /* Points to MAIL cmd for error messages */
+static BOOL update_waiting; /* TRUE to update the "wait" database */
/*************************************************
transport_rcpt_address(addr, include_affixes));
set_errno(addrlist, save_errno, message, DEFER, FALSE);
retry_add_item(addr, addr->address_retry_key, 0);
- host->update_waiting = FALSE;
+ update_waiting = FALSE;
return -1;
}
if (host->next != NULL) log_write(0, LOG_MAIN, "%s", addr->message);
- /* Do not put this message on the list of those waiting for this host,
- as otherwise it is likely to be tried too often. */
+ /* Do not put this message on the list of those waiting for specific
+ hosts, as otherwise it is likely to be tried too often. */
- host->update_waiting = FALSE;
+ update_waiting = FALSE;
/* Add a retry item for the address so that it doesn't get tried
again too soon. */
continue_hostname, continue_host_address);
}
+/* Set the flag requesting that these hosts be added to the waiting
+database if the delivery fails temporarily or if we are running with
+queue_smtp or a 2-stage queue run. This gets unset for certain
+kinds of error, typically those that are specific to the message. */
+
+update_waiting = TRUE;
+
/* If a host list is not defined for the addresses - they must all have the
same one in order to be passed to a single transport - or if the transport has
a host list with hosts_override set, use the host list supplied with the
nexthost = host->next;
- /* Set the flag requesting that this host be added to the waiting
- database if the delivery fails temporarily or if we are running with
- queue_smtp or a 2-stage queue run. This gets unset for certain
- kinds of error, typically those that are specific to the message. */
-
- host->update_waiting = TRUE;
-
/* If the address hasn't yet been obtained from the host name, look it up
now, unless the host is already marked as unusable. If it is marked as
unusable, it means that the router was unable to find its IP address (in
/* If there was a retry message key, implying that previously there
was a message-specific defer, we don't want to update the list of
- messages waiting for this host. */
+ messages waiting for these hosts. */
- if (retry_message_key != NULL) host->update_waiting = FALSE;
+ if (retry_message_key != NULL) update_waiting = FALSE;
continue; /* With the next host or IP address */
}
}
to the retry chain. Note that if there was a message defer but now there is
a host defer, the message defer record gets deleted. That seems perfectly
reasonable. Also, stop the message from being remembered as waiting
- for this host. */
+ for specific hosts. */
if (message_defer || retry_message_key != NULL)
{
}
retry_add_item(addrlist, retry_message_key,
rf_message | rf_host | delete_flag);
- host->update_waiting = FALSE;
+ update_waiting = FALSE;
}
/* Any return other than DEFER (that is, OK or ERROR) means that the
}
/* Update the database which keeps information about which messages are waiting
-for which hosts to become available. Each host in the list has a flag which is
-set if the data is to be updated. For some message-specific errors, the flag is
-turned off because we don't want follow-on deliveries in those cases. */
+for which hosts to become available. For some message-specific errors, the
+update_waiting flag is turned off because we don't want follow-on deliveries in
+those cases. */
-transport_update_waiting(hostlist, tblock->name);
+if (update_waiting) transport_update_waiting(hostlist, tblock->name);
END_TRANSPORT:
--- /dev/null
+# Exim test configuration 0543
+
+exim_path = EXIM_PATH
+host_lookup_order = bydns
+rfc1413_query_timeout = 0s
+spool_directory = DIR/spool
+log_file_path = DIR/spool/log/%slog
+gecos_pattern = ""
+gecos_name = CALLER_NAME
+
+# ----- Main settings -----
+
+qualify_domain = test.ex
+queue_run_in_order
+
+
+# ----- Routers -----
+
+begin routers
+
+smarthost:
+ driver = accept
+ transport = smtp
+
+
+# ----- Transports -----
+
+begin transports
+
+smtp:
+ driver = smtp
+ hosts = thisloop.test.ex
+ hosts_max_try = 1
+ allow_localhost
+ port = PORT_S
+
+
+# ----- Retry -----
+
+
+begin retry
+
+* * F,5d,15m
+
+
+# End
1999-03-02 09:44:33 Start queue run: pid=pppp
1999-03-02 09:44:33 10HmaZ-0005vi-00 == userx@domain1 R=others T=smtp defer (-44): SMTP error from remote mail server after RCPT TO:<userx@domain1>: host 127.0.0.1 [127.0.0.1]: 450 Temporary error
1999-03-02 09:44:33 10HmaZ-0005vi-00 => usery@domain1 R=others T=smtp H=127.0.0.1 [127.0.0.1]
-1999-03-02 09:44:33 10HmbA-0005vi-00 == userx@domain1 R=others T=smtp defer (-44): SMTP error from remote mail server after RCPT TO:<userx@domain1>: host 127.0.0.1 [127.0.0.1]: 450 Temporary error
-1999-03-02 09:44:33 10HmbA-0005vi-00 => usery@domain1 R=others T=smtp H=127.0.0.1 [127.0.0.1]*
-1999-03-02 09:44:33 10HmaZ-0005vi-00 == userx@domain1 R=others T=smtp defer (-44): SMTP error from remote mail server after RCPT TO:<userx@domain1>: host 127.0.0.1 [127.0.0.1]: 450 Temporary error
1999-03-02 09:44:33 10HmbA-0005vi-00 == userx@domain1 routing defer (-51): retry time not reached
+1999-03-02 09:44:33 10HmbA-0005vi-00 == usery@domain1 R=others T=smtp defer (-44): SMTP error from remote mail server after RCPT TO:<usery@domain1>: host 127.0.0.1 [127.0.0.1]: 450 Temporary error
+1999-03-02 09:44:33 10HmbA-0005vi-00 == userx@domain1 routing defer (-51): retry time not reached
+1999-03-02 09:44:33 10HmbA-0005vi-00 == usery@domain1 routing defer (-51): retry time not reached
1999-03-02 09:44:33 End queue run: pid=pppp
1999-03-02 09:44:33 Start queue run: pid=pppp
1999-03-02 09:44:33 10HmaZ-0005vi-00 == userx@domain1 routing defer (-51): retry time not reached
1999-03-02 09:44:33 10HmbA-0005vi-00 == userx@domain1 routing defer (-51): retry time not reached
+1999-03-02 09:44:33 10HmbA-0005vi-00 == usery@domain1 routing defer (-51): retry time not reached
1999-03-02 09:44:33 End queue run: pid=pppp
1999-03-02 09:44:33 Start queue run: pid=pppp
1999-03-02 09:44:33 10HmbA-0005vi-00 == userx@domain1 routing defer (-51): retry time not reached
+1999-03-02 09:44:33 10HmbA-0005vi-00 == usery@domain1 routing defer (-51): retry time not reached
1999-03-02 09:44:33 End queue run: pid=pppp
1999-03-02 09:44:33 10HmaZ-0005vi-00 127.0.0.1 [127.0.0.1] Connection refused
1999-03-02 09:44:33 10HmaZ-0005vi-00 == userx@domain1 R=others T=smtp defer (dd): Connection refused
--- /dev/null
+1999-03-02 09:44:33 10HmaX-0005vi-00 <= CALLER@test.ex U=CALLER P=local S=sss
+1999-03-02 09:44:33 10HmaY-0005vi-00 <= CALLER@test.ex U=CALLER P=local S=sss
+1999-03-02 09:44:33 10HmaY-0005vi-00 SMTP error from remote mail server after RCPT TO:<usery@domain1>: host thisloop.test.ex [127.0.0.1]: 451 Later, please
+1999-03-02 09:44:33 10HmaY-0005vi-00 == usery@domain1 R=smarthost T=smtp defer (-44): SMTP error from remote mail server after RCPT TO:<usery@domain1>: host thisloop.test.ex [127.0.0.1]: 451 Later, please
+1999-03-02 09:44:33 Start queue run: pid=pppp
+1999-03-02 09:44:33 10HmaX-0005vi-00 => userx@domain1 R=smarthost T=smtp H=thisloop.test.ex [127.0.0.1]
+1999-03-02 09:44:33 10HmaX-0005vi-00 Completed
+1999-03-02 09:44:33 10HmaY-0005vi-00 == usery@domain1 routing defer (-51): retry time not reached
+1999-03-02 09:44:33 End queue run: pid=pppp
250 Sender OK
RCPT TO:
450 Temporary error
-RCPT TO:
-250 OK
-DATA
-354 Go ahead
-.
-250 OK
-MAIL FROM:
-250 Sender OK
-RCPT TO:
-450 Temporary error
QUIT
250 OK
****
--- /dev/null
+# 4xx retries with 2nd message down one SMTP connection
+need_ipv4
+#
+# Put a message on the queue
+exim -odq userx@domain1
+Test message 1
+****
+# Arrange a second message that's had a 4xx
+server PORT_S
+220 ESMTP
+EHLO
+250-OK
+250 HELP
+MAIL
+250 OK
+RCPT
+451 Later, please
+QUIT
+220 OK
+****
+exim -odi usery@domain1
+Test message 2
+****
+1
+dump wait-smtp
+# A queue run will try these in order, since queue_run_in_order is set.
+# Arrange that the first one is accepted, so it should fire up the second
+# on the same connection. But it should respect the retry time.
+server PORT_S
+220 ESMTP
+EHLO
+250-OK
+250 HELP
+MAIL
+250 OK
+RCPT
+250 OK
+DATA
+354 More...
+.
+250 OK
+QUIT
+250 OK
+****
+exim -q -d-all+route
+****
+no_msglog_check
locking TESTSUITE/spool/db/retry.lockfile
no retry data available
added retry item for R:userx@test.ex: errno=-44 more_errno=dd,A flags=0
-locking TESTSUITE/spool/db/wait-t1.lockfile
reading retry information for R:userx@test.ex from subprocess
added retry item
LOG: MAIN
no host retry record
no message retry record
added retry item for R:userx@test.ex: errno=-44 more_errno=dd,A flags=0
-locking TESTSUITE/spool/db/wait-t1.lockfile
reading retry information for R:userx@test.ex from subprocess
existing delete item dropped
added retry item
no host retry record
no message retry record
added retry item for R:userx@test.ex: errno=-44 more_errno=dd,A flags=0
-locking TESTSUITE/spool/db/wait-t1.lockfile
reading retry information for R:userx@test.ex from subprocess
existing delete item dropped
added retry item
no retry data available
added retry item for R:userx@test.ex: errno=-44 more_errno=dd,A flags=0
added retry item for R:usery@test.ex: errno=-44 more_errno=dd,A flags=0
-locking TESTSUITE/spool/db/wait-t1.lockfile
reading retry information for R:userx@test.ex from subprocess
added retry item
reading retry information for R:usery@test.ex from subprocess
no message retry record
added retry item for R:userx@test.ex: errno=-44 more_errno=dd,A flags=0
added retry item for R:usery@test.ex: errno=-44 more_errno=dd,A flags=0
-locking TESTSUITE/spool/db/wait-t1.lockfile
reading retry information for R:userx@test.ex from subprocess
existing delete item dropped
added retry item
LOG: MAIN
=> d1@myhost.test.ex R=ut4 T=ut4 H=127.0.0.1 [127.0.0.1]
locking TESTSUITE/spool/db/retry.lockfile
-locking TESTSUITE/spool/db/wait-ut4.lockfile
LOG: MAIN
== d2@myhost.test.ex R=ut4 T=ut4 defer (-44): SMTP error from remote mail server after RCPT TO:<d2@myhost.test.ex>: host 127.0.0.1 [127.0.0.1]: 450 soft error
locking TESTSUITE/spool/db/retry.lockfile
=> d1@myhost.test.ex P=<> R=ut4 T=ut4 H=127.0.0.1 [127.0.0.1]
log writing disabled
locking TESTSUITE/spool/db/retry.lockfile
-locking TESTSUITE/spool/db/wait-ut4.lockfile
LOG: MAIN
== d2@myhost.test.ex R=ut4 T=ut4 defer (-44): SMTP error from remote mail server after RCPT TO:<d2@myhost.test.ex>: host 127.0.0.1 [127.0.0.1]: 450 soft error
log writing disabled
=> f1@myhost.test.ex P=<CALLER@myhost.test.ex> R=ut6 T=ut6 H=127.0.0.1 [127.0.0.1]
log writing disabled
locking TESTSUITE/spool/db/retry.lockfile
-locking TESTSUITE/spool/db/wait-ut6.lockfile
LOG: MAIN
== f2@myhost.test.ex R=ut6 T=ut6 defer (-44): SMTP error from remote mail server after RCPT TO:<f2@myhost.test.ex>: host 127.0.0.1 [127.0.0.1]: 450 soft error
log writing disabled
V4NET.0.0.0 [V4NET.0.0.0] Network Error
set_process_info: pppp delivering 10HmaX-0005vi-00: just tried V4NET.0.0.0 [V4NET.0.0.0] for x@y: result DEFER
added retry item for T:V4NET.0.0.0:V4NET.0.0.0:1224: errno=dd more_errno=dd,A flags=2
-locking TESTSUITE/spool/db/wait-smtp.lockfile
-locked TESTSUITE/spool/db/wait-smtp.lockfile
-EXIM_DBOPEN(TESTSUITE/spool/db/wait-smtp)
-returned from EXIM_DBOPEN
-opened hints database TESTSUITE/spool/db/wait-smtp: flags=O_RDWR
-dbfn_read: key=V4NET.0.0.0
-dbfn_write: key=V4NET.0.0.0
set_process_info: pppp delivering 10HmaX-0005vi-00 (just run smtp for x@y in subprocess)
search_tidyup called
set_process_info: pppp delivering 10HmaX-0005vi-00: waiting for a remote delivery subprocess to finish
--- /dev/null
+Exim version x.yz ....
+configuration file is TESTSUITE/test-config
+admin user
+LOG: queue_run MAIN
+ Start queue run: pid=pppp
+locking TESTSUITE/spool/db/retry.lockfile
+>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
+Considering: userx@domain1
+unique = userx@domain1
+userx@domain1: queued for routing
+>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
+routing userx@domain1
+--------> smarthost router <--------
+local_part=userx domain=domain1
+calling smarthost router
+smarthost router called for userx@domain1
+ domain = domain1
+set transport smtp
+queued for smtp transport: local_part = userx
+domain = domain1
+ errors_to=NULL
+ domain_data=NULL localpart_data=NULL
+routed by smarthost router
+ envelope to: userx@domain1
+ transport: smtp
+>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
+After routing:
+ Local deliveries:
+ Remote deliveries:
+ userx@domain1
+ Failed addresses:
+ Deferred addresses:
+locking TESTSUITE/spool/db/retry.lockfile
+locking TESTSUITE/spool/db/wait-smtp.lockfile
+LOG: MAIN
+ => userx@domain1 R=smarthost T=smtp H=thisloop.test.ex [127.0.0.1]
+LOG: MAIN
+ Completed
+locking TESTSUITE/spool/db/retry.lockfile
+>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
+Considering: usery@domain1
+unique = usery@domain1
+LOG: retry_defer MAIN
+ == usery@domain1 routing defer (-51): retry time not reached
+>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
+After routing:
+ Local deliveries:
+ Remote deliveries:
+ Failed addresses:
+ Deferred addresses:
+ usery@domain1
+LOG: queue_run MAIN
+ End queue run: pid=pppp
+>>>>>>>>>>>>>>>> Exim pid=pppp terminating with rc=0 >>>>>>>>>>>>>>>>
250 OK
MAIL FROM:<CALLER@test.ex>
250 Sender OK
-RCPT TO:<userx@domain1>
-450 Temporary error
RCPT TO:<usery@domain1>
-250 OK
-DATA
-354 Go ahead
-Received: from CALLER by the.local.host.name with local (Exim x.yz)
- (envelope-from <CALLER@test.ex>)
- id 10HmbA-0005vi-00; Tue, 2 Mar 1999 09:44:33 +0000
-Message-Id: <E10HmbA-0005vi-00@the.local.host.name>
-From: CALLER_NAME <CALLER@test.ex>
-Date: Tue, 2 Mar 1999 09:44:33 +0000
-
-Test message 2
-.
-250 OK
-MAIL FROM:<CALLER@test.ex>
-250 Sender OK
-RCPT TO:<userx@domain1>
450 Temporary error
QUIT
250 OK
--- /dev/null
++++++++++++++++++++++++++++
+** Failed to open database lock file TESTSUITE/spool/db/wait-smtp.lockfile: No such file or directory
+
+******** SERVER ********
+Listening on port 1224 ...
+Connection request from [127.0.0.1]
+220 ESMTP
+EHLO the.local.host.name
+250-OK
+250 HELP
+MAIL FROM:<CALLER@test.ex>
+250 OK
+RCPT TO:<usery@domain1>
+451 Later, please
+QUIT
+220 OK
+End of script
+Listening on port 1224 ...
+Connection request from [127.0.0.1]
+220 ESMTP
+EHLO the.local.host.name
+250-OK
+250 HELP
+MAIL FROM:<CALLER@test.ex>
+250 OK
+RCPT TO:<userx@domain1>
+250 OK
+DATA
+354 More...
+Received: from CALLER by the.local.host.name with local (Exim x.yz)
+ (envelope-from <CALLER@test.ex>)
+ id 10HmaX-0005vi-00
+ for userx@domain1; Tue, 2 Mar 1999 09:44:33 +0000
+Message-Id: <E10HmaX-0005vi-00@the.local.host.name>
+From: CALLER_NAME <CALLER@test.ex>
+Date: Tue, 2 Mar 1999 09:44:33 +0000
+
+Test message 1
+.
+250 OK
+QUIT
+250 OK
+End of script
git://git.exim.org / exim.git / commitdiff