From: Philip Hazel Date: Fri, 8 Oct 2004 10:38:47 +0000 (+0000) Subject: Start X-Git-Tag: exim-4_50~145 X-Git-Url: https://git.exim.org/users/jgh/exim.git/commitdiff_plain/e05f33e0b79c14608757a60f2f3f8588008355f7 Start --- diff --git a/doc/doc-misc/ABOUT b/doc/doc-misc/ABOUT new file mode 100644 index 000000000..7f6e294c9 --- /dev/null +++ b/doc/doc-misc/ABOUT @@ -0,0 +1,11 @@ +$Cambridge: exim/doc/doc-misc/ABOUT,v 1.1 2004/10/08 10:38:47 ph10 Exp $ + +CVS directory exim/exim-doc/doc-misc +------------------------------------ + +This directory contains some miscellaneous documentation files that do not form +part of Exim distributions, but are related to its maintenance and development. +Those whose names start with "Ext-" are external documents that won't be +modified (and hence have no local CVS Ids). + +End diff --git a/doc/doc-misc/Ext-maildir b/doc/doc-misc/Ext-maildir new file mode 100644 index 000000000..b523dee4a --- /dev/null +++ b/doc/doc-misc/Ext-maildir @@ -0,0 +1,109 @@ +The following information is from the maildir man page of qmail. + +INTRODUCTION + maildir is a structure for directories of incoming mail + messages. It solves the reliability problems that plague + mbox files and mh folders. + +RELIABILITY ISSUES + A machine may crash while it is delivering a message. For + both mbox files and mh folders this means that the message + will be silently truncated. Even worse: for mbox format, + if the message is truncated in the middle of a line, it + will be silently joined to the next message. The mail + transport agent will try again later to deliver the mes- + sage, but it is unacceptable that a corrupted message + should show up at all. In maildir, every message is guar- + anteed complete upon delivery. + + A machine may have two programs simultaneously delivering + mail to the same user. The mbox and mh formats require + the programs to update a single central file. If the pro- + grams do not use some locking mechanism, the central file + will be corrupted. There are several mbox and mh locking + mechanisms, none of which work portably and reliably. In + contrast, in maildir, no locks are ever necessary. Dif- + ferent delivery processes never touch the same file. + + A user may try to delete messages from his mailbox at the + same moment that the machine delivers a new message. For + mbox and mh formats, the user's mail-reading program must + know what locking mechanism the mail-delivery programs + use. In contrast, in maildir, any delivered message can + be safely updated or deleted by a mail-reading program. + + Many sites use Sun's Network Failure System (NFS), presum- + ably because the operating system vendor does not offer + anything else. NFS exacerbates all of the above problems. + Some NFS implementations don't provide any reliable lock- + ing mechanism. With mbox and mh formats, if two machines + deliver mail to the same user, or if a user reads mail + anywhere except the delivery machine, the user's mail is + at risk. maildir works without trouble over NFS. + +THE MAILDIR STRUCTURE + A directory in maildir format has three subdirectories, + all on the same filesystem: tmp, new, and cur. + + Each file in new is a newly delivered mail message. The + modification time of the file is the delivery date of the + message. The message is delivered without an extra UUCP- + style From_ line, without any >From quoting, and without + an extra blank line at the end. The message is normally + in RFC 822 format, starting with a Return-Path line and a + Delivered-To line, but it could contain arbitrary binary + data. It might not even end with a newline. + + Files in cur are just like files in new. The big differ- + ence is that files in cur are no longer new mail: they + have been seen by the user's mail-reading program. + +HOW A MESSAGE IS DELIVERED + The tmp directory is used to ensure reliable delivery, as + discussed here. + + A program delivers a mail message in six steps. First, it + chdir()s to the maildir directory. Second, it stat()s the + name tmp/time.pid.host, where time is the number of sec- + onds since the beginning of 1970 GMT, pid is the program's + process ID, and host is the host name. Third, if stat() + returned anything other than ENOENT, the program sleeps + for two seconds, updates time, and tries the stat() again, + a limited number of times. Fourth, the program creates + tmp/time.pid.host. Fifth, the program NFS-writes the mes- + sage to the file. Sixth, the program link()s the file to + new/time.pid.host. At that instant the message has been + successfully delivered. + + The delivery program is required to start a 24-hour timer + before creating tmp/time.pid.host, and to abort the deliv- + ery if the timer expires. Upon error, timeout, or normal + completion, the delivery program may attempt to unlink() + tmp/time.pid.host. + + NFS-writing means (1) as usual, checking the number of + bytes returned from each write() call; (2) calling fsync() + and checking its return value; (3) calling close() and + checking its return value. (Standard NFS implementations + handle fsync() incorrectly but make up for it by abusing + close().) + +HOW A MESSAGE IS READ + A mail reader operates as follows. + + It looks through the new directory for new messages. Say + there is a new message, new/unique. The reader may freely + display the contents of new/unique, delete new/unique, or + rename new/unique as cur/unique:info. See + http://pobox.com/~djb/maildir.html for the meaning of + info. + + The reader is also expected to look through the tmp direc- + tory and to clean up any old files found there. A file in + tmp may be safely removed if it has not been accessed in + 36 hours. + + It is a good idea for readers to skip all filenames in new + and cur starting with a dot. Other than this, readers + should not attempt to parse filenames. +### diff --git a/doc/doc-misc/Ext-maildir++ b/doc/doc-misc/Ext-maildir++ new file mode 100644 index 000000000..b2fc58045 --- /dev/null +++ b/doc/doc-misc/Ext-maildir++ @@ -0,0 +1,394 @@ + Maildir++ + + In this document: + * HOWTO.maildirquota + * Mission statement + * Definitions and goals + * Contents of a maildirsize + * Calculating maildirsize + * Calculating the quota for a Maildir++ + * Delivering to a Maildir++ + * Reading from a Maildir++ + * Bugs + +HOWTO.maildirquota + + The remaining portion of this document is a technical description of + the maildir quota extension. This section is a brief overview of this + extension. + + What is a maildirquota? + + If you would like to have a quota on your maildir mailboxes, the best + solution is to always use filesystem-based quotas: per-user usage + quotas that is enforced by the operating system. + + This is the best solution when the default Maildir is located in each + account's home directory. This solution will NOT work if Maildirs are + stored elsewhere, or if you have a large virtual domain setup where a + single userid is used to hold many individual Maildirs, one for each + virtual user. + + This extension to the maildir format allows a "voluntary" maildir + quota implementation that does not rely on filesystem-based quotas. + + When maildirquota will not work. + + For this quota mechanism to work, all software that accesses a maildir + must observe this quota protocol. It follows that this quota mechanism + can be easily circumvented if users have direct (shell) access to the + filesystem containing the users' maildirs. + + Furthermore, this quota mechanism is not 100% effective. It is + possible to have a situation where someone may go over quota. This + quota implementation uses a deliverate trade-off. It is necessary to + use some form of locking in order to have a complete bulletproof quota + enforcement, but maildirs mail stores were explicitly designed to + avoid any kind of locking. This quota approach does not use locking, + and the tradeoff is that sometimes it is possible for a few extra + messages to be delivered to the maildir, before the door is + permanently shot. + + For best performance, all maildir clients should support this quota + extension, however there's a wide degree of tolerance here. As long as + the mail delivery agent that puts new messages into a Maildir uses + this extension, the quota will be enforced without excessive + degradation. + + In the worst case scenario, quotas are automatically recalculated + every fifteen minutes. If a maildir goes over quota, and a mail client + that does not support this quota extension removes enough mail from + the maildir, the mail delivery agent will not be immediately informed + that the maildir is now under quota. However, eventually the correct + quota will be recalculated and mail delivery will resume. + + Mail user agents sometimes put messages into the maildir themselves. + Messages added to a maildir by a mail user agent that does not + understand the quota extension will not be immediately counted towards + the overall quota, and may not be counted for an extensive period of + time. Additionally, if there are a lot of messages that have been + added to a maildir from these mail user agents, quota recalculation + may impose non-trivial load on the system, as the quota recalculator + will have to issue the stat system call for each message. + + How to implement the quota + + The best way to do that is to modify your mail server to implement the + protocol defined by this document. Not everyone, of course, has this + ability. Therefore, an alternate approach is available. + + This package creates a very short utility called "deliverquota". It + will NOT be installed anywhere by default, unless this maildir quota + implementation is a part of a larger package, in which case the parent + package may install this utility somewhere. If you obtained the + maildir package separately, you will need to compile it by running the + configure script, then by running make. + + deliverquota takes two arguments. deliverquota reads the message from + standard input, then delivers it to the maildir specified by the first + argument to deliverquota. The second argument specifies the actual + quota for this maildir, as defined elsewhere in this document. + deliverquota will deliver the message to the maildir, making a best + effort not to exceed the stated quota. If the maildir is over quota, + deliverquota terminates with exit code 77. Otherwise, it delivers the + message, updates the quota, and terminates with exit code 0. + + Therefore, proceed as follows: + * Copy deliverquota to some convenient location, say /usr/local/bin. + * Configure your mail server to use deliverquota. For example, if + you use Qmail and your maildirs are all located in $HOME/Maildir, + replace the './Maildir/' argument to qmail-start with the + following: +'| /usr/local/bin/deliverquota ./Maildir 1000000S' + + + + + This sets a one million byte limit on all Maildirs. As I + mentioned, this is meaningless if login access is available, + because the individual account owner can create his own + $HOME/.qmail file, and ignore deliverquota. Note that in this + case, you MUST use apostrophes on the qmail-start command line, in + order to quote this as one argument. + + If you would like to use different quotas for different users, you + will have to put together a separate process or a script that looks up + the appropriate quota for the recipient, and runs deliverquota + specifying the quota. If no login access to the mail server is + available, you can simply create a separate $HOME/.qmail for every + recipient. + + That's pretty much it. If you handle a moderate amount of mail, I have + one more suggestion. For the first couple of weeks, run deliverquota + setting the second argument to an empty string. This disables quota + enforcement, however it still activates certain optimizations that + permit very fast quota recalculation. Messages delivered by + deliverquota have their message size encoded in their filename; this + makes it possible to avoid stat-ing the message in the Maildir, when + recalculating the quota. Then, after most messages in your maildirs + have been delivered by deliverquota, activate the quotas!!! + + maildirquota-enhanced applications + + This is a list of applications that have been enhanced to support the + maildirquota extension: + * maildrop - mail delivery agent/mail filter. + * SqWebmail - webmail CGI binary. + + These applications fall into two classes: + * Mail delivery agents. These applications read some externally + defined table of mail recipients and their maildir quota. + * Mail clients. These applications read maildir quota information + that has been defined by the mail delivery agent. + + Mail clients generally do not need any additional setup in order to + use the maildirquota extension. They will automatically read and + implement any quota specification set by the mail delivery agent. + + On the other hand, mail delivery agents will require some kind of + configuration in order to activate the maildirquota extension for some + or all recipients. The instructions for doing that depends upon the + mail delivery agent. The documentation for the mail delivery agent + should be consulted for additional information. + _________________________________________________________________ + +Mission statement + + Maildir++ is a mail storage structure that's based on the Maildir + structure, first used in the Qmail mail server. Actually, Maildir++ is + just a minor extension to the standard Maildir structure. + + For more information, see http://www.qmail.org/man/man5/maildir.html. + I am not going to include the definition of a Maildir in this + document. Consider it included right here. This document only + describes the differences. + + Maildir++ adds a couple of things to a standard Maildir: folders and + quotas. + + Quotas enforce a maximum allowable size of a Maildir. In many + situations, using the quota mechanism of the underlying filesystem + won't work very well. If a filesystem quota mechanism is used, then + when a Maildir goes over quota, Qmail does not bounce additional mail, + but keeps it queued, changing one bad situation into another bad + situation. Not only know you have an account that's backed up, but now + your queue starts to back up too. + +Definitions, and goals + + Maildir++ and Maildir shall be completely interchangeable. A Maildir++ + client will be able to use a standard Maildir, automatically + "upgrading" it in the process. A Maildir client will be able to use a + Maildir++ just like a regular Maildir. Of course, a plain Maildir + client won't be able to enforce a quota, and won't be able to access + messages stored in folders. + + Folders are created as subdirectories under the main Maildir. The name + of the subdirectory always starts with a period. For example, a folder + named "Important" will be a subdirectory called ".Important". You + can't have subdirectories that start with two periods. + + A Maildir++ client ignores anything in the main Maildir that starts + with a period, but is not a subdirectory. + + Each subdirectory is a fully-fledged Maildir of its own, that is you + have .Important/tmp, .Important/new, and .Important/cur. Everything + that applies to the main Maildir applies equally well to the + subdirectory, including automatically cleaning up old files in tmp. A + Maildir++ enhancement is that a message can be moved between folders + and/or the main Maildir simply by moving/renaming the file (into the + cur subdirectory of the destination folder). Therefore, the entire + Maildir++ must reside on the same filesystem. + + Within each subdirectory there's an empty file, maildirfolder. Its + existence tells the mail delivery agent that this Maildir is a really + a folder underneath a parent Maildir++. + + Only one special folder is reserved: Trash (subdirectory .Trash). + Instead of marking deleted messages with the D flag, Maildir++ clients + move the message into the Trash folder. Maildir++ readers are + responsible for expunging messages from Trash after a system-defined + retention interval. + + When a Maildir++ reader sees a message marked with a D flag it may at + its option: remove the message immediately, move it into Trash, or + ignore it. + + Can folders have subfolders, defined in a recursive fashion? The + answer is no. If you want to have a client with a hierarchy of + folders, emulate it. Pick a hierarchy separator character, say ":". + Then, folder foo/bar is subdirectory .foo:bar. + + This is all that there's to say about folders. The rest of this + document deals with quotas. + + The purpose of quotas is to temporarily disable a Maildir, if it goes + over the quota. There is one and only major goal that this quota + implementation tries to achieve: + * Place as little overhead as possible on the mail system that's + delivering to the Maildir++ + + That's it. To achieve that goal, certain compromises are made: + * Mail delivery will stop as soon as possible after Maildir++'s size + goes over quota. Certain race conditions may happen with Maildir++ + going a lot over quota, in rare circumstances. That is taken into + account, and the situation will eventually resolve itself, but you + should not simply take your systemwide quota, multiply it by the + number of mail accounts, and allocate that much disk space. Always + leave room to spare. + * How well the quota mechanism will work will depend on whether or + not everything that accesses the Maildir++ is a Maildir++ client. + You can have a transition period where some of your mail clients + are just Maildir clients, and things should run more or less well. + There will be some additional load because the size of the Maildir + will be recalculated more often, but the additional load shouldn't + be noticeable. + + This won't be a perfect solution, but it will hopefully be good + enough. Maildirs are simply designed to rely on the filesystem to + enforce individual quotas. If a filesystem-based quota works for you, + use it. + + A Maildir++ may contain the following additional file: maildirsize. + +Contents of maildirsize + + maildirsize contains two or more lines terminated by newline + characters. + + The first line contains a copy of the quota definition as used by the + system's mail server. Each application that uses the maildir must know + what it's quota is. Instead of configuring each application with the + quota logic, and making sure that every application's quota definition + for the same maildir is exactly the same, the quota specification used + by the system mail server is saved as the first line of the + maildirsize file. All other application that enforce the maildir quota + simply read the first line of maildirsize. + + The quota definition is a list, separate by commas. Each member of the + list consists of an integer followed by a letter, specifying the + nature of the quota. Currently defined quota types are 'S' - total + size of all messages, and 'C' - the maximum count of messages in the + maildir. For example, 10000000S,1000C specifies a quota of 10,000,000 + bytes or 1,000 messages, whichever comes first. + + All remaining lines all contain two integers separated by a single + space. The first integer is interpreted as a byte count. The second + integer is interpreted as a file count. A Maildir++ writer can add up + all byte counts and file counts from maildirsize and enforce a quota + based either on number of messages or the total size of all the + messages. + +Calculating maildirsize + + In most cases, changes to maildirsize are recorded by appending an + additional line. Under some conditions maildirsize has to be + recalculated from scratch. These conditions are defined later. This is + the procedure that's used to recalculate maildirsize: + 1. If we find a maildirfolder within the directory, we're delivering + to a folder, so back up to the parent directory, and start again. + 2. Read the contents of the new and cur subdirectories. Also, read + the contents of the new and cur subdirectories in each Maildir++ + folder, except Trash. Before reading each subdirectory, stat() the + subdirectory itself, and keep track of the latest timestamp you + get. + 3. If the filename of each message is of the form xxxxx,S=nnnnn or + xxxxx,S=nnnnn:xxxxx where "xxxxx" represents arbitrary text, then + use nnnnn as the size of the file (which will be conveniently + recorded in the filename by a Maildir++ writer, within the + conventions of filename naming in a Maildir). If the message was + not written by a Maildir++ writer, stat() it to obtain the message + size. If stat() fails, a race condition removed the file, so just + ignore it and move on to the next one. + 4. When done, you have the grand total of the number of messages and + their total size. Create a new maildirsize by: creating the file + in the tmp subdirectory, observing the conventions for writing to + a Maildir. Then rename the file as maildirsize.Afterwards, stat + all new and cur subdirectories again. If you find a timestamp + later than the saved timestamp, REMOVE maildirsize. + 5. Before running this calculation procedure, the Maildir++ user + wanted to know the size of the Maildir++, so return the calculated + values. This is done even if maildirsize was removed. + +Calculating the quota for a Maildir++ + + This is the procedure for reading the contents of maildirsize for the + purpose of determine if the Maildir++ is over quota. + 1. If maildirsize does not exist, or if its size is at least 5120 + bytes, recalculate it using the procedure defined above, and use + the recalculated numbers. Otherwise, read the contents of + maildirsize, and add up the totals. + 2. The most efficient way of doing this is to: open maildirsize, then + start reading it into a 5120 byte buffer (some broken NFS + implementations may return less than 5120 bytes read even before + reaching the end of the file). If we fill it, which, in most + cases, will happen with one read, close it, and run the + recalculation procedure. + 3. In many cases the quota calculation is for the purpose of adding + or removing messages from a Maildir++, so keep the file descriptor + to maildirsize open. A file descriptor will not be available if + quota recalculation ended up removing maildirsize due to a race + condition, so the caller may or may not get a file descriptor + together with the Maildir++ size. + 4. If the numbers we got indicated that the Maidlir++ is over quota, + some additional logic is in order: if we did not recalculate + maildirsize, if the numbers in maildirsize indicated that we are + over quota, then if maildirsize was more than one line long, or if + the timestamp on maildirsize indicated that it's at least 15 + minutes old, throw out the totals, and recalculate maildirsize + from scratch. + + Eventually the 5120 byte limitation will always cause maildirsize to + be recalculated, which will compensate for any race conditions which + previously threw off the totals. Each time a message is delivered or + removed from a Maildir++, one line is added to maildirsize (this is + described below in greater detail). Most messages are less than 10K + long, so each line appended to maildirsize will be either between + seven and nine bytes long (four bytes for message count, space, digit + 1, newline, optional minus sign in front of both counts if the message + was removed). This results in about 640 Maildir++ operations before a + recalculation is forced. Since most messages are added once and + removed once from a Maildir, expect recalculation to happen + approximately every 320 messages, keeping the overhead of a + recalculation to a minimum. Even if most messages include large + attachments, most attachments are less than 100K long, which brings + down the average recalculation frequency to about 150 messages. + + Also, the effect of having non-Maildir++ clients accessing the + Maildir++ is reduced by forcing a recalculation when we're potentially + over quota. Even if non-Maildir++ clients are used to remove messages + from the Maildir, the fact that the Maildir++ is still over quota will + be verified every 15 minutes. + +Delivering to a Maildir++ + + Delivering to a Maildir++ is like delivering to a Maildir, with the + following exceptions: + 1. Follow the usual Maildir conventions for naming the filename used + to store the message, except that append ,S=nnnnn to the name of + the file, where nnnnn is the size of the file. This eliminates the + need to stat() most messages when calculating the quota. If the + size of the message is not known at the beginning, append ,S=nnnnn + when renaming the message from tmp to new. + 2. As soon as the size of the message is known (hopefully before it + is written into tmp), calculate Maildir++'s quota, using the + procedure defined previously. If the message is over quota, back + out, cleaning up anything that was created in tmp. + 3. If a file descriptor to maildirsize was opened for us, after + moving the file from tmp to new append a line to the file + containing the message size, and "1". + +Reading from a Maildir++ + + Maildir++ readers should mind the following additional tasks: + 1. Make sure to create the maildirfolder file in any new folders + created within the Maildir++. + 2. When moving a message to the Trash folder, append a line to + maildirsize, containing a negative message size and a '-1'. + 3. When moving a message from the Trash folder, follow the steps + described in "Delivering to Maildir++", as far as quota logic + goes. That is, refuse to move messages out of Trash if the + Maildir++ is over quota. + 4. Moving a message between other folders carries no additional + requirements. + diff --git a/doc/doc-misc/Ext-mbx-locking b/doc/doc-misc/Ext-mbx-locking new file mode 100644 index 000000000..f1b0523f6 --- /dev/null +++ b/doc/doc-misc/Ext-mbx-locking @@ -0,0 +1,400 @@ + UNIX Advisory File Locking Implications on c-client + Mark Crispin, 28 November 1995 + + + THIS DOCUMENT HAS BEEN UPDATED TO REFLECT THE CODE IN THE + IMAP-4 TOOLKIT AS OF NOVEMBER 28, 1995. SOME STATEMENTS + IN THIS DOCUMENT DO NOT APPLY TO EARLIER VERSIONS OF THE + IMAP TOOLKIT. + +INTRODUCTION + + Advisory locking is a mechanism by which cooperating processes +can signal to each other their usage of a resource and whether or not +that usage is critical. It is not a mechanism to protect against +processes which do not cooperate in the locking. + + The most basic form of locking involves a counter. This counter +is -1 when the resource is available. If a process wants the lock, it +executes an atomic increment-and-test-if-zero. If the value is zero, +the process has the lock and can execute the critical code that needs +exclusive usage of a resource. When it is finished, it sets the lock +back to -1. In C terms: + + while (++lock) /* try to get lock */ + invoke_other_threads (); /* failed, try again */ + . + . /* critical code here */ + . + lock = -1; /* release lock */ + + This particular form of locking appears most commonly in +multi-threaded applications such as operating system kernels. It +makes several presumptions: + (1) it is alright to keep testing the lock (no overflow) + (2) the critical resource is single-access only + (3) there is shared writeable memory between the two threads + (4) the threads can be trusted to release the lock when finished + + In applications programming on multi-user systems, most commonly +the other threads are in an entirely different process, which may even +be logged in as a different user. Few operating systems offer shared +writeable memory between such processes. + + A means of communicating this is by use of a file with a mutually +agreed upon name. A binary semaphore can be passed by means of the +existance or non-existance of that file, provided that there is an +atomic means to create a file if and only if that file does not exist. +In C terms: + + /* try to get lock */ + while ((fd = open ("lockfile",O_WRONLY|O_CREAT|O_EXCL,0666)) < 0) + sleep (1); /* failed, try again */ + close (fd); /* got the lock */ + . + . /* critical code here */ + . + unlink ("lockfile"); /* release lock */ + + This form of locking makes fewer presumptions, but it still is +guilty of presumptions (2) and (4) above. Presumption (2) limits the +ability to have processes sharing a resource in a non-conflicting +fashion (e.g. reading from a file). Presumption (4) leads to +deadlocks should the process crash while it has a resource locked. + + Most modern operating systems provide a resource locking system +call that has none of these presumptions. In particular, a mechanism +is provided for identifying shared locks as opposed to exclusive +locks. A shared lock permits other processes to obtain a shared lock, +but denies exclusive locks. In other words: + + current state want shared want exclusive + ------------- ----------- -------------- + unlocked YES YES + locked shared YES NO + locked exclusive NO NO + + Furthermore, the operating system automatically relinquishes all +locks held by that process when it terminates. + + A useful operation is the ability to upgrade a shared lock to +exclusive (provided there are no other shared users of the lock) and +to downgrade an exclusive lock to shared. It is important that at no +time is the lock ever removed; a process upgrading to exclusive must +not relenquish its shared lock. + + Most commonly, the resources being locked are files. Shared +locks are particularly important with files; multiple simultaneous +processes can read from a file, but only one can safely write at a +time. Some writes may be safer than others; an append to the end of +the file is safer than changing existing file data. In turn, changing +a file record in place is safer than rewriting the file with an +entirely different structure. + + +FILE LOCKING ON UNIX + + In the oldest versions of UNIX, the use of a semaphore lockfile +was the only available form of locking. Advisory locking system calls +were not added to UNIX until after the BSD vs. System V split. Both +of these system calls deal with file resources only. + + Most systems only have one or the other form of locking. AIX +emulates the BSD form of locking as a jacket into the System V form. +Ultrix and OSF/1 implement both forms. + +BSD + + BSD added the flock() system call. It offers capabilities to +acquire shared lock, acquire exclusive lock, and unlock. Optionally, +the process can request an immediate error return instead of blocking +when the lock is unavailable. + + +FLOCK() BUGS + + flock() advertises that it permits upgrading of shared locks to +exclusive and downgrading of exclusive locks to shared, but it does so +by releasing the former lock and then trying to acquire the new lock. +This creates a window of vulnerability in which another process can +grab the exclusive lock. Therefore, this capability is not useful, +although many programmers have been deluded by incautious reading of +the flock() man page to believe otherwise. This problem can be +programmed around, once the programmer is aware of it. + + flock() always returns as if it succeeded on NFS files, when in +fact it is a no-op. There is no way around this. + + Leaving aside these two problems, flock() works remarkably well, +and has shown itself to be robust and trustworthy. + +SYSTEM V/POSIX + + System V added new functions to the fnctl() system call, and a +simple interface through the lockf() subroutine. This was +subsequently included in POSIX. Both offer the facility to apply the +lock to a particular region of the file instead of to the entire file. +lockf() only supports exclusive locks, and calls fcntl() internally; +hence it won't be discussed further. + + Functionally, fcntl() locking is a superset of flock(); it is +possible to implement a flock() emulator using fcntl(), with one minor +exception: it is not possible to acquire an exclusive lock if the file +is not open for write. + + The fcntl() locking functions are: query lock station of a file +region, lock/unlock a region, and lock/unlock a region and block until +have the lock. The locks may be shared or exclusive. By means of the +statd and lockd daemons, fcntl() locking is available on NFS files. + + When statd is started at system boot, it reads its /etc/state +file (which contains the number of times it has been invoked) and +/etc/sm directory (which contains a list of all remote sites which are +client or server locking with this site), and notifies the statd on +each of these systems that it has been restarted. Each statd then +notifies the local lockd of the restart of that system. + + lockd receives fcntl() requests for NFS files. It communicates +with the lockd at the server and requests it to apply the lock, and +with the statd to request it for notification when the server goes +down. It blocks until all these requests are completed. + + There is quite a mythos about fcntl() locking. + + One religion holds that fcntl() locking is the best thing since +sliced bread, and that programs which use flock() should be converted +to fcntl() so that NFS locking will work. However, as noted above, +very few systems support both calls, so such an exercise is pointless +except on Ultrix and OSF/1. + + Another religion, which I adhere to, has the opposite viewpoint. + + +FCNTL() BUGS + + For all of the hairy code to do individual section locking of a +file, it's clear that the designers of fcntl() locking never +considered some very basic locking operations. It's as if all they +knew about locking they got out of some CS textbook with not +investigation of real-world needs. + + It is not possible to acquire an exclusive lock unless the file +is open for write. You could have append with shared read, and thus +you could have a case in which a read-only access may need to go +exclusive. This problem can be programmed around once the programmer +is aware of it. + + If the file is opened on another file designator in the same +process, the file is unlocked even if no attempt is made to do any +form of locking on the second designator. This is a very bad bug. It +means that an application must keep track of all the files that it has +opened and locked. + + If there is no statd/lockd on the NFS server, fcntl() will hang +forever waiting for them to appear. This is a bad bug. It means that +any attempt to lock on a server that doesn't run these daemons will +hang. There is no way for an application to request flock() style +``try to lock, but no-op if the mechanism ain't there''. + + There is a rumor to the effect that fcntl() will hang forever on +local files too if there is no local statd/lockd. These daemons are +running on mailer.u, although they appear not to have much CPU time. +A useful experiment would be to kill them and see if imapd is affected +in any way, but I decline to do so without an OK from UCS! ;-) If +killing statd/lockd can be done without breaking fcntl() on local +files, this would become one of the primary means of dealing with this +problem. + + The statd and lockd daemons have quite a reputation for extreme +fragility. There have been numerous reports about the locking +mechanism being wedged on a systemwide or even clusterwide basis, +requiring a reboot to clear. It is rumored that this wedge, once it +happens, also blocks local locking. Presumably killing and restarting +statd would suffice to clear the wedge, but I haven't verified this. + + There appears to be a limit to how many locks may be in use at a +time on the system, although the documentation only mentions it in +passing. On some of their systems, UCS has increased lockd's ``size +of the socket buffer'', whatever that means. + +C-CLIENT USAGE + + c-client uses flock(). On System V systems, flock() is simulated +by an emulator that calls fcntl(). This emulator is provided by some +systems (e.g. AIX), or uses c-client's flock.c module. + + +BEZERK AND MMDF + + Locking in the traditional UNIX formats was largely dictated by +the status quo in other applications; however, additional protection +is added against inadvertantly running multiple instances of a +c-client application on the same mail file. + + (1) c-client attempts to create a .lock file (mail file name with +``.lock'' appended) whenever it reads from, or writes to, the mail +file. This is an exclusive lock, and is held only for short periods +of time while c-client is actually doing the I/O. There is a 5-minute +timeout for this lock, after which it is broken on the presumption +that it is a stale lock. If it can not create the .lock file due to +an EACCES (protection failure) error, it once silently proceeded +without this lock; this was for systems which protect /usr/spool/mail +from unprivileged processes creating files. Today, c-client reports +an error unless it is built otherwise. The purpose of this lock is to +prevent against unfavorable interactions with mail delivery. + + (2) c-client applies a shared flock() to the mail file whenever +it reads from the mail file, and an exclusive flock() whenever it +writes to the mail file. This lock is freed as soon as it finishes +reading. The purpose of this lock is to prevent against unfavorable +interactions with mail delivery. + + (3) c-client applies an exclusive flock() to a file on /tmp +(whose name represents the device and inode number of the file) when +it opens the mail file. This lock is maintained throughout the +session, although c-client has a feature (called ``kiss of death'') +which permits c-client to forcibly and irreversibly seize the lock +from a cooperating c-client application that surrenders the lock on +demand. The purpose of this lock is to prevent against unfavorable +interactions with other instances of c-client (rewriting the mail +file). + + Mail delivery daemons use lock (1), (2), or both. Lock (1) works +over NFS; lock (2) is the only one that works on sites that protect +/usr/spool/mail against unprivileged file creation. Prudent mail +delivery daemons use both forms of locking, and of course so does +c-client. + + If only lock (2) is used, then multiple processes can read from +the mail file simultaneously, although in real life this doesn't +really change things. The normal state of locks (1) and (2) is +unlocked except for very brief periods. + + +TENEX AND MTX + + The design of the locking mechanism of these formats was +motivated by a design to enable multiple simultaneous read/write +access. It is almost the reverse of how locking works with +bezerk/mmdf. + + (1) c-client applies a shared flock() to the mail file when it +opens the mail file. It upgrades this lock to exclusive whenever it +tries to expunge the mail file. Because of the flock() bug that +upgrading a lock actually releases it, it will not do so until it has +acquired an exclusive lock (2) first. The purpose of this lock is to +prevent against expunge taking place while some other c-client has the +mail file open (and thus knows where all the messages are). + + (2) c-client applies a shared flock() to a file on /tmp (whose +name represents the device and inode number of the file) when it +parses the mail file. It applies an exclusive flock() to this file +when it appends new mail to the mail file, as well as before it +attempts to upgrade lock (1) to exclusive. The purpose of this lock +is to prevent against data being appended while some other c-client is +parsing mail in the file (to prevent reading of incomplete messages). +It also protects against the lock-releasing timing race on lock (1). + +OBSERVATIONS + + In a perfect world, locking works. You are protected against +unfavorable interactions with the mailer and against your own mistake +by running more than one instance of your mail reader. In tenex/mtx +formats, you have the additional benefit that multiple simultaneous +read/write access works, with the sole restriction being that you +can't expunge if there are any sharers of the mail file. + + If the mail file is NFS-mounted, then flock() locking is a silent +no-op. This is the way BSD implements flock(), and c-client's +emulation of flock() through fcntl() tests for NFS files and +duplicates this functionality. There is no locking protection for +tenex/mtx mail files at all, and only protection against the mailer +for bezerk/mmdf mail files. This has been the accepted state of +affairs on UNIX for many sad years. + + If you can not create .lock files, it should not affect locking, +since the flock() locks suffice for all protection. This is, however, +not true if the mailer does not check for flock() locking, or if the +the mail file is NFS-mounted. + + What this means is that there is *no* locking protection at all +in the case of a client using an NFS-mounted /usr/spool/mail that does +not permit file creation by unprivileged programs. It is impossible, +under these circumstances, for an unprivileged program to do anything +about it. Worse, if EACCES errors on .lock file creation are no-op'ed +, the user won't even know about it. This is arguably a site +configuration error. + + The problem with not being able to create .lock files exists on +System V as well, but the failure modes for flock() -- which is +implemented via fcntl() -- are different. + + On System V, if the mail file is NFS-mounted and either the +client or the server lacks a functioning statd/lockd pair, then the +lock attempt would have hung forever if it weren't for the fact that +c-client tests for NFS and no-ops the flock() emulator in this case. +Systemwide or clusterwide failures of statd/lockd have been known to +occur which cause all locks in all processes to hang (including +local?). Without the special NFS test made by c-client, there would +be no way to request BSD-style no-op behavior, nor is there any way to +determine that this is happening other than the system being hung. + + The additional locking introduced by c-client was shown to cause +much more stress on the System V locking mechanism than has +traditionally been placed upon it. If it was stressed too far, all +hell broke loose. Fortunately, this is now past history. + +TRADEOFFS + + c-client based applications have a reasonable chance of winning +as long as you don't use NFS for remote access to mail files. That's +what IMAP is for, after all. It is, however, very important to +realize that you can *not* use the lock-upgrade feature by itself +because it releases the lock as an interim step -- you need to have +lock-upgrading guarded by another lock. + + If you have the misfortune of using System V, you are likely to +run into problems sooner or later having to do with statd/lockd. You +basically end up with one of three unsatisfactory choices: + 1) Grit your teeth and live with it. + 2) Try to make it work: + a) avoid NFS access so as not to stress statd/lockd. + b) try to understand the code in statd/lockd and hack it + to be more robust. + c) hunt out the system limit of locks, if there is one, + and increase it. Figure on at least two locks per + simultaneous imapd process and four locks per Pine + process. Better yet, make the limit be 10 times the + maximum number of processes. + d) increase the socket buffer (-S switch to lockd) if + it is offered. I don't know what this actually does, + but giving lockd more resources to do its work can't + hurt. Maybe. + 3) Decide that it can't possibly work, and turn off the + fcntl() calls in your program. + 4) If nuking statd/lockd can be done without breaking local + locking, then do so. This would make SVR4 have the same + limitations as BSD locking, with a couple of additional + bugs. + 5) Check for NFS, and don't do the fcntl() in the NFS case. + This is what c-client does. + + Note that if you are going to use NFS to access files on a server +which does not have statd/lockd running, your only choice is (3), (4), +or (5). Here again, IMAP can bail you out. + + These problems aren't unique to c-client applications; they have +also been reported with Elm, Mediamail, and other email tools. + + Of the other two SVR4 locking bugs: + + Programmer awareness is necessary to deal with the bug that you +can not get an exclusive lock unless the file is open for write. I +believe that c-client has fixed all of these cases. + + The problem about opening a second designator smashing any +current locks on the file has not been addressed satisfactorily yet. +This is not an easy problem to deal with, especially in c-client which +really doesn't know what other files/streams may be open by Pine. + + Aren't you so happy that you bought an System V system? diff --git a/doc/doc-misc/LongTermIssues b/doc/doc-misc/LongTermIssues new file mode 100644 index 000000000..808ba3f8c --- /dev/null +++ b/doc/doc-misc/LongTermIssues @@ -0,0 +1,200 @@ +$Cambridge: exim/doc/doc-misc/LongTermIssues,v 1.1 2004/10/08 10:38:47 ph10 Exp $ + +Exim Long Term Issues +--------------------- + +I restarted this list from scratch for Exim 4. I amalgamated it with another +list when creating the CVS repository (October 2004). But it still probably +needs a substantial spring clean. Some of it is very old now. + + +AUTOCONF +-------- + +Somebody once tried to \(autoconf)\ Exim, but found it too big a job. I now +have some experience with using \(autoconf)\ for PCRE, and I think maybe some +use could be made of it. I don't, however, believe that \(all)\ Exim build-time +configuration should be done that way. The reason is that, unlike something +like PCRE, there is quite a lot of information that is "user choice". Giving it +all as options to a \(configure)\ command does not seem the best way of doing +things. + +Whenever I build something that needs more than a couple of obvious options to +\(configure)\, I always save them in a file anyway, so I know what I did for +next time. Therefore, I think it is sensible to retain the current Local file +structure for all the user choice configuration. + +However, it might be helpful to use \(autoconf)\ to dig out various bits of +information about the operating system. At present, the \(OS/Makefile-*)\ files +have hard-wired settings, and maybe this information could be figured out by +running \(autoconf)\, which would save having to keep maintaining these files. + +I would arrange things so that \(configure)\ is run automatically the first +time that \(make)\ is run, but it would be possible to run it manually first, +to override defaults. (For example, if you have both \(cc)\ and \(gcc)\ +installed on your system, as I do, you need to be able to specify which to +use.) I will need to do some experiments to see exactly how this would work. + + +EXIMON and other utilities +-------------------------- + +. Consider optionally making it possible to link with something other than + Athena widgets - for example, gtk. Or indeed re-write the whole thing! + + +GENERAL +------- + +. Convert os.c into a directory of separate functions, with the macro + switches defined elsewhere. Then make it into a library. + +. Use a pointer to an address structure for expanding $domain etc, to make it + easier to save/restore this collection of variables. But note that $domain + and $local_part aren't always in an address. Check out when these are set. + Note also the new $address_data possibility. + +. Spool_in and spool_out - speed up by using a table? + +. Find a more compact way of encoding the options interpretation, and also of + checking for incompatible options. + +. Find a more compact way of passing an open SMTP channel without having + to use options. What about the TLS state information? Could use a pipe to + pass more data. + +. Some people have suggested separately loadable modules. But do all systems + have them? Is this going too far for just a few specialist users? In + particular, people want to be able to replace the logging with his own code. + Can we arrange this without going for the separately loaded modules? (cf the + incoming checking code.) + +. SIGHUP the daemon - don't close the sockets; instead pass a list of them + somewhere for the new daemon to pick up. Iff started by exim or root, of + course. There might be quite a long list of them - argv might not be the best + idea. If this were done, then a non-setuid exim daemon could be SIGHUPped. + +. Parallel deliveries. Currently dead host information doesn't get propagated + between them very well. Is there anyway this could be improved? + +. In some environments the use of gethostbyname() seems to cause problems. + Check out its use, and see if having a "force DNS" option could be helpful. + But people would have to know what they were doing. + +. accept_max_per_host is a slow, linear search. If smtp_accept_max is large, + this can be very slow. Is there some way we can speed this up? Some kind of + index based on the IP address? Remember, this is in the daemon, so it must + not consume store. + +. Change the names of all the pcre_ stuff to, say, PCRE_ so that Exim can be + linked with libraries or whatever that also use an external PCRE library. + +. Look at code in pidentd for running Exim in wait mode from inetd and re-using + the socket. This would allow it to run more tidily as non-root. + +. Think up some scheme for checking for orphan files in the spool directories. + Perhaps -bp should always do it, but it would be nice to have it done + automatically now and again. Maybe we just leave this for a cron job? Perhaps + a new -bx, e.g. -bpck or something. Better, perhaps, is a separate Perl + script. Orphan = a file that is over 24h old (or 1s when test harness) and + either doesn't end in -D or -H, or is a -D without a matching -H (or vice + versa). + +. Make set_process_info buffer bigger, and put the overflowed message at the + end, thereby leaving the start. + +. Swamping with delays in checking for reserved hosts - the connections are + counted in the total allowed. Can we improve on this somehow? Maybe shared + memory can help here. Think about different states and different limits. + +. Lists that must use colons: can we check for other cases, and fix them up + before passing them on? Is it worth it? + +. Linux for S/390 - create configuration? + +. Process receiving error message fails - can we get more info, such as the + stdout/stderr? + +. dbmbuild - if renaming one of .dir/.pag fails, reinstate the other. Should + there be a lock? + +. Write a script to check for format problems in the source - formats that are + not fixed strings and are built from outside code. + +. freeze_tell: Don't if message is a bounce message containing From: the local + machine - even if the bounce comes from another host. + +. Add additional data into the "frozen" log message at end of delivery, e.g. if + remote host was the local host or whatever. At least some cross referencing. + +. Someone had a requirement to install the Exim binary in a different place to + the utilities, etc. Also, for different builds on the same host and + architecture. + +. Include (part of?) the ppid in the message id? Or a random number? + +. Re-implement the code in readconf that reads error names for retry rules. + Make it use a table for most of the error types. Then see if we can usefully + add any additional error types. + +. Should there be "exim -bP acls" etc? It would mean inventing some kind of + "hide" facility within the ACL syntax. + +. VERY LONG TERM: the message ID is too small now, with the recent changes to + cram in the sub-second time. It would be a big project to extend it; Exim + would have to recognize both forms for a while, and become stable, before + generating the new form. Probably a runtime switch needed. The new form needs + at least microsecond time (or more?) and should probably cope with 64-bit + pids, just to be safe (or leave expansion space that could be used for that). + It should also be able to hold big enough things in base 36. + +. Take a look at libexec. + +. Sort out the stcncpy/strlcpy issue once and for all. Time things. + +. Error in transport filter. See test 407. All 3 processes see errors - which + one should be noticed? Transport_filter_temp_errors may be needed. + +. Think about 5xx thresholds -- too many and you're out. What about 4xx? + +. autoreply - should it call /usr/sbin/sendmail? Provide a way of not passing + -C and -D when creating the message ('cause it won't be privileged). + +. Strings containing \000 - anything we can do? + +. OpenSSL - can we pass an opened file for certificate? Repeatedly? + Otherwise pre-initialize while root? There do seem to be functions for + manipulating certificates, but documentation is scarce. Can we just load the + certificate in as root in the server? + +. Consider using poll() to close unwanted fds. Is this efficient? Perhaps it + doesn't matter for the daemon. + +. On a 64-bit system there are some cast warnings for casting addresses to + ints. Either we must find a way of not warning, or we'll have to use unions + to get round it. + +. Run splint on the source? + +. It has been suggested that rejection because not authenticated should use + 530 and not 550, but this is hard to detect because of the way ACLs work. + +. When there is a sender verify failure, $acl_verify_message contains "sender + verify failed", not the details of the failure. Should this change? Some of + the waffly details are added later in smtp_in.c. In the ACL that text is in + sender_verified_failed->user_message. + +. An empty string for a transport filter currently causes an error. Should it + ignore? Tricky because of special expansion rules for commands. + +. GFDL for documentation (www.gnu.org/licenses/fdl.html)? The 1.2 version of + this licence is still quite new (it is dated November 2002) so I think + waiting for reaction/opinion is the best plan. There are Debian concerns + about this licence. At very least, no Invariant Sections and no Cover Texts + can be used. + +. Allow $recipients in other places. Not clear what this value should be if, + say, the system filter has overridden them. Default would be envelope + recipients, as now. + +End diff --git a/doc/doc-misc/RFC.conform b/doc/doc-misc/RFC.conform new file mode 100644 index 000000000..2fc57cdf2 --- /dev/null +++ b/doc/doc-misc/RFC.conform @@ -0,0 +1,401 @@ +$Cambridge: exim/doc/doc-misc/RFC.conform,v 1.1 2004/10/08 10:38:47 ph10 Exp $ + +Conformance with RFCs +--------------------- + +Exim is written to follow the rules laid down in the RFCs. However, there are +some circumstances where it either extends what is specified, or chooses not to +follow them strictly, for various reasons. Sometimes variations are controlled +by an option, which may default on or off. This document lists the variations +from the latest email RFCs, and discusses their background and implications. + +Last Updated: 25 January 1999 + + +1. RFC 822 +---------- + +The original specification of the format of Internet mail messages is RFC 822, +later clarified and modified by RFC 1123. At the time of writing (January 1999) +a new RFC (currently known as draft-ietf-drums-msg-fmt-07) which updates and +consolidates all the material related to the message format is at a late stage +of drafting, and is expected to become an Internet Standard in due course. + +The following is (I hope) a complete list of major variations from the draft +RFC. References in square brackets are to the -07 draft. + + +1.1 Line termination [2.1, 2.3] +------------------------------- + +[Lines are terminated by CRLF; isolated CR and LF are not permitted.] + +The CRLF requirement has to be interpreted carefully, because the RFC also says +that it does not cover the internal format "used by sites". Exim keeps messages +on its spool in Unix format, using only LF as the line terminator, and also +does local deliveries using only LF. I believe this is compliant with the RFC, +as these are both "internal formats". + +Messages sent out by SMTP have CRLF line terminators. However, isolated CR +characters are treated as any other data characters, because Exim is eight-bit +clean (see 1.2 below). + +See 2.1 below for a discussion of line terminators in incoming messages. + + +1.2 Eight-bit characters [2.1] +------------------------------ + +[Messages consist of 7-bit characters.] + +Exim is eight-bit clean. It does not do any processing of the characters in the +body of a message. + + +1.3 Maximum line length [2.1, 2.3] +---------------------------------- + +[The maximum length of a line is 998 characters.] + +Exim does not enforce any limit on line length. + + +1.4 The "phrase" part of an address [3.4] +----------------------------------------- + +[The phrase is a sequence of "words"; a word is an "atom" or a quoted string.] + +The characters that can be used in an "atom" do not include the full stop +(dot, period). Thus a header line such as + + To: John Q. Public + +is syntactically invalid under a strict interpretation of the RFC because the +dot in the phrase part is not quoted. However, many MTAs do not enforce this +restriction, so Exim was changed to be relaxed about it as well. In fact, the +draft RFC is moving towards allowing this. In section [4.1], which is defining +"obsolete" syntax that programs must accept (but not generate), it says this: + + The period character is added to obs-phrase. + + Note: The period character in obs-phrase is not a form that was allowed + in earlier versions of this or any other standard. Period (nor any other + character from specials) was not allowed in phrase because it introduced + a parsing difficulty distinguishing between phrases and portions of an + addr-spec (see section 4.4). It appears here because the period + character is currently used in many messages in the display-name portion + of addresses, especially for initials in names, and therefore must be + interpreted properly. In the future, period may appear in the regular + syntax of phrase. + + +1.5 Source routed addresses [4.4] +--------------------------------- + +[Source routed addresses are always enclosed in <>.] + +Source routed addresses are declared obsolete in the draft RFC, but MTAs are +still required to handle them. Strictly, a source-routed address must be +enclosed in <> characters, so a header such as + + From: @a,@b:c@d + +is syntactally invalid. Exim does not enforce this restriction. + + +1.6 Local parts [3.4.1] +----------------------- + +[Dots in unquoted local parts may not be consecutive or at either end.] + +Exim allows unquoted local parts to begin or end with a dot (period, full +stop), and it also permits two consecutive dots in a local part. + + + +2. RFC 821 +---------- + +The original specification of SMTP is RFC 821, later clarified and modified by +RFC 1123. Domain name system requirements and their implications for mail are +covered in RFCs 1035 and 974. A scheme for extending the SMTP protocol is +described in RFC 1869, and there are subsequent RFCs specifying particular +extensions. + +At the time of writing (January 1999) a new RFC (currently known as +draft-ietf-drums-smtpupd-09) which updates and consolidates all the material +connected with SMTP message transmission is at a late stage of drafting, and is +expected to become an Internet Standard in due course. + +The new draft is written using the terms MUST, SHOULD, and MAY, which, when +written in capital letters, have precise meanings. To quote from the draft: + + "MUST" or "MUST NOT" identify absolute requirements for conformance to + this specification. Implementations that do not conform to them lie + outside the scope of this specification and often will not + interoperate properly with SMTP implementations that do conform. + Implementations that are fully conforming also adhere to all "SHOULD" + and "SHOULD NOT" requirements. Implementations that adhere to all + "MUST" ("MUST NOT") but not to all of these are considered to be + partially conforming. Such implementations may interoperate properly + with fully conforming ones and with each other, but this will + typically be the case only if great care is taken. Consequently, an + implementation should violate "SHOULD" ("SHOULD NOT") requirements + only under exceptional and well-understood circumstances. + +The implementation of Exim is intended to conform to the spirit of this +paragraph. The following is (I hope) a complete list of major variations +from the draft RFC. In addition to the items listed here, there are other minor +extensions such as the tolerance of white space in places where it is not +strictly permitted by the RFC. References in square brackets are to the -09 +draft sections, and brief summaries of the RFC requirement are also given in +square brackets. + + +2.1 Line termination [2.3.7, 4.1.1.4] +------------------------------------- + +[SMTP lines are terminated by CRLF.] + +Exim recognizes LF without CR as a line terminator in all forms of input. For +SMTP input, any preceding CR is discarded. An early version of Exim followed +the RFC strictly, and did not recognize LF without CR in SMTP input. However, +it seems that sites on the net send out messages with just LF terminators, +despite the warnings in the RFCs, and other MTAs handle this, so Exim was +changed. However, there is a compile time macro called STRICT_CRLF which can be +set to restore the strict behaviour, though this is undocumented. + + +2.2 Eight-bit characters [2.4.1] +-------------------------------- + +[SMTP transmits only 7-bit characters.] + +Exim is eight-bit clean, and makes no attempt to modify the data in a message +in any way. In particular, for messages containing characters with the top bit +set, it neither tries to negotiate 8-bit transmission, nor converts such +characters into an encoded form. In other words, it adopts the "just send 8" +strategy. It can be configured to send out 8BITMIME in its response to EHLO +(which it does not do by default), and it recognizes the 8BITMIME keyword on +incoming messages, but neither of these affect its handling of message data. +"Just send 8" is the strategy of a number of MTAs; it is argued that it +achieves what the user wants more often than other strategies. + + +2.3 Use of EHLO/HELO [3.2] +-------------------------- + +[Client MTAs should always start with EHLO, not HELO.] + +Exim sends EHLO only when it finds the string "ESMTP" in an SMTP greeting +message. If EHLO is refused with a 5xx return code, it then reverts to HELO as +required, but it does not contain logic for converting to HELO on other errors +such as loss of connection or timeout after EHLO. That is one reason why it +doesn't always send EHLO; there are reported to be ancient SMTP servers out +there which collapse on receiving EHLO. (There is also at least one server +whose banner reads " ignores ESMTP", but it is RFC 821 compliant in +that it responds with 5O0 to EHLO, so Exim successfully reverts to HELO.) + + +2.4 Closing the connection [4.1.1.10] +------------------------------------- + +[Client must wait for response to QUIT before closing the connection.] + +Exim closes the connection immediately after sending QUIT, without waiting for +the reply. There was a lot of discussion about this on one of the mailing +lists. The conclusion was that this behaviour is fine on Unix systems, which +have TCP/IP implementations that close down the underlying channel tidily even +when the associated process has terminated. Indeed, not waiting may be +beneficial, as it moves the TIME_WAIT state (waiting to ensure there's no more +data in transit) from the server to the client system. On some other operating +systems (I understand) it is a disaster to terminate the sending process +without waiting for the QUIT response, because all the data about the +connection lives in the client's process space, and is therefore thrown away +before the response arrives. The subsequent arrival of the response then causes +bad behaviour. + + +2.5 IPv6 address literals [4.1.2] +--------------------------------- + +[IPv6 address literals are introduced by "IPv6".] + +Exim recognizes IPv6 literals as just the colon-separated hexadecimal form of +an IPv6 address, for example 1080:0:0:0:8:800:200C:417A, without the need for a +prefix. At present, it does not even recognize the prefix. When IPv6 becomes +more widespread, Exim will follow whatever the common usage is. + + +2.6 Underscores in domain names [4.1.2] +--------------------------------------- + +[Underscores are not legal in domain names.] + +RFC 822 allows all characters except specials, space, and controls in domain +names, but the SMTP RFCs are stricter, allowing only letters, digits, and +hyphen. Exim is compliant when checking incoming addresses in SMTP commands, +but it is more relaxed by default when checking domain names that are supplied +by EHLO or HELO commands, because many client workstations get set up with +underscores in their names. There is an option that can be set to cause Exim to +refuse underscores. (There are also options to specify certain hosts from which +it will accept any old junk after EHLO or HELO. Such is the woeful state of +some SMTP clients.) + + +2.7 Removal of return-path headers [4.4] +---------------------------------------- + +[Relaying MTAs should not remove return-path.] + +Exim removes Return-Path: headers from all messages, if return_path_remove is +set (the default). It does not attempt to determine if it is being a relay or +not. Indeed, for some messages it might be both a relay and a final destination +MTA for the same message. + + +2.8 Randomizing the order of addresses of multihomed hosts [5] +-------------------------------------------------------------- + +[Multihomed host addresses should not be randomized.] + +Exim does randomize a list of several addresses for a single host, because +caching in resolvers will defeat the round-robinning that many namerservers +use. (Note: this is not the same as randomizing equal-valued MX records. That +is required by the RFC.) + + +2.9 Handling "MX points to self" [5] +------------------------------------ + +[MX points to self must be treated as an error.] + +The RFC doesn't allow for the possibility of special-purpose routing in the +case when the lowest numbered MX record points to the local host. The default +Exim configuration is compliant, but it is possible to configure Exim to behave +differently, and there are several situations where this can be useful. + + +2.10 Source routing [6.1] +------------------------- + +[Source routes should be stripped.] + +The new RFC has moved forward in deprecating source-routed email addresses. +Exim does not strip them down by default, but can be made to do so by setting +collapse_source_routes. However, even when it is not stripping them down, it +does not add host routing to reverse-paths when processing a source-routed +forward-path. + + +2.11 Loop detection [6.2] +------------------------- + +[Loop count for Received: headers should be at least 100.] + +Exim's default setting of the received_headers_max option is 30. Most messages +these days seem to accumulate less than half a dozen Received: headers, and +even a couple of forwardings don't bring this anywhere near 30. + + +2.12 Addition of missing headers [6.3] +-------------------------------------- + +[Missing headers may be added, and domains qualified, only if client is +identified.] + +Exim always adds Message-Id: and Date: headers if these are missing, whatever +the source of the message, and likewise when it expands non-fully-qualified +domains, it does so independently of the message's source. + + +2.13 Syntax of MAIL and RCPT commands [4.1.1.2, 4.1.1.3] +-------------------------------------------------------- + +Exim is more relaxed than the RFC requires: + +(1) Trailing white space is ignored. + +(2) It permits white space after the "FROM" and "TO" keywords. + +(3) It does not insist on the address being enclosed in <> characters. In fact, + it recognizes addresses in RFC 822 format here, except that domain + components are restricted to containing only letters, digits, and hyphens. + +(4) Local parts are permitted to contain null components, that is, may start or + end with an unquoted full stop (period) or contain two consecutive + unquoted full stops. + + +2.14 Non-fully-qualified domains [2.3.5] +---------------------------------------- + +[All domains must be fully qualified.] + +A domain that is not fully qualified has some of its trailing components +missing, and is normally a local alias of some sort, for example, just a +single-component host name. + +Exim can be configured to "widen" non-fully-qualified domains, either by using +the facilities of the DNS resolver, or by an explicit list of widening strings. +When this is done, it applies to addresses received by SMTP from other hosts, +as well as to locally-originated addresses. Address re-writing could also be +used for this purpose. + + +2.15 Unqualified addresses [4.1.2] +---------------------------------- + +[Addresses in SMTP commands must include domains.] + +An unqualified address consists of a local part without a domain. Do not +confuse "qualified address" and "qualified domain". A qualified address may +include a non-fully-qualified domain. + +There is one exception to the RFC rule: it is required that the unqualified +address "" always be accepted. Apart from this, Exim rejects +domainless addresses in SMTP commands by default, but it can be configured with +a list of hosts and/or networks that are permitted to send addresses without +domains in SMTP commands. Any such address that is accepted (including +) is qualified by adding the value of the qualify_domain option. + + +2.16 VRFY and EXPN [3.5.1, 3.5.2, 3.5.3, 7.3] +--------------------------------------------- + +[VRFY and EXPN should be supported.] + +Exim does not support VRFY and EXPN by default, but a list of hosts and +networks for which they are permitted can be given. + + +2.17 Checking of EHLO/HELO commands [4.1.4] +------------------------------------------- + +[Client must send EHLO. Server must not refuse message if EHLO/HELO check +fails.] + +Exim, as a client, always sends EHLO or HELO (see 2.3 above). As a server, it +does not insist on there having been a valid EHLO or HELO command before the +start of a message transaction. Any EHLO or HELO command that is received is +rejected only if it contains a syntax error. That is, it is never rejected on +the basis of any validation checking that may be performed on the data it +contains. + +However, Exim can be configured to insist that (a) there is valid EHLO/HELO +command before any message transaction and (b) the domain in that command +matches the domain obtained by looking up the IP address of the sending host. +It is possible to specify exception lists of hosts and/or networks for which +this check does not apply. + + +2.18 Format of delivery error messages [3.7] +-------------------------------------------- + +[Standard report formats should be used if possible.] + +Exim's delivery failure reports do not conform to the format described in RFC +1894. + + +## End ## diff --git a/doc/doc-misc/TexiNotes b/doc/doc-misc/TexiNotes new file mode 100644 index 000000000..9b16a7ae1 --- /dev/null +++ b/doc/doc-misc/TexiNotes @@ -0,0 +1,193 @@ +$Cambridge: exim/doc/doc-misc/TexiNotes,v 1.1 2004/10/08 10:38:47 ph10 Exp $ + +Notes for conversion of sgcal input into Texinfo input +------------------------------------------------------ + +(Dated 6 August 1996) + +The escape character is @. Only @ and curly brackets are sensitive. Get them in +by @@ @{ and @} if required. + +@: after a dot that is not a sentence end. + +@. instead of . if sentence ends with capital letter + +@copyright{} for copyright + +@minus{} is a slighly longer minus sign + +Input file ends with .texinfo usually. + +MUST start the file with + + \input texinfo + @c %**start of header + @setfilename INFO-FILE-NAME + @settitle NAME_OF_MANUAL + $c %**end of header + +Then, typically + + @ifinfo + summary and copyright + @end ifinfo + +Followed by + + @titlepage + title and copyright + @end titlepage + +Then the top node and master menu - for info file only + + @node Top, First Chapter, (dir), (dir) + @comment node-name next, previous, up + @top + + @menu + * First Chapter:: The first chapter is the + only chapter in the sample + * Concept Index:: An index + @end menu + + +Then the body + + @node First Chapter, Concept Index, Top, Top + @comment node-name next, previous, up + @chapter First Chapter + @cindex Sample index entry + + This is the contents of the first chapter + @cindex Another sample index + + +Then stuff about indexes and tables of contents + + @node Concept Index, , First Chapter, Top + @unnumbered Concept Index + + @printindex cp + + @contents + +MUST end the file with + + @bye + + +. NEWLINE AND NO-FILL MODE + + @page for new page + @* forces a line break + + +. LINE CENTERING + + @center stuff + + +. ROMAN, ITALIC, BOLD ITALIC, SMALL CAPS + + @code{...} for 'code' => `...' in info + @file{...} for file names => `...' in info + @samp{...} for sample text => `...' in info + @var{...} for variable => caps in info + @dfn{...} defining a term => double quotes in info + @emph{...} produces italic + @strong{...} produces bold + @sc{...} small caps but with letters in lower case. + @i italic ) + @b bold ) no effect on info file + @r roman ) + + +. TABBING + +. CHAPTERS & SECTIONS + + @chapter + @unnumbered <title> is an unnumbered chapter + @section + + + +. SECTION + +. FANCY VS PLAIN + + @iftex ... @end iftex for printed only; likewise @ifinfo ... @end ifinfo + + +. LEAVING BLANK SPACE + + @sp 10 + + +. EM & NEM + + no can no + +. DISPLAY ASIS + + @example ... @end example + @display ... @end display no change of font => rm + + +. COMMENTS + + @comment or @c introduces comment lines + + +. NUMBERED LISTS + + @enumerate + @item + first item + + @item + second + @end enumerate + + + +. BULLETED LISTS + + @itemize @bullet + ... + + + +. CROSS REFERENCES + + @xref start sentence + @ref{name} + @pxref (parenthesized) + + 5 args: node name (required), cross-ref name, topic description, name of + info file, name of printed manual. + + + +. TABLES + + @table for two-column tables + @table @asis + + @item first column + second column + + @item ... + + + +. INDEX + + @cindex concept index + @findex function index + @vindex variable index + @kindex key index + @pindex program index + @tindex data type index + +*** diff --git a/doc/doc-misc/WishList b/doc/doc-misc/WishList new file mode 100644 index 000000000..bee4f328a --- /dev/null +++ b/doc/doc-misc/WishList @@ -0,0 +1,1727 @@ +$Cambridge: exim/doc/doc-misc/WishList,v 1.1 2004/10/08 10:38:47 ph10 Exp $ + +EXIM 4 WISH LIST +---------------- + +Even when it was first released, Exim 4 had a Wish List because not all the +things suggested for it were implemented. The list has not stopped growing... + +Another reason it is so long is that I have retained some items from the Exim 3 +Wish List that never got implemented, but which seem reasonable possibilities +for later addition to Exim 4. + +I have guessed at the amount of work involved, and categorized the items as +Tiny, Small, Medium, Large, or Unknown. The guesses are not based on any +detailed investigation, so must be taken as very rough. + + +------------------------------------------------------------------------------ +------------------------------------------------------------------------------ +----- Retained from the Exim 3 Wish List ------ +------------------------------------------------------------------------------ +------------------------------------------------------------------------------ + +(10) 13-Jul-98 M more flexibility for pipe returns +Ben Smithurst + +The ability to specify more precisely what happens concerning the return code +from the pipe and the presence/absence of STDOUT/STDERR is requested. The +particular configuration that was requested was: + +> if the command exited EX_OK, *and* produced nothing on STDOUT or +> STDERR, it succeeded... +> if the command exited EX_TEMPFAIL, defer, regardless of +> STDOUT/STDERR... +> otherwise freeze the message (this will get my attention by way of +> freeze_tell_mailmaster)... +------------------------------------------------------------------------------ + +(11) 17-Jul-98 G support for DSN +Andy Mell + +It is unclear to me how this should work in the presence of aliases and +forwarding. Local deliveries would have to explicitly configured as deliveries +or relaying or whatever. A substantial amount of code is probably needed. + +Jeffrey Goldberg +I have nothing to add except to say that for many of the reasons you've +stated, I don't think that DSN is coherent enough to be worth the effort +to implement. + +Another comment: + + I thought the RFC was pretty clear on this. In a nutshell, if the + delivery rewrites the envelope from address, it's considered a + terminal delivery (i.e. delivery to a mailing list exploder), otherwise + treat it as a forwarding operation (the /etc/aliases case). I would + treat a .forward expansion as a final delivery event (it got to the + user as far as the MTA is concerned). + + Yes, we need the DSN syntax. We also require the complete semantics of + NOTIFY=SUCCESS,FAILURE for our application to work. + + Electronic Bill Presentment is really going to push the need for + DSN support in MTAs. We just don't want to get stuck in a situation + where we're faced with a non-DSN-aware MTA when we go to install + our bill/statement engine, thus our interest in what the MTA vendors + are planning to do about DSN. +------------------------------------------------------------------------------ + +(41) 14-Oct-98 M Find a way of modifying header lines +Oliver Smith + +The problem with header_remove followed by header_add is that you can't refer +to the previous value of the header when adding a replacement. This could be +solved with a replace_header option. +------------------------------------------------------------------------------ + +(43) 15-Oct-98 M Sender rewrite *after* SMTP incoming checks +Andreas Edler + +The anti-relaying check happens after the sender has been rewritten; there are +times when it would be helpful to do the check on the original sender, not on +the rewritten one. Quite how to configure this I'm not sure. + +A related suggestion (from Steve Sargent) is to retain the original sender +address and make it accessible somehow. +------------------------------------------------------------------------------ + +(46) 20-Oct-98 L SMTP protocol hooks +Malcolm Ray + +"But there are enough broken SMTP implementations to make me wonder whether +there isn't a case for providing hooks for tweaking the SMTP transport's +protocol exchange. Something which would allow me to say things like 'if, when +talking to lame.example.com, you get a 251 response to a MAIL command, rewrite +the response to 501 before continuing'." +------------------------------------------------------------------------------ + +(50) 13-Nov-98 M A "Focus" option for eximon +Frank Elsner + +This is the opposite of "Hide"; it just displays a certain subset. Hmm. Could +something clever be done with regular expressions? +------------------------------------------------------------------------------ + +(61) 22-Dec-98 M Send failed error messages to somebody +Harald Meland + +With sendmail, the failed error message is made into a error message, +with both envelope sender and recipient set to MAILER-DAEMON. The +original, bogus-envelope-sender message is then available to whoever +receives MAILER-DAEMON's mail. A more flexible approach would be to +specify a specific recipient. +------------------------------------------------------------------------------ + +(81) 01-Mar-99 M Addition of Content-MD5 support +Martin Hamilton + +Martin supplied a suggested patch at +http://www.net.lut.ac.uk/~martin/antispam/exim-hacks/ +------------------------------------------------------------------------------ + +(85) 15-Mar-99 M ability to rewrite addresses in non-standard headers +Dave Lewney +John Holman + +Such as "return-receipt-to". See also 41. +------------------------------------------------------------------------------ + +(90) 21-Apr-99 M change wild prefix/suffix greediness +Ben Smithurst + +Currently, when prefix or suffix containing * is set on a director, and the +fixed part occurs more than once in a local part, the length of the prefix or +suffix is maximized. For example, with suffix = -* and a local part of +foo-bar-baz the suffix is taken as bar-baz, leaving the local part as foo. +An option is proposed to invert this rule. +------------------------------------------------------------------------------ + +(91) 26-Apr-99 S make queue_run_in_order to newest first +"Andreas M. Kirchwitz" + +The tidiest thing would be to have queue_run_order={random,oldest,newest}, +and make queue_run_in_order obsolete. +------------------------------------------------------------------------------ + +(93) 04-May-1999 L fallback_transport + +This would be a generic transport option, specifying a different transport to +be used if the first one failed. Failed hard, or failed soft? Or an option? +And if failed hard, is a bounce message sent as well, or not? There are uid +issues. Remote delivery would have to be done always in a subprocess so that +the main process could retain privilege in case the fallback transport was +local. That could be conditional. That's why this is labelled "Large". Some of +the things people want to do with this can be done by variations in the +routers, e.g. use $message_age to switch routers. +------------------------------------------------------------------------------ + +(94) 13-May-1999 M message to go with -Mg +Dave Holland +Alan Thew + +So the admin can pass back a reason. +------------------------------------------------------------------------------ + +(99) 28-May-1999 M header to list failures for syntax_errors_to +mark david mcCreary + +"I use the syntax_errors_to feature to email a copy of the error message. +It would be helpful to have the X-Failed-Receipients header in there, +identifying which addreses(s) are the problem, so that I don't have to +parse the body of the email message to figure out which addresses." +------------------------------------------------------------------------------ + +(100) 04-Jun-1999 S admin_users option, like trusted_users +Paul Mansfield +------------------------------------------------------------------------------ + +(102) 21-Jun-1999 M expanded basic variables +Julian King + +Oh, and a wishlist entry, qualify_domain, and preferably other variables +can be set with a $lookup in the first part of the exim configuration +file, perhaps by an equivalent to backticks in shell script ("`command`")? +------------------------------------------------------------------------------ + +(105) 28-Jun-1999 M MIME-format bounce messages +Paul Makepeace + +"Is there any work going/gone on/planned to enable exim to report delivery +status notifications using RFC1892 multipart/report MIME messages? It would be +great to have errors reported in a message/rfc822 attachment." + +Jeffrey Goldberg +"I like plain bounces, so would hope that if you do this, that it be +configurable. I think that even for those who want it, it shouldn't be very +high on the wish list priority." + +Other suggestions: toggle for bounces/warnings; override max_return for +certain addresses; use plain text if original not MIME. See Paul's hack +for background of what to do. + +Nigel suggests using a specially named autoreply transport to generate bounces; +people could then replace this with another transport (e.g. pipe) if they want +to customize it themselves. + +Eli Chen posted an unconditional patch for 3.32 that does some of this work. +That could form a basis. +------------------------------------------------------------------------------ + +(107) 12-Jul-1999 S defer transport at given load level +Marc Haber + +------------------------------------------------------------------------------ + +(108) 16-Jul-1999 S remote sort by numbers of recipients +mark david mcCreary + +In the absence of remote_sort, sort remote domains by the number of recipients +in each. +------------------------------------------------------------------------------ + +(114) 11-Nov-1999 S List of possible outgoing interfaces + +Allow the smtp "interface" option to be a list: try them in turn until one +is found to work. Also allow masks to specify a range of addresses. +------------------------------------------------------------------------------ + +(123) 23-Dec-1999 L Use AUTH + TURN for dial-in hosts +Andrew Tverdokhleb + +The way to do this would be to have Exim deliver messages into per-host +directories in, say, BSMTP format. Accept TURN if authenticated, and cause it +to run a helper program that is passed the socket in order to deliver the mail. +Provide a helper program! +------------------------------------------------------------------------------ + +(125) 04-Jan-2000 L Use shared memory segment for queue list +Theo Schlossnagle + +The idea is that a queue-runner that finds no existing shared segment should +create one (if configured - possibly some fixed size) and all Exim processes +should maintain a list of messages in it, thereby saving on directory scans +when there are lots of messages. This needs a lot of careful thought to try to +eliminate any possibility of data loss. The interlocking could be quite tricky. +Further posters suggested using a db file to hold the list. See also 127. +------------------------------------------------------------------------------ + +(129) 14-Jan-2000 L Dynamically loadable lookup modules +Steve Haslam + +Suggested patch provided. +------------------------------------------------------------------------------ + +(131) 17-Jan-2000 T Facility for assuming existence for EACCES +Peter Radcliffe + +The opposite option for "+" in require_files: assume existence if cannot +peer into the directory (+ assumes non-existence). +------------------------------------------------------------------------------ + +(131) 29-Feb-2000 M? Control total number of outgoing SMTP calls +Brian White + +This is for hosts with slow connections. Could some modification of +serialize_hosts be used for this? Or maybe use a semaphore? They seem to +be quite widely available. +------------------------------------------------------------------------------ + +(132) 01-Mar-2000 S Lookup host name from outgoing interface +Vadim Vygonets + +Instead of primary_hostname, look up the name for the interface that is being +used for sending. Suggested patch supplied, but this should be an option of the +smtp transport. +------------------------------------------------------------------------------ + +(133) 06-Mar-2000 S Filter option not to log "previously sent" +Bruce Bowler + +This is when using the "log" option of the autoreply driver. +------------------------------------------------------------------------------ + +(134) 09-Mar-2000 S Option to remove attachments when bouncing +------------------------------------------------------------------------------ + +(136) 13-Mar-2000 S/M Option for aliasfile to suppress "me too" + +Could be tricky determining who "me" is. +------------------------------------------------------------------------------ + +(143) 08-May-2000 S Make quota_warn_threshold into a list +David Carter + +So several warnings could be generated as the mailbox got bigger and bigger. +------------------------------------------------------------------------------ + +(146) 15-May-2000 M Allow SMTP error codes in retry rules + +This would allow special handling of certain errors from certain hosts. In +particular, it would allow failing of certain 4xx codes. + +This is now available for 4xx responses to RCPT commands. Is anything more +needed? +------------------------------------------------------------------------------ + +(148) 15-May-2000 S Warn recipient if message rejected for quota excession. +Heinz Ekker + +Maybe not all that small, because the possibility of retrying must be taken +into account. +------------------------------------------------------------------------------ + +(149) 19-May-2000 L Make added headers visible in filters and other places +Hans Morten Kind + +Headers added by directors/routers are not visible in subsequent processing. +This is a request to make them visible. What about removed headers? This could +be tricky to specify, hence the L. + +A separate but related issue is the effect of headers added by "unseen" +directors. These are documented in chapter 19 as not being accumulated. Should +any change be made? +------------------------------------------------------------------------------ + +(155) 16-Jun-2000 M Special handling for certain hosts +mark david mcCreary + +A means of changing the transport depending on the host name/IP of the most +preferred MX record so that all domains that route to certain hosts can be +handled specially. Maybe this could be a variable that is available in the +expansion of the "transport" option. +------------------------------------------------------------------------------ + +(158) 29-Jun-2000 S Configure "From" in bounces +Ben Parker + +Cf Reply-To. +------------------------------------------------------------------------------ + +(159) 07-Jul-2000 M Keep messages for fixed time +Gary Palmer + +An option to keep messages on the queue for a specified time, even if all their +destination hosts have timed out. +------------------------------------------------------------------------------ + +(164) 17-Aug-2000 S sender_unqualified_auth_hosts + +To allow authenticated hosts to send unqualified addresses. Presumably it +needs received_... as well. +------------------------------------------------------------------------------ + +(167) 05-Sep-2000 L Support for ODBC + +This would allow access to databases that don't have native support built into +Exim. See http://www.openlinksw.com/info/docs/rel3doc/unix/odbcsdk.htm +------------------------------------------------------------------------------ + +(168) 06-Sep-2000 M Deliver messages that alias to nothing to a given address +Dr ZP Han + +If other people are managing alias lists, and one is empty, bounce that +delivery to a given address rather than freezing the message. Use the errors_to +address? +------------------------------------------------------------------------------ + +(172) 11-Sep-2000 S Allow file/directory in appendfile to override +"Michael J. Tubby" + +When appendfile is called from forward or filter files, it ignores file or +directory settings. Maybe they should override. The path set by the forward or +filter is available in $address_file these days, so it could be used to create +a longer path. +------------------------------------------------------------------------------ + +(173) 18-Sep-2000 S A way of doing lsearches with EOL terminated keys +Jason Robertson + +This is for looking up things like subject contents. Probably need an option to +exim_dbmbuild to make them into DBM files. +------------------------------------------------------------------------------ + +(174) 19-Sep-2000 S A way of using a different port for fallback hosts. +Dean Brooks +------------------------------------------------------------------------------ + +(181) 10-Nov-2000 S Compile-time options for ignoring Sendmail options + +So that new ones could be accommodated easily. +------------------------------------------------------------------------------ + +(183) 04-Dec-2000 L dns_means_nonexist_after +Dave C. + +In other words, wait a bit before giving up. This needs a mechanism for +remembering, which is not currently available. To be borne in mind for the +future. +------------------------------------------------------------------------------ + +(184) 04-Dec-2000 M Log more details of local caller +J. Nick Koston + +"I was wondering if it was possible for exim to log the parent pid's cwd and +exe when it is called from a script/invoked by actually running /usr/sbin/exim +or /usr/sbin/sendmail." Question: is this information actually/easily +available to Exim? Needs investigation. +------------------------------------------------------------------------------ + +(186) 19-Dec-2000 S A simple utility to reset a retry time +Marc Haber + +Basically, to do what exim_fixdb "delete" can do, but straightforwardly. There +could be an interface from eximon. +------------------------------------------------------------------------------ + +(187) 02-Jan-2001 M Wildcarding in headers_remove +Tamas TEVESZ + +What I'd like to see is it to handle globs (or regexps, but i'm not sure this +latter would worth the hassle), in a way like: + + headers_remove = "X-*:Additional-header" +------------------------------------------------------------------------------ + +(188) 02-Jan-2001 S Make pipe timeout a temporary error +Georg v.Zezschwitz + +A way to make a timeout into a temporary error. +------------------------------------------------------------------------------ + +(190) 03-Jan-2001 M Multiple message operations in eximon +------------------------------------------------------------------------------ + +(195) 19-Mar-2001 T TCP window size + +TCP window size for receiving/sending, SMTP client/server. +------------------------------------------------------------------------------ +------------------------------------------------------------------------------ + + + +------------------------------------------------------------------------------ +------------------------------------------------------------------------------ +----- Things that didn't make it into Exim 4 ------ +------------------------------------------------------------------------------ +------------------------------------------------------------------------------ + +. An option to send messages to postmaster when ignore_errmsg_errors_after +times out. + +. When an address is being routed, its constituents are in $local_part and +$domain, but there is currently no variable that contains the whole thing. It +could be put into $recipient, but that risks confusion with $recipients +(which is available in system filters). Maybe $address could be used? + +. The ability to relay to host X without knowing all the domains that host X +might have. At ACL time, one would need to verify the recipient, and determine +that it routed to host X. + +. A new lookup library that operates on a specially prepared file of IP +addresses and masks so that a single "lookup" yields a yes/no answer. This +should be a freestanding thing - needs a utility to build the file from a list. + +. People want to change the wording of messages; can we find an efficient way +of allowing this? (Maybe put all messages into a separate module?) The problem +is not in the messages themselves, but in the values that get inserted into +messages. Would have to invent a new kind of function that used identified +values rather than positional ones. Use GNU gettext? + +. Invent lf_hosts for those that may use LF without CR. Any other RFC +things we need to worry about? + +. A user would really like to see something similar, perhaps with +"ID=$authenticated_id", similar to "helo=" and "ident=" in the default received +header. BUT there are security issues. Maybe give it as a commented out option +in the default configuration? + +. Consider expanding further options that take integer values. What about +smtp_xxx options for different limits at different times of day (for example)? +What about tls_advertise_hosts (so can look at incoming IP/port)? + +. How about a "hold hosts" option (cf hold_domains) to hold delivery to certain +hosts? + +. Allow user filters to use "headers add", but probably not remove. Or maybe +just implement "allow" options for both of these features. + +. Have the return from pipe in a variable, so that (e.g. error_message_file) +can make use of it. + +. Implement randomize for ldap/sql servers. + +. Add an option for ETRN that says "wait for the command to finish, and use its +stdout as the SMTP response." + +. -odsomething for "ignore retry when doing immediate delivery". + +. Add an option to the smtp transport to make it treat 5xx on connection as if +it were 4xx. Or possible add a sophisticated "after command X, treat xxx as +yyy". + +. A way of rewriting addresses in non-standard header lines such as +Mail-Followup-To. + +. Global option to enable initgroups() for exim uid. Default off. + +. When verifying a sender, should it be rewritten with any T rewrites, because +it would be so rewritten if it actually was a recipient in a message? + +. Sean Witham wants a way of defining macros that are not privileged, and a +sort of #ifdef structure that allows for different configurations in the same +file. + +. Allow :fail: to specify that 551 be used instead of 550. Maybe allow a code +at the start, optionally? What about :defer:? + +. SMTP timeout in middle of receiving message: log sender address if known, and +possibly message_id if known. + +. Make -brw show rewrites for transports too. + +. Have the MTA log destinations that have timed-out on a ident request and +no longer send rfc1413_queries to them. Add an option for how not to cache +these entries. + +. Options and/or a utility to enable non-privileged users to view the queue +(e.g. -bpp), manipulate their own messages, etc. + +. Specify a port along with a host in a route_list. + +. A generalized "From" escaping scheme that also escapes >From so that the +whole thing can be reversed. + +. There was a request for the \dns_again_means_nonexist\ option not to be +instantaneous, but to operate only after the DNS has been giving "try again" +for some time. Use the misc hints database. + + + +------------------------------------------------------------------------------ +------------------------------------------------------------------------------ +----- The Exim 4 Wish List ------ +------------------------------------------------------------------------------ +------------------------------------------------------------------------------ + +(1) 01-Jan-02 U Use of dynamically loaded libraries. + +People want Exim to use dynamically loaded modules for a variety of reasons. +When I started to create Exim, I never expected anything other than source +distribution; the RPMs and inclusions in OS distributions caught me by +surprise. I know very little about the mechanics of dynamic loading, but I'm +aware that not all operating systems support it. I'm also aware that not all +people support it! + +Furthermore, a way round this might be to supply more hooks along the lines of +local_scan(). Then people can write their own dynamic loaders if they want. +------------------------------------------------------------------------------ + +(3) 01-Jan-02 U Test for over-quota at SMTP time + +This is a hard one, because the only way to test for over quota is to try to +deliver a message, certainly if system quotas are being used. And also, the +only available size at RCPT time is the SIZE option, though of course the test +could be run at DATA time. I think maybe we leave this one to an external +program, and require people to use ${run} to access the data. Let someone else +figure out how to extract the current mailbox size! + +One suggestion is to implement + + ${file_size:/path/to/file} + ${directory_size:/path/to/directory} + +so that explicit checks can be done. It may be necessary to have four +operators, two being based on the block count, and two showing the "visible" +size. Directory scanning is expensive; is there any scope for caching? It would +seem not (you don't often get two addresses to the same user). +------------------------------------------------------------------------------ + +(4) 01-Jan-02 S Option to reject if no From: or Date: header line + +Exim, in common with many other MTAs, inserts a From: or Date: header line if +one is missing. (It also inserts a blank Bcc:, but that is no longer needed by +RFC 2822 - it was by 822.) The suggestion is an option to give an error +instead. This could be done by making it possible to detect these insertions in +the acl_smtp_data ACL. +------------------------------------------------------------------------------ + +(6) 01-Jan-02 S Option to disable the use of -t +Dave C. + +Would require work so that Exim itself doesn't use -t. +------------------------------------------------------------------------------ + +(7) 01-Jan-02 M Avoid showing LDAP passwords in log lines for LDAP errors +John W Baxter + +May be tricky, because at the higher levels, the format of the query is not +understood. +------------------------------------------------------------------------------ + +(8) 01-Jan-02 S Expand once_repeat in autoreply +John Jetmore +------------------------------------------------------------------------------ + +(9) 01-Jan-02 S Headers as well as body in file for autoreply +Florian Laws +------------------------------------------------------------------------------ + +(10) 01-Jan-02 T Make "true" and "false" valid expansion conditions + +This might help with "and" and "or" when one of the sub-conditions is, for +example, a lookup. +------------------------------------------------------------------------------ + +(11) 01-Jan-02 S Allow a filter to include another file. +------------------------------------------------------------------------------ + +(12) 01-Jan-02 M Support for different SQL servers per query + +In other words, the global mysql_servers etc. is too restrictive. +------------------------------------------------------------------------------ + +(14) 01-Jan-02 M? Support for Sendmail milters + +This could perhaps be done by extending the local_scan() idea and providing a +"standard" module which interfaced to milter. +------------------------------------------------------------------------------ + +(15) 01-Jan-02 M More hooks like local_scan() + +One request has been for a similar hook at logging time. For other SMTP +interactions, maybe a hook into the ACL? See also 79 and 218. +------------------------------------------------------------------------------ + +(17) 11-Jan-02 M The construction of config.h needs refactoring + +This has been hacked about substantially since the original implementation. +Given that there is a program (buildconfig), the messing around with the +environment could be abolished. Also, the distinction between "yes" and "no" +isn't always properly made (tests for #ifdef don't care about the value). +------------------------------------------------------------------------------ + +(18) 24-Jan-02 S Make $value retain its value after a top-level expansion + +This was specifically for use in filter files. Currently it reverts to empty +as a consequence of save/restore for every lookup. It might be confusing to +do otherwise, however. +------------------------------------------------------------------------------ + +(19) 29-Jan-02 L Use of multiple DBM libraries + +The problem is how to handle conflicting function names. Much research is +needed. +------------------------------------------------------------------------------ + +(20) 29-Jan-02 S Make system filter refreeze after manual thaw + +Currently, a "freeze" in a system filter doesn't freeze after a manual thaw. +------------------------------------------------------------------------------ + +(21) 12-Feb-02 S Expand return_size_limit +Joachim Wieland + +Is this really worth it? A per-transport value is also suggested - that would +mean remembering the value with each failed address and taking a minimum or +a maximimum (which?). +------------------------------------------------------------------------------ + +(24) 21-Feb-02 ? A way of testing TLS using -bh +------------------------------------------------------------------------------ + +(27) 06-Mar-02 M Distinguishing between different temporary callout errors + +The request was to distinguish between a 4xx error and a failure to connect. +Problem is: how to cope when there is more than one host? Maybe only if ALL +fail to connect. An option like /callout_no_connect_ok. +------------------------------------------------------------------------------ + +(30) 12-Mar-02 S Add "recipients" precondition to routers. + +This would avoid having to use "condition". (See also requirement for $address +mentioned above.) However, it would also require adding a caching feature, and +probably $recipient_data (cf $domain_data). +------------------------------------------------------------------------------ + +(31) 21-Mar-02 S Variables that indicate 8-bit message and 8-bit host, and + a way of using them to suppress a transport filter + +A variable that is set if the message contains 8-bit characters, and another +that is set during the smtp transport if the host supports 8-bit. Then we also +need a condition that's expanded in the transport to control whether the filter +is run or not (e.g. transport_filter_condition). +------------------------------------------------------------------------------ + +(32) 22-Mar-02 M More info about callout fails for header sender verify + +When there's a callout failure for an envelope address, the error message +contains details (by default) of the callout commands. This doesn't happen +for addresses in the header because there may be more than one of them, and +deciding how to give that information is tricky. Can we do better? +------------------------------------------------------------------------------ + +(33) 25-Mar-02 S Option to assume nomatch in dnslist lookups that time out + +Currently this causes a DEFER. +------------------------------------------------------------------------------ + +(34) 26-Mar-02 S Access to DNS lookup functions via local_scan() API + +This would make local_scan() writers lives easier for DNS usage. +------------------------------------------------------------------------------ + +(36) 02-Apr-02 ? A way of throttling, but allowing, relaying that would + otherwise be denied + +This was suggested in connection with anonymizing messages. The "wait" command +in ACLs goes some way towards this. Is it enough? +------------------------------------------------------------------------------ + +(41) 17-Apr-02 T Make config.samples available as a directory for ftp + +This is so that people can browse individual samples directly. +------------------------------------------------------------------------------ + +(42) 23-Apr-02 T An option not to flatten newlines in $message_body. + +Or maybe better to provide $message_body_nl so as to have both. +------------------------------------------------------------------------------ + +(43) 23-Apr-02 T An option to treat 4xx as 5xx from STARTTLS + +This would make Exim retry in clear unless the host is in hosts_require_tls. +------------------------------------------------------------------------------ + +(44) 24-Apr-02 ? Use errors_to for timeouts after redirect syntax errors + +A syntax error in redirection data (with skip_syntax_errors false) causes a +defer. Eventually, the address may time out. This suggestion is that, when it +does, the bounce is sent to errors_to rather than to the sender. +------------------------------------------------------------------------------ + +(45) 13-May-02 T smtp_etrn_serialize_id = .... + +The default behaviour would be equivalent to + + smtp_etrn_serialize_id = $smtp_command_argument +------------------------------------------------------------------------------ + +(47) 16-May-02 S Access to all addresses in batched local delivery +Miquel van Smoorenburg + +In a batched local delivery with more than one recipient, there's no way to +access the list of recipients for doing custom things, such as stuffing them +all into a header. (BSMTP is the only approach; not everybody can use it.) +Suggested patch supplied. +------------------------------------------------------------------------------ + +(48) 21-May-02 M Support for ATRN (server and client) +Brian Candler + +Server: If Exim had the ability to accept an ATRN command and then simply +invoke an external program, passing the SMTP stream on stdin and stdout and +the authenticated id as a parameter, that would do the job nicely. + +Client: We need a variant of 'exim -bs' which would connect to a specified +host, send AUTH/ATRN, and then accept incoming messages as usual. +------------------------------------------------------------------------------ + +(50) 22-May-02 T Add comment (duplicate address) to Envelope-To: + +This is just to minimize the confusion some people have. +------------------------------------------------------------------------------ + +(51) 07-Jun-02 S Option to use another address in callout MAIL FROM + +This would be an address to try if MAIL FROM:<> failed. Is this actually going +to be helpful? See also 101. +------------------------------------------------------------------------------ + +(53) 11-Jun-02 S Make local_scan() dynamically loadable + +David Woodhouse sent a patch. There's a more sophisticated one from Marc +Merlin. (See also Peter Benie's comments.) But should the base Exim have all +this in it? +------------------------------------------------------------------------------ + +(54) 11-Jun-02 S Ignore -Ac if called as mailq + +I am not sure if this makes sense. This flag requests a listing of a different +mail queue, but Exim doesn't work like that. Is is not better for people to be +aware of this? +------------------------------------------------------------------------------ + +(55) 13-Jun-02 M Rewriting whole header lines +Dave C. + +Current rewriting rules apply to individual addresses in header lines. This +feature would use a regex to match whole lines and replace them. It could be +useful for patching up syntactically invalid lines from crappy clients, before +the syntax check kicks in. (It might also be useful for hiding local host names +in Received: headers.) +------------------------------------------------------------------------------ + +(58) 26-Jun-02 ? Extend PAM support + +Apparently PAM can do challenge-response authentication. The Exim interface +can't handle this. Investigate and think about how to do this. +------------------------------------------------------------------------------ + +(59) 26-Jun-02 M A "custom" authenticator + +... that is simply a front end to external code. For example, there may be +an external API that hides the user password and does CRAM-MD5 when passed the +details of the challenge and response. +------------------------------------------------------------------------------ + +(60) 27-Jun-02 S Make trusted_users a local part list + +So that it can use lsearch etc. +------------------------------------------------------------------------------ + +(62) 28-Jun-02 S Remove headers before DATA ACL +Patrice Fournier + +"I'd like to be able to give Exim a list of headers that must be removed +from the message at arrival, before data_acl processing (and before the +rcpt_acl warn headers are added to the message)." +------------------------------------------------------------------------------ + +(63) 28-Jun-02 S Access to ACL-added headers in ACLs +Patrice Fournier + +"I'd like also to be able to look at the already added headers by a +rcpt_acl when still checking rcpt_acl (either later in the acl for the same +RCPT TO or for another RCPT TO)." +------------------------------------------------------------------------------ + +(65) 28-Jun-02 M Expand fallback hosts + +See also 174 of the Exim 3 list. +------------------------------------------------------------------------------ + +(66) 01-Jul-02 M Use Berkeley DB 4 concurrent access features + +This might give better performance on very busy sites by reducing the +contention for access to hints databases. Rob Butler points out that this could +also be useful to allow updates of other DB files used by Exim to happen +concurrently. Another thing to think about with BDB is the possible use of +B-trees. +------------------------------------------------------------------------------ + +(68) 01-Jul-02 S Add sender host to delivery line + +"Would it be possible to have a "sending_host_on_delivery" option that +logs the IP of the sending host in the => line?" Also requested was amount of +data transmitted for a non-delivery attempt. +------------------------------------------------------------------------------ + +(69) 03-Jul-02 T Log selector to log whoson checs +Matt Bernstein + +"I'd quite like a log_selector option which could spot you'd done a whoson +lookup in your DATA ACL and maybe log it as W=user." +------------------------------------------------------------------------------ + +(70) 09-Jul-02 S A way of changing the RCPT address in an accept router + +So as to avoid duplication problems when sending multiple addresses in multiple +copies to the same address. +------------------------------------------------------------------------------ + +(73) 17-Jul-02 M Match a list from within a condition + +e.g. ${if matchdomain {$domain}{+domainlist} ... + ${if matchhost {$sender_host_address}{1.2.3.4/10:2.3.4.5/16}... + +Thought needed about how to handle host names. This may be too messy to specify +cleanly. + +22-Apr-04: Implemented for domains, addresses, and local parts. Hosts are +too messy! +------------------------------------------------------------------------------ + +(74) 22-Jul-02 M Extend -bV to do more semantic checking + +For example, diagnose "local_hosts" that should probably be "+local_hosts". +------------------------------------------------------------------------------ + +(75) 23-Jul-02 S Reference option on command line + +The idea here is that a spam scanner that re-injects a message can supply a +reference on the command line that gets logged with R=. +------------------------------------------------------------------------------ + +(78) 30-Jul-02 S Expand queue_only (and/or queue_only_file) + +The requirement is to make it possible to queue messages if certain conditions +are met (e.g. messages from certain local users). See also 93. + +This control can now be achieved in the ACL - is this still needed? +------------------------------------------------------------------------------ + +(79) 31-Jul-02 S Additional info for log lines + +An option to set an expanded string to be added to <= lines. And also for the +other delivery lines? See also 15. +------------------------------------------------------------------------------ + +(84) 09-Aug-02 S Make interfaces available in a variable + +Something like $local_interfaces. Maybe limit the max length. +------------------------------------------------------------------------------ + +(85) 12-Aug-02 S/M Notice database connection failures + +The small version of this just removes a server from the list within a single +Exim process when a connection to it fails. The bigger project would use the +retry database - but that has implications for bottlenecking and may not be +helpful. See also item 109. Another suggestion is to randomize the order in +which database servers are tried (randomize_database_servers). And another is +to measure response times and remember which server is fastest. +------------------------------------------------------------------------------ + +(87) 12-Aug-02 M Partial lookups for query-style lookups + +The suggestion is to allow the lookup to contain a keystring (same syntax as +single-key lookups) which is then permuted and place in a suitable variable +each time - $permuted_key or something. +------------------------------------------------------------------------------ + +(88) 20-Aug-02 S Allow special retrying for forced defer + +See also 146 in Exim 3 wish list above. +------------------------------------------------------------------------------ + +(89) 20-Aug-02 S Also allow retry rules on routers and transports +------------------------------------------------------------------------------ + +(90) 23-Aug-02 M Macros with arguments, a la C + +I don't like this, because of the cost of frequent interpretation. +------------------------------------------------------------------------------ + +(93) 27-Aug-02 S queue_only_condition +Peter A. Savitch + +queue_only_condition global option, expanded string. This contain +condition, which if evaluated to `no' or `false' or `0', behaves like +queue_only (queue_only_load ?). Don't know what to do is the string +expansion fails with DEFER (either force queueing or continue with +immediate delivery). Another option can control Exim behaviour if the +expansion fails. Don't know how the name for it ;-) See also 78. + +This control can now be achieved in the ACL - is the new feature now needed? +------------------------------------------------------------------------------ + +(95) 27-Aug-02 S Log all parents as a router option + +So that specific addresses can be logged like this. Should there be more log +selector options per router? Per transport? +------------------------------------------------------------------------------ + +(99) 28-Aug-02 L Test pre-conditions in order given + +This would get round certain problems with require_files. However, it is +totally incompatible, and therefore an "Exim 5" wish. +------------------------------------------------------------------------------ + +(101) 02-Oct-02 M Callout and <> rejections + +Some people don't want to fail the callout if the MAIL FROM:<> command is +rejected. Think of a way of handling this tidily. See also 51. +------------------------------------------------------------------------------ + +(102) 03-Oct-02 M Log option to suppress message-id logging + +M because it would involve a change to eximstats. +------------------------------------------------------------------------------ + +(106) 09-Oct-02 S Appendfile to create directory not as user + +Arrange for the setup entry to appendfile to create the directory under some +other uid (and with given owners/permissions?) +------------------------------------------------------------------------------ + +(109) 15-Oct-02 M Remember when LDAP (etc) servers are down + +The idea would be to use some kind of retry rule, just like for hosts. +See also 85. +------------------------------------------------------------------------------ + +(110) 18-Oct-02 M errors_to for pipe command in filter + +To work in the same was as errors_to for deliver commands. +------------------------------------------------------------------------------ + +(113) 15-Nov-02 M support for XMLRPC + +Patch supplied for 4.10 by Joel Vandal. +------------------------------------------------------------------------------ + +(114) 04-Dec-02 M local_scan: return message on accept + +(This actually dates from earlier.) The problem with this is that the string +currently passes into $local_scan_data. Thus, an incompatible change of some +sort would be required. Possibly a global that local_scan can set? +------------------------------------------------------------------------------ + +(118) 10-Dec-02 S access to Perl from local_scan +------------------------------------------------------------------------------ + +(119) 12-Dec-02 M ability to specify additional headers in an autoreply + +This is so that vacation messages etc can have MIME headers that specify, for +example, the character set. +------------------------------------------------------------------------------ + +(125) 02-Jan-03 M Per-host daemon logging + +"So what I would like is an option like debug_hosts, that allows to specify +an hostlist, and if the current incoming/outgoing hosts matches, creates a +logfile like $hostname_(in|out).log in my logdirectory." +------------------------------------------------------------------------------ + +(127) 06-Jan-03 M Different messages for different callout failures + +The real requirement here is to detect when a callout "MAIL FROM:<>" failed, so +that a specific warning about that can be sent, different to the message when a +callout "RCPT TO:" fails. I think this is in fact now mostly done. +------------------------------------------------------------------------------ + +(129) 09-Jan-03 M Keep track of DNSBL timeouts, and refrain from calling + +If so configured, keep track of DNSBL timeouts in a hints record, and don't +retry that DNSBL for a while after (a sufficient number of) timeouts. It is +effectively disabled for a while. Log enable/disable, of course. Another +thought is an option not to apply +defer_unknown unless *all* DNSBL lookups in +a list defer. +------------------------------------------------------------------------------ + +(130) 09-Jan-03 M A number of LDAP-related things +Peter A. Savitch + +OpenLDAP 2.1 is going to be more popular (2.1.9 is available with many +bug fixes). TLS-enabled LDAP is an interesting and usefull thing. +I can try to implement some things and send the patches, like with +ldapi. + +How do You see: + +1) The propagation of TLS options (key, certificate, CA certificate) + to the OpenLDAP library. + +2) (was dereferencing; done in 4.23). +------------------------------------------------------------------------------ + +(131) 09-Jan-03 S Additional variables +Peter A. Savitch + +$root_uid Why? + +(Some that were previously here are done) + +$smtp_accept_count -- used for acl_smtp_connect + +$queue_runners -- children of the listening daemon could use this + value for controlling the number of queue runners + +I don't like either of these because they cannot be real-time values. They +would be snapshots of the values at the time the process was forked from the +daemon, and I fear they would just be confusing. For processes that were not +forked from the daemon they couldn't be set at all. +------------------------------------------------------------------------------ + +(131) 09-Jan-03 S Additional options +Peter A. Savitch + +exim_processes_max +exim_file_descriptors_max +queue_run_condition -- to deprecate queue_run_max, better system + load control + +Given Exim's distributed nature, I'm not at all sure how the first two of these +can usefully be implemented. +------------------------------------------------------------------------------ + +(132) 16-Jan-03 M Option for when a transport filter fails (crashes) + +Freezing is one obvious option. Currently, Exim just retries. Another user +wanted to retry without the filter, but that is much harder. +------------------------------------------------------------------------------ + +(136) 24-Jan-03 M Make "personal" available as a condition for use in routers +------------------------------------------------------------------------------ + +(138) 28-Jan-03 M A variable containing what was matched in a host list + +Or, presumably, other lists. This is so that ACL messages can say things like +"your host name matches xxxx". Note: not the same as $domain_data. Also, this +could be tricky with lookups and things that match in files. +------------------------------------------------------------------------------ + +(143) 06-Mar-03 L Ability to have multiple authenticators of same type + +For example, to have two PLAIN authenticators; if the first fails, try the +second. +------------------------------------------------------------------------------ + +(144) 07-Mar-03 T ACL control = local_scan_skip to skip the local scan + +A bigger project would be control = local_scan <xxx> where xxx could select +different local_scan functions (possibly by dynamic loading). + +This can now be simulated using the fact that ACL variables are preserved, +so it doesn't look as it once did. +------------------------------------------------------------------------------ + +(145) 07-Mar-03 T Export string_cat() to local_scan() +------------------------------------------------------------------------------ + +(147) 17-Mar-03 T Option to treat 5xx as 4xx if received on initial connection + +This issue is controversial. That may be a good reason for not changing +anything. +------------------------------------------------------------------------------ + +(153) 25-Apr-03 S A way of making log_as_local apply to the smtp transport + +Either an option on the transport, or log_remote_as_local for the router. +Messy, either way. Maybe log_local_as_local and log_remote_as_local, and +deprecate log_as_local? +------------------------------------------------------------------------------ + +(154) 01-May-03 M Teergrubing at the CR/LF level + +It is believed that the most effective way to teergrube is to insert a delay +between transmitting CR and LF in the SMTP response. Furthermore, this is also +the best place to test for bad synchronization (i.e. at the last possible +time). +------------------------------------------------------------------------------ + +(155) 01-May-03 S "control=no_pipelining" for connect and EHLO ACLs + +Yet more flexibility! Maybe this should be a more general control for what is +sent in response to EHLO. +------------------------------------------------------------------------------ + +(156) 06-May-02 M Finer-grained synchronisation checking + +On operating systems that can be asked whether any sent bytes have not yet been +ACK'd at the TCP/IP level, a finer-grained check for proper synchronisation can +be done. All bytes must have been ACK'd if the client has received the previous +response before sending the next command. See also 293. +------------------------------------------------------------------------------ + +(157) 07-May-03 M Newline as a list item separator + +This will make life easier for lists obtained form databases where the +separator is naturally a newline. +------------------------------------------------------------------------------ + +(158) 13-May-03 M Ability to add to OK message for SMTP commands + +For sending reasons for slow response, etc. +------------------------------------------------------------------------------ + +(160) 19-May-03 M Remove headers using wild cards +------------------------------------------------------------------------------ + +(162) 28-May-03 M/L Use of real numbers in filters, expansions, and options + +The motivation for this is for handling spam scores that are real numbers. The +questions are (a) how widely should it spread and (b) whether floating point or +fixed point representations should be used. And what about the eval operator? +------------------------------------------------------------------------------ + +(164) 02-Jun-03 S Set variables for interface and port in smtp transport + +These could be useful for varying HELO data etc. See also several other +items about interfaces above. +------------------------------------------------------------------------------ + +(166) 18-Jun-03 S CN verification in client TLS code + +A tls_verify_cn option is suggested by Sven Geggus. +------------------------------------------------------------------------------ + +(168) 19-Jun-03 S Ability to add a header recording envelope rewrites + +Current code adds a deleted header with only some information. Maybe what is +needed is a flag for a rewrite rule. +------------------------------------------------------------------------------ + +(169) 19-Jun-03 M A way of detecting timeouts in callout returns + +------------------------------------------------------------------------------ + +(170) 23-Jun-03 S Option to accept rather than defer after local scan timeout + +Suggested patch supplied. +------------------------------------------------------------------------------ + +(171) 23-Jun-03 S Option to make timeout a soft failure on pipe transport + +------------------------------------------------------------------------------ + +(172) 23-Jun-03 M Option to make SQL query to specific server + +------------------------------------------------------------------------------ + +(175) 04-Jul-03 S show_all_ancestors_in_errmsg for the redirect router + +This is the opposite of hide_child_in_errmsg in effect. +------------------------------------------------------------------------------ + +(180) 14-Jul-03 M Extend never_users to be more flexible + +e.g. never_users = ! mailnull : ! cyrus : !mailman : 0-100 +------------------------------------------------------------------------------ + +(183) 16-Jul-03 S freeze_tell_text to add custom text to the message + +------------------------------------------------------------------------------ + +(185) 24-Jul-03 S An expansion operator that decodes RFC 2047 strings + +------------------------------------------------------------------------------ + +(188) 13-Aug-03 T batch_max=0 to mean unlimited + +------------------------------------------------------------------------------ + +(189) 22-Aug-03 S Allow filter "logwrite" to write to syslog + +I feel this is a dangerous facility, and also of very minority interest, at +least for user's filters. Allowing a system filter to write to mainlog or +syslog may be different. However, writing the main log would only be possible +if the filter runs as root or exim. +------------------------------------------------------------------------------ + +(190) 22-Aug-03 S A way of testing "forced delivery" in filter and routers +------------------------------------------------------------------------------ + +(191) 26-Aug-03 M Preserve $address_data for a verified recipient + +The idea is to preserve it in the recipients data structure so that local_scan +can have access to it. The value could also be used as the initial value of +$address_data while routing. +------------------------------------------------------------------------------ + +(192) 05-Sep-03 M Better handling of TXT records for dnslists + +When multiple lists are accessible via a merged lookup, handling TXT records +is difficult. An option for doing the TXT lookup in a sub-list has been +suggested, with syntax such as + + dnslists = list.example.org=127.0.0.2%dialups \ + ,127.0.0.3%relays \ + ,127.0.0.5%spews +------------------------------------------------------------------------------ + +(194) 10-Sep-03 M $addresslist_data to be like $host_data/$domain_data + +------------------------------------------------------------------------------ + +(195) 29-Sep-03 M A variable containing the error for verify = header_syntax + +Maybe there should always be a variable with the error message for all the +different kinds of verify failure. +------------------------------------------------------------------------------ + +(196) 30-Sep-03 S A way of detecting whether it was HELO or EHLO in the ACL + +$received_protocol isn't reset until after the command is accepted (which +seems right), and $smtp_data shows only the arguments. Maybe $smtp_command? +------------------------------------------------------------------------------ + +(197) 30-Sep-03 S MACROS_DROP_PRIVS and ALT_CONFIG_DROP_PRIVS + +Now that alternative configurations can be restricted to certain directories, +some more flexibility can be allowed. Not by default, though. +------------------------------------------------------------------------------ + +(198) 01-Oct-03 M Accept mail after local_scan() crash instead of defer + +This may not be as easy to implement as it sounds; one is never sure of the +environment after a crash. Is is actually a good idea? The crashing local_scan +may have wrecked the memory in arbitrary ways; for example, screwing up the +recipients list... +------------------------------------------------------------------------------ + +(199) 01-Oct-03 M ${pipe which will pipe the message to a script ... + +... and otherwise behave as ${run. Probably needs to have locking out features +so that it can be turned off for users .forwards if the sysadmin so desires. +------------------------------------------------------------------------------ + +(200) 07-Oct-03 L Alternative ways of storing hints + +People want to store hints in databases. Some assert that SQL databases can +be made to perform satisfactorily. If a general interface could be worked on, +people could at least try different strategies. See also 66 above, which is +specifically concerned with Berkeley DB. Another possible option is a switch to +disable smtp-wait hints - to avoid contention problems. +------------------------------------------------------------------------------ + +(201) 07-Oct-03 M A "soft bounce" feature + +This is an option that turns all hard bounces into soft bounces. The idea is +that it can be used as a safety-net while testing configurations. Instead of a +local bounce, the message stays on the queue; instead of 5xx SMTP responses, +4xx ones are given. + +The ability to do the opposite - turn 4xx into 5xx under certain circumstances +might also be useful (e.g. after a certain time). This might best be done by +extending the retry logic to recognize 4xx as a special error. (This is now +done.) +------------------------------------------------------------------------------ + +(202) 10-Oct-03 S -bvsomething to do a callout after the verify +------------------------------------------------------------------------------ + +(203) 14-Oct-03 S verify=something to easily check for header presence + +This is purely cosmetic; "condition" can already be used. +------------------------------------------------------------------------------ + +(204) 27-Oct-03 S an inverted queue_only_file + +That is, queue if a file does NOT exist. +------------------------------------------------------------------------------ + +(205) 27-Oct-03 S expand smtp_accept_queue_per_connection + +------------------------------------------------------------------------------ + +(206) 27-Oct-03 S appendfile: a variable containing the maildir base name + +------------------------------------------------------------------------------ + +(207) 29-Oct-03 S ability to keep trusted users in a file - expand it. +------------------------------------------------------------------------------ + +(208) 31-Oct-03 M cache temporary verification errors and fail after a time + +This request was for a way of turning temporary verification failures into +permanent ones after some fixed time. +------------------------------------------------------------------------------ + +(209) 31-Oct-03 S a way of making crashes in pipe commands temporary errors + +------------------------------------------------------------------------------ + +(210) 31-Oct-03 S runtime option to change the daemon name used for tcprwappers + +A patch for compile time was supplied, but this seems better as a runtime +option, for use with multiple Exim daemons. +------------------------------------------------------------------------------ + +(211) 31-Oct-03 S ability to disable debugging output from -bh & -bhc +------------------------------------------------------------------------------ + +(212) 31-Oct-03 M specify headers lines in HELO ACL to be added to all msgs +------------------------------------------------------------------------------ + +(214) 05-Nov-03 S Put the wild part of local part prefix/suffx in variables + +Unfortunately, this isn't quite as trivial as it seems. +------------------------------------------------------------------------------ + +(215) 14-Nov-03 S A way of turning off message-submission fix-ups + +Globally, and perhaps also via an ACL control so that it can be done on a +per-message basis. +------------------------------------------------------------------------------ + +(215) 26-Nov-03 M/L Conversion of IDNA domain names for logging + +IDNA (RFCs 3490-3492) converts domains names containing non-ASCII characters +into ASCII strings of a special form. Exim will of course handle these. +However, it might be nice to convert them to a local code for logging. This +might be quite a big project: there's also output from -bp and eximon queue +display and no doubt other places as well. (Utilities that process the logs, +e.g. exigrep, eximstats, will be automatically handled if the logs are +changed.) +------------------------------------------------------------------------------ + +(216) 27-Nov-03 S Option to bounce if required TLS doesn't happen + +This is for the smtp transport with hosts_require_tls set. Currently, it +defers. Possibly the best approach is to make the error one that can be seen by +the retry logic. +------------------------------------------------------------------------------ + +(217) 27-Nov-03 M A function to pass back variables from Perl + +This is a function that can be called from Perl, to take a name and a value and +put that value into an Exim variable. +------------------------------------------------------------------------------ + +(218) 01-Dec-03 M A local_scan-like hook at system filter time + +That is, make a C API available for custom filtering at this point. +------------------------------------------------------------------------------ + +(221) 18-Dec-03 U Merge routers and ACLs - or at least make more similar + +"It will be very useful to be able to use most of the ACL conditions +(authenticated, hosts, senders, sender_domains, ... ) in routers and also the +possibility to have multiple conditions in routers. It will be great to also +be able to set variables in routers like in acl's." This is effectively a +radical suggestion for a complete re-design, and is therefore BIG. +------------------------------------------------------------------------------ + +(222) 19-Dec-03 S Iterative option for dnsdb + +A way of getting a dnsdb lookup to chop off components until something is +found: e.g. ${lookup dndsb-i{ns=a.b.c.d}} would look for nameservers for +a.b.c.d, then b.c.d, etc. +------------------------------------------------------------------------------ + +(223) 22-Dec-03 S Support SOA lookup in dnsdb lookups +------------------------------------------------------------------------------ + +(225) 22-Dec-03 M Add acl= to routers + +This would use an ACL to "control access" to a router, opening up a number +of interesting possibilities. Details of possible limitations need to be +investigated. +------------------------------------------------------------------------------ + +(226) 23-Dec-03 S A way of treating DEFER as fail in dnsdb lookups + +(i.e. the dnsdb lookup failed, so accept the message) +------------------------------------------------------------------------------ + +(227) 30-Jan-04 M A configuration .if facility + +"Second with the .ifdef and such, it would be nice to have a base .if, +so I could do something like +.if DEFINED_DATA == xyz +configuration here +.elseif DEFINED_DATA == abc +configuration here +.else +configuration here +.endif +also this would be nice at least in my case in the system filters, but +isn't required but you could pass the defined data to the system, in +variables." +------------------------------------------------------------------------------ + +(229) 30-Jan-04 M New expansion mechanism: {list ...} + +"Proposed syntax: {list {separator}{item}{item}...} +This first expands the contents of {separator} and all of the {item}s, +then constructs a separator-delimited list. The twist is: if an {item} +generates the empty string, no separator will be generated for it. +The entire construct will fail is {separator} fails, or all {item}s +fail. If just some {item}s fail, they will be treated as if they +generated empty strings. +Examples: + {list {,}{aaaaaa}{bbbbbb}{cccccc}} -> aaaaaa,bbbbbb,cccccc + {list {,}{:fail:}{bbbbbb}{cccccc}} -> bbbbbb,cccccc + {list {,}{aaaaaa}{:fail:}{cccccc}} -> aaaaaa,cccccc + {list { }{aaaaaa}{bbbbbb}{}} -> aaaaaa bbbbbb + {list { }{:fail:}{:fail:}{:fail:}} -> :fail: + {list {:fail:}{aaaaa}{bbbb}{cccc}} -> :fail: +See particularly examples 2-4, which handle the case of a missing first +and last item with ease; doing this using {if ...} would be quite difficult!" +------------------------------------------------------------------------------ + +(230) 30-Jan-04 M Find IP addresses of a domain's nameservers + +This needs some way of processing a list of things in a similar way, which +should perhaps be a more general facility. +------------------------------------------------------------------------------ + +(231) 30-Jan-04 ? -C has a number of problems when used for real + +-C was intended for testing; people are using it for "alternate" +configurations, and it doesn't work too well. Can a better way of doing this be +invented? +------------------------------------------------------------------------------ + +(232) 02-Feb-04 ? Make parts of the code loadable + +The idea being that drivers, etc. could be compiled separately. There are, of +course, security issues. This is not something I want to go into at present. +------------------------------------------------------------------------------ + +(235) 02-Feb-04 T Make smtp_accept_count available as a variable + +This is for use in ACLs. Of course, it is a snapshot of the count at the +start of the receiving process. +------------------------------------------------------------------------------ + +(236) 02-Feb-04 S String in local_scan that's added to the binary version string +------------------------------------------------------------------------------ + +(237) 02-Feb-04 M Add_header in ACLs because "message" is overloaded + +This would be useful for verbs where "message" is an error message. +------------------------------------------------------------------------------ + +(238) 05-Feb-04 S ${address to handle multiple addresses + +At present, ${address expects to see just one address. An extension would let +it handle header lines with multiple addresses, just retaining the actual +addresses. Or perhaps a new operator is needed? +------------------------------------------------------------------------------ + +(239) 23-Feb-04 ? Expansion items for encryption/decryption + +Perhaps for some kind of cookie handling? This would need an external crypto +library, because there's no crypto code in Exim itself. +------------------------------------------------------------------------------ + +(240) 23-Feb-04 ? Some way to know if a ip is a mx for a given domain + +Some kind of iterative operation for dnsdb might be a general way of providing +this. +------------------------------------------------------------------------------ + +(242) 01-Mar-04 ? Run a filter from an expansion condition + +This would add a lot of power to ACLs, but its implementation might be tricky +because of the possibility of recursion. +------------------------------------------------------------------------------ + +(243) 01-Mar-04 ? Run an ACL from an expansion condition + +The problem here is knowing what data is available at an arbitrary time. +------------------------------------------------------------------------------ + +(244) 01-Mar-04 ? Add an on-success event to transports + +This could just be an expansion string, whose value is either ignored or +logged, but it could be used to run SQL updates or run programs etc. +However, what is "success" when a transport has multiple recipients? +------------------------------------------------------------------------------ + +(245) 01-Mar-04 M Add all the string expansion conditions to filters + +Some thought would be needed on how to design the syntax for this. +------------------------------------------------------------------------------ + +(247) 09-Mar-04 S IP addresses that are never looked up + +It would be nice if we could prevent this for certain IP addresses for +which we _know_ we'll never get a valid PTR record, like 2002::/16. +So a new option might reasonably default to: + + hosts_never_lookup = <; 2002::/16 +------------------------------------------------------------------------------ + +(253) 05-Apr-04 M Use ESMTP and TLS for recipient callout verification + +The best way to do this would involve quite a bit of refactoring so as to +abstract some of the code from the smtp transport into subroutines that could +also be used from the callout code. The tls parameters should probably be +taken from the transport. That might also require some substantial code +refactoring. See also 294. +------------------------------------------------------------------------------ + +(260) 30-Apr-04 S Respect +tls_cipher +tls_peerdn in rejectlog entries + +------------------------------------------------------------------------------ + +(261) 05-May-04 S Add a "required_version" option + +So that configurations can insist on a specific Exim version. +------------------------------------------------------------------------------ + +(262) 10-May-04 S Add "scratch" ACL variables + +The idea is for variables that are flushed at the start of each ACL. I'm not +really convinced that these are worth implementing. +------------------------------------------------------------------------------ + +(263) 10-May-04 S Add variable $router_name $transport_name + +These could be used in debug_print settings, which are output during -bt, and +thus don't need the privilege to run with -d. +------------------------------------------------------------------------------ + +(265) 25-May-04 M An init.d script for exim is needed + +The old sendmail script used to "just work" because it just did -bd -q 20m or +whatever. Newer versions start more than one sendmail daemon, so do not work. +------------------------------------------------------------------------------ + +(267) 25-May-04 S tarpitting delay option + +A modifier that sets a delay between lines for multiline responses. +------------------------------------------------------------------------------ + +(268) 25-May-04 S? Add a PID to every log line + +Given that pids are reused non-cyclically these days, is this actually useful? +------------------------------------------------------------------------------ + +(269) 26-May-04 U Run both a system and a user filter in test mode + + exim -bF systemfilter -bf userfilter -f sender@dom < message + +This would allow testing the way the userfilter handles the system +variables set by the systemfilter. +------------------------------------------------------------------------------ + +(270) 01-Jun-04 M Add headers at top and middle + +Various initiatives like SPF and DomainKeys require header lines to be added +above or in the middle of existing headers. Exim always adds at the bottom. +When these requirements are more standard and clearer, some way of controlling +where header lines are added will probably become necessary. Some new syntax +will be required. + +This can now be done fairly generally from local_scan(), and at the start and +after the Received: block from an ACL. Is anything more needed? +------------------------------------------------------------------------------ + +(271) 02-Jun-04 L Callouts at routing time + +From a user's message: + +> I would like to be able to: +>[...] +> 2) Forcing callouts as address verification at router level +> [ check_callout just like check_local_user ] +> +> I would like to redirect messages in some domain to "domain with callout +> verification" and to "domain without callout verification" +> +> e.g. +> userA@domain.in -> userX@doamin.out-verify (use callout to verify) +> userB@domain.in -> userY@doamin.out-noverify (do not use callout verify) +> +> [both out-* domains delivered via "callout ready" transports] + +Other versions of the wish: +* limiting callouts in acls to specific transport + verify = recipient/callout=5s,transport:intranet_smtp +* adding "select transport" to ACL conditions + accept domains = +local_domains + transport = cyrus_ltcp + verify = recipient/callout=5s +------------------------------------------------------------------------------ + +(272) 07-Jun-04 S Expand hosts_randomize + +It occurs in manualroute and in smtp. +------------------------------------------------------------------------------ + +(278) 21-Jun-04 M quota_warn_message_file option + +Similar to the bounce and delivery warn message files. +------------------------------------------------------------------------------ + +(280) 23-Jun-04 M A way of adding a header line after callout defer_ok + +This would record that, e.g., a sender domain verified, but the callout +could not be done. +------------------------------------------------------------------------------ + +(285) 16-Jul-04 M Separate and independent log_selector for rejectlog + +For example: mainlog_selector and rejectlog_selector, with log_selector setting +both of them. +------------------------------------------------------------------------------ + +(286) 21-Jul-04 M Distinguishing a larger number of errors + +For instance, detecting "connection reset by peer" (ENETRESET or ECONNRESET) +might be useful. +------------------------------------------------------------------------------ + +(288) 10-Aug-04 M Option for verify to require MX + +e.g. verify=sender/require_mx +I'm not too keen because this is rather special purpose, and of course could +only apply if the verification happened to hit a dnslookup router. +------------------------------------------------------------------------------ + +(289) 10-Aug-04 L Option to treat defers in database lookups as "not found" + +This is so that alternatives can be coded for when databases are down. A +suggested patch has been sent, but it just catches all instances of "defer" +from a lookup in an expansion string. These can occur for a number of different +reasons, not just connection failures. I think that we need a specific +"connection failed" indicator. Also, what about lookups in lists? +------------------------------------------------------------------------------ + +(291) 13-Aug-04 M An ACL or "local_scan()" to be run on size excession + +The idea is to give something a chance to look at the data so far received when +more than message_size_limit (or some other limit) has arrived. I am not sure +how useful this would actually be in practice. +------------------------------------------------------------------------------ + +(292) 13-Aug-04 M Overall timeout for message reception + +A client could in priciple keep an SMTP connection open for a very long time by +trickling in data very slowly. Also, after message_size_limit is exceeded, Exim +continues to swallow the data (though it does not write it to disk) until the +end is reached. Again, the connection could be held open for a very long time. +Some kind of overall time limit for an SMTP connection, possibly reset at the +start of each message, might be helpful in these situations. +------------------------------------------------------------------------------ + +(294) 23-Aug-04 L Callouts and AUTH and LMTP + +People want to do callouts using LMTP as well as SMTP, and that would also +include sockets as well as TCP/IP connections. Also, people want to make use of +AUTH during the callout checking, on all types of connection. I suppose that +means making TLS available as well. This probably means a rewrite of the code +that actually does the callout. Should we use the relevant transport in a new +"callout" mode instead of keeping things separate? See also 253. +------------------------------------------------------------------------------ + +(296) 09-Sep-04 S Make deliver_time work for == lines as well as => + +What about ** lines? +------------------------------------------------------------------------------ +--- HWM 297 ------------------------------------------------------------------ +---------------------------- End of WishList --------------------------------- diff --git a/doc/doc-scripts/ABOUT b/doc/doc-scripts/ABOUT new file mode 100644 index 000000000..623e88c2e --- /dev/null +++ b/doc/doc-scripts/ABOUT @@ -0,0 +1,9 @@ +$Cambridge: exim/doc/doc-scripts/ABOUT,v 1.1 2004/10/08 10:38:48 ph10 Exp $ + +CVS directory exim/exim-doc/doc-scripts +--------------------------------------- + +This directory contains various scripts that are used to build the distributed +documentation from its source files. + +End diff --git a/doc/doc-src/ABOUT b/doc/doc-src/ABOUT new file mode 100644 index 000000000..9e281e996 --- /dev/null +++ b/doc/doc-src/ABOUT @@ -0,0 +1,11 @@ +$Cambridge: exim/doc/doc-src/ABOUT,v 1.1 2004/10/08 10:38:48 ph10 Exp $ + +CVS directory exim/exim-doc/doc-src +----------------------------------- + +This directory contains documentation files that are processed in some way in +order to make the documentation files that form part of Exim distributions. A +non-standard document processor is currently in use (October 2004), but in the +long term something more standard will have to take over. + +End diff --git a/doc/doc-txt/ABOUT b/doc/doc-txt/ABOUT new file mode 100644 index 000000000..b9c355a5a --- /dev/null +++ b/doc/doc-txt/ABOUT @@ -0,0 +1,9 @@ +$Cambridge: exim/doc/doc-txt/ABOUT,v 1.1 2004/10/08 10:38:48 ph10 Exp $ + +CVS directory exim/exim-doc/doc-txt +----------------------------------- + +This directory contains various documentation files that exist only as plain +text files, and are distributed in that format. + +End diff --git a/src/ABOUT b/src/ABOUT new file mode 100644 index 000000000..35857bf71 --- /dev/null +++ b/src/ABOUT @@ -0,0 +1,11 @@ +$Cambridge: exim/src/ABOUT,v 1.1 2004/10/08 10:38:48 ph10 Exp $ + +CVS directory exim/exim-src +--------------------------- + +This directory contains everything that is included in an Exim distribution +tarball, with the exception of the doc directory and an empty Local directory. +You can build Exim from the contents of this directory by adding a Local +directory that contains appropriate configuration files. + +End