1 EXPORTABLE EXIM TEST SUITE
2 --------------------------
4 This document last updated for:
6 Test Suite Version: 4.87
13 For a long time, the Exim test suite was confined to Philip Hazel's
14 workstation, because it relied on that particular environment. The problem is
15 that an MTA such as Exim interacts a great deal with its environment, so if you
16 run it somewhere else, the output will be different, which makes automatic
17 checking difficult. Even in a single environment, things are not all that easy.
18 For instance, if Exim delivers a message, the log line (which one would want to
19 compare) contains a timestamp and an Exim message id that will be different
20 each time. This issue is dealt with by a Perl script that munges the output by
21 recognizing changing sequences and replacing them with fixed values before
22 doing a comparison. Another problem with exporting the original test suite is
23 that it assumes a version of Exim with more or less every optional feature
26 This README describes a new test suite that is intended to be exportable and to
27 run in a number of different environments. The tests themselves are in no
28 particular order; they accumulated over the years as Exim was extended and
29 modified. They vary greatly in size and complexity. Some were specifically
30 constructed to test new features; others were made to demonstrate that a bug
33 A few of the original tests have had to be omitted from this more general
34 suite because differences in operating system behaviour make it impossible to
35 generalize them. An example is a test that uses a version of Exim that is
36 setuid to the Exim user rather than root, with the deliver_drop_privilege
37 option set. In Linux, such a binary is able to deliver a message as the caller
38 of Exim, because it can revert to the caller's uid. In FreeBSD this is not the
45 In order to run this test suite, the following requirements must be met:
47 (1) You should run the tests on a matching version of Exim, because the suite
48 is continuously updated to test the latest features and bug fixes. The
49 version you test does not, however, have to be installed as the live
50 version. You can of course try the tests on any version of Exim, but some
51 may fail. In particular, the test suite will fall apart horrible with
52 versions of Exim prior to 4.54.
54 (2) You can use any non-root login to run the tests, but there must be access
55 via "sudo" to root from this login. Privilege is required to override
56 configuration change checks and for things like cleaning up spool files,
57 but on the other hand, the tests themselves need to call Exim from a
58 non-root process. The use of "sudo" is the easiest way to achieve all this.
59 The test script uses "sudo" to do a number of things as root, so it is best
60 if you set a sudo timeout so that you do not have to keep typing a
61 password. For example, if you put
63 Defaults timestamp_timeout=480
65 in /etc/sudoers, a password lasts for 8 hours (a working day). It is
66 not permitted to run the tests as the Exim user because the test suite
67 tracks the two users independently. Using the same user would result
68 in false positives on some tests.
70 Further, some tests invoke sudo in an environment where there might not be
71 a TTY, so tickets should be global, not per-TTY. Taking this all together
72 and assuming a user of "exim-build", you might have this in sudoers:
74 Defaults:exim-build timestamp_timeout=480,!tty_tickets
76 (3) The login under which you run the tests must have the exim group as a
77 secondary so that it has access to logs, spool files, etc. However, it
78 should have a different primary group (eg. "users" vs. "eximgroup"). The
79 login should not be one of the names "userx", "usery", "userz", or a few
80 other simple ones such as "abcd" and "xyz" and single letters that are used
81 in the tests. The test suite expects the login to have a gecos name; I think
82 it will now run if the gecos field is empty but there may be anomalies.
83 The login must not contain a dash or an equal sign. (Otherwise some tests
84 about local_from_{suffix,prefix} will fail.)
86 (4) The directory into which you unpack the test suite must be accessible by
87 the Exim user, so that code running as exim can access the files therein.
88 This includes search-access on all path elements leading to it. A
89 world-readable directory is fine. However, there may be problems if the
90 path name of the directory is excessively long. This is because it
91 sometimes appears in log lines or debug output, and if it is truncated, it
92 is no longer recognized.
94 (5) Exim must be built with its user and group specified at build time, and
95 with certain minimum facilities, namely:
97 Routers: accept, dnslookup, manualroute, redirect
98 Transports: appendfile, autoreply, pipe, smtp
100 Authenticators: plaintext
102 Most Exim binaries will have these included.
104 (6) A C compiler is needed to build some test programs, and the test script is
105 written in Perl, so you need that.
107 (7) Some of the tests run Exim as a daemon, and others use a testing server
108 (described below). These require TCP ports. In the configurations and
109 scripts, the ports are parameterized, but at present, fixed values are
110 written into the controlling script. These are ports 1224 to 1229. If these
111 ports are not available for use, some of the tests will fail.
113 (8) There is an underlying assumption that the host on which the tests are
114 being run has an IPv4 address (which the test script seeks out). If there
115 is also an IPv6 address, additional tests are run when the Exim binary
116 contains IPv6 support. There are checks in the scripts for a running IPv4
117 interface; when one is not found, some tests are skipped (with a warning
118 message). The local net may not be in 10.250.0/16 as that is used by the suite.
120 (9) Exim must be built with TRUSTED_CONFIG_LIST support, so that the test
121 configs can be placed into it. A suitable file location is .../exim/test/trusted_configs
122 with content .../exim/test/test-config [fill out the ... to make full
123 paths]. This file should be owner/group matching CONFIGURE_OWNER/GROUP,
124 or root/root, and it has to be accessible for the login, under which
125 you run the tests. The config files in .../exim/test/confs/ should be
126 owner/group the same. DISABLE_D_OPTION must not be used. If ALT_CONFIG_PREFIX is used, it
127 must contain the directory of the test-suite. WHITELIST_D_MACROS should contain:
129 DIR:EXIM_PATH:AA:ACL:ACLRCPT:ACL_MAIL:ACL_PREDATA:ACL_RCPT:AFFIX:ALLOW:ARG1:ARG2:AUTHF:AUTHS:AUTH_ID_DOMAIN:BAD:BANNER:BB:BR:BRB:CERT:COM:COMMAND_USER:CONNECTCOND:CONTROL:CREQCIP:CREQMAC:CRL:CSS:D6:DATA:DCF:DDF:DEFAULTDWC:DELAY:DETAILS:DRATELIMIT:DYNAMIC_OPTION:ELI:ERROR_DETAILS:ERT:FAKE:FALLBACK:FILTER:FILTER_PREPEND_HOME:FORBID:FORBID_SMTP_CODE:FUSER:HAI:HAP:HARDLIMIT:HEADER_LINE_MAXSIZE:HEADER_MAXSIZE:HELO_MSG:HL:HOSTS:HOSTS_AVOID_TLS:HOSTS_MAX_TRY:HVH:IFACE:IGNORE_QUOTA:INC:INSERT:IP1:IP2:LAST:LDAPSERVERS:LENCHECK:LIMIT:LIST:LOG_SELECTOR:MAXNM:MESSAGE_LOGS:MSIZE:NOTDAEMON:ONCE:ONLY:OPT:OPTION:ORDER:PAH:PEX:PORT:PTBC:QDG:QOLL:QUOTA:QUOTA_FILECOUNT:QWM:RCPT_MSG:REMEMBER:REQUIRE:RETRY:RETRY1:RETRY2:RETURN:RETURN_ERROR_DETAILS:REWRITE:ROUTE_DATA:RRATELIMIT:SELECTOR:SELF:SERVER:SERVERS:SREQCIP:SREQMAC:SRV:STRICT:SUB:SUBMISSION_OPTIONS:TIMEOUTDEFER:TIMES:TRUSTED:TRYCLEAR:UL:USE_SENDER:UTF8:VALUE:WMF
131 (10) Exim must *not* be built with USE_READLINE, as the test-suite's automation
132 assumes the simpler I/O model.
133 Exim must *not* be built with HEADERS_CHARSET set to UTF-8.
135 (11) If building any dynamically-loaded lookups, LOOKUP_MODULE_DIR should
136 be set to .../exim/test/eximdir.
143 If the Exim binary that is being tested contains extra functionality in
144 addition to the minimum specified above, additional tests are run to exercise
145 the extra functionality, except for a few special cases such as the databases
146 (MySQL, PostgreSQL, LDAP) where special data is needed for the tests.
149 RUNNING THE TEST SUITE
150 ----------------------
152 (1) Clone the git tree for Exim. This include both the Exim source and the
155 (2) cd into the test/ subdirectory (where this README lives).
157 (3) Run "./configure" and then "make". This builds a few auxiliary programs that
160 (4) echo $PWD/test-config >> your_TRUSTED_CONFIG_LIST_filename
161 Typically that is .../exim/test/trusted_configs
163 (5) Run "./runtest" (a Perl script) as described below.
165 (6) If you want to see what tests are available, run "./listtests".
168 BREAKING OUT OF THE TEST SCRIPT
169 -------------------------------
171 If you abandon the test run by typing ^C, the interrupt may be passed to a
172 program that the script is running, or it may be passed to the script itself.
173 In the former case, the script should detect that the program has ended
174 abnormally. In both cases, the script tries to clean up everything, including
175 killing any Exim daemons that it has started. However, there may be race
176 conditions in which the clean up does not happen. If, after breaking out of a
177 run, you see strange errors in the next run, look for any left-over Exim
178 daemons, and kill them by hand.
184 The individual test scripts are in subdirectories of the "scripts" directory.
185 If you do not supply any arguments to ./listtests, it scans all the scripts in
186 all the directories, and outputs the heading line from each script. The output
187 is piped through "less", and begins like this:
190 Basic/0001 Basic configuration setting
191 Basic/0002 Common string expansions
192 Basic/0003 Caseless address blocking
195 Lines that start === give the name of the subdirectory containing the test
196 scripts that follow. If you supply an argument to ./listtests, it is used as a
197 Perl pattern to match case-independently against the names of the
198 subdirectories. Only those that match are scanned. For example, "./listtests
201 === 1000-Basic-ipv6 ===
202 === Requires: support IPv6
203 Basic-ipv6/1000 -bh and non-canonical IPv6 addresses
204 Basic-ipv6/1001 recognizing IPv6 address in HELO/EHLO
206 === 2250-dnsdb-ipv6 ===
207 === Requires: support IPv6
209 dnsdb-ipv6/2250 dnsdb ipv6 lookup in string expansions
211 If you supply a second argument to ./listtests, it is used as a Perl pattern to
212 match case-independently against the individual script titles. For example,
213 "./listtests . mx" lists all tests whose titles contain "mx", because "."
214 matches all the subdirectory names.
220 If you do not supply any arguments to ./runtest, it searches for an Exim
221 source tree at the same level as the test suite directory. A source tree
222 is a source tree, if it contains a build-* directory.
224 It then looks for an Exim binary in a "build" directory of that source
225 tree. If there are several Exim source trees, it chooses the latest
226 version of Exim. Consider the following example:
229 exim-4.60/ exim-4.62/ exim-testsuite-x.xx/
231 A simple ./runtest from within the test suite will use a 4.62 binary if it
232 finds one, otherwise a 4.60 binary. If a binary cannot be found, the script
233 prompts for one. Alternatively, you can supply the binary on the command line:
235 ./runtest /usr/exim/bin/exim
237 A matching test suite is released with each Exim release; if you use a test
238 suite that does not match the binary, some tests may fail.
240 The test suite uses some of the Exim utilities (such as exim_dbmbuild), and it
241 expects to find them in the same directory as Exim itself. If they are not
242 found, the tests that use them are omitted. A suitable comment is output.
244 On the ./runtest command line, following the name of the binary, if present,
245 there may be a number of options and then one or two numbers. The full syntax
248 ./runtest [binary name] [runtest options] [exim options] \
249 [first test] [last test]
251 There are some options for the ./runtest script itself:
253 -CONTINUE This will allow the script to move past some failing tests. It will
254 write a simple failure line with the test number in a temporary
255 logfile test/failed-summary.log. Unexpected exit codes will still
256 stall the test execution and require interaction.
258 -DEBUG This option is for debugging the test script. It causes some
259 tracing information to be output.
261 -DIFF By default, file comparisons are done using a private compare
262 command called "cf", which is built from source that is provided in
263 the src directory. This is a command I've had for nearly 20 years -
264 look at the source comments for its history - whose output I
265 prefer. However, if you want to use "diff" instead, give -DIFF as a
266 runtest option. In that case, "diff -u" is used for comparisons.
267 (If it turns out that most people prefer to use diff, I'll change
272 This allows "overrides" for the test results. It's intended
273 use is to deal with distro specific differences in the test
274 output. The default flavour is "FOO" if autodetection fails.
275 (Autodetection is possible for known flavours only. Known
276 flavours are computed after file name extensions in stdout/*
279 If during the test run differences between the current and
280 the expected output are found and no flavour file exists already,
281 you may update the "common" expected output or you may create a
282 flavour file. If a flavour file already exists, any updates will go
283 into that flavour file!
285 -KEEP Normally, after a successful run, the test output files are
286 deleted. This option prevents this. It is useful when running a
287 single test, in order to look at the actual output before it is
288 modified for comparison with saved output.
290 -NOIPV4 Pretend that an IPv4 interface was not found. This is useful for
291 testing that the test suite correctly skips tests that require
292 a running IPv4 interface.
294 -NOIPV6 Pretend that an IPv6 interface was not found. This is useful for
295 testing that the test suite correctly skips tests that require
296 a running IPv6 interface.
298 -UPDATE If this option is set, any detected changes in test output are
299 automatically accepted and used to update the stored copies of the
300 output. It is a dangerous option, but it useful for the test suite
301 maintainer after making a change to the code that affects a lot of
302 tests (for example, the wording of a message).
304 -SLOW For very slow hosts that appear to have Heisenbugs, delay before
305 comparing output files from a testcase
307 -TLS <client> For cross-library testing. Specify 'openssl" or 'gnutls'
308 as the client; the other is used as the server (assumes that
309 both have been built: set up Local/Makefile for OpenSSL and
310 "make exim_openssl", then for GnuTLS and "make exim_gnutls")
312 The options for ./runtest must be given first (but after the name of the
313 binary, if present). Any further options, that is, items on the command line
314 that start with a hyphen, are passed to the Exim binary when it is run as part
315 of a test. The only sensible use of this is to pass "-d" in order to run a test
316 with debugging enabled. Any other options are likely to conflict with options
317 that are set in the tests. Some tests are already set up to run with debugging.
318 In these cases, -d on the command line overrides their own debug settings.
320 The final two arguments specify the range of tests to be run. Test numbers lie
321 in the range 1 to 9999. If no numbers are given, the defaults are 1 and 8999
322 (sic). Tests with higher numbers (9000 upwards) are not run automatically
323 because they require specific data (such as a particular MySQL table) that is
324 unlikely to be generally available.
326 Tests that require certain optional features of Exim are grouped by number, so
327 in any given range, not all the tests will exist. Non-existent tests are just
328 skipped, but if there are no tests at all in the given range, a message is
331 If you give only one number, just that test is run (if it exists). Instead of a
332 second number, you can give the character "+", which is interpreted as "to the
333 end". Normally this is 8999; if the starting number is 9000 or higher, "+" is
334 interpreted as 9999. Examples:
338 ./runtest /usr/sbin/exim 5000 +
339 ./runtest -DIFF -d 81
341 When the script starts up, the first thing it does is to check that you have
342 sudo access to root. Then it outputs the version number of the Exim binary that
343 it is testing, and also information about the optional facilities that are
344 present (obtained from "exim -bV"). This is followed by some environmental
345 information, including the current login id and the hosts's IP address. The
346 script checks that the current user is in the Exim group, and that the Exim
347 user has access to the test suite directory.
349 The script outputs the list of tests requested, and a list of tests that will
350 be omitted because the relevant optional facilities are not in the binary. You
351 are then invited to press Return to start the tests running.
357 When all goes well, the only permanent output is the identity of the tests as
358 they are run, and "Script completed" for each test script, for example:
360 Basic/0001 Basic configuration setting
362 Basic/0002 Basic string expansions
364 Basic/0003 Caseless address blocking
366 Basic/0004 Caseful address blocking
368 Basic/0005 -bs to simple local delivery
371 While a script is running, it shows "Test n" on the screen, for each of the
372 Exim tests within the script. There may also be comments from some tests when a
373 delay is expected, for example, if there is a "sleep" while testing a timeout.
375 Before each set of optional tests, an extra identifying line is output. For
378 >>> The following tests require: authenticator cram_md5
379 CRAM-MD5/2500 CRAM-MD5 server tests
381 CRAM-MD5/2501 CRAM-MD5 client tests
384 If a test fails, you are shown the output of the text comparison that failed,
385 and prompted as to what to do next. The output is shown using the "less"
386 command, or "more" if "less" is not available. The options for "less" are set
387 to that it automatically exits if there is less that a screenful of output. By
388 default, the output is from the "cf" program, and might look like this:
390 DBM/1300 DBM files and exim_dbmbuild
392 Lines 7-9 of "test-stdout-munged" do not match lines 7-11 of "stdout/1300".
394 exim_dbmbuild exit code = 1
395 Continued set of lines is too long: max permitted length is 99999
396 exim_dbmbuild exit code = 1
399 exim_dbmbuild exit code = 2
400 Continued set of lines is too long: max permitted length is 99999
402 exim_dbmbuild exit code = 2
405 "test-stdout-munged" contains 16 lines; "stdout/1300" contains 18 lines.
407 Continue, Retry, Update & retry, Quit? [Q]
409 This example was generated by running the test with a version of Exim
410 that had a bug in the exim_dbmbuild utility (the bug was fixed at release
411 4.53). See "How the tests work" below for a description of the files that are
412 used. In this case, the standard output differed from what was expected.
414 The reply to the prompt must either be empty, in which case it takes the
415 default that is given in brackets (in this case Q), or a single letter, in
416 upper or lower case (in this case, one of C, R, U, or Q). If you type anything
417 else, the prompt is repeated.
419 "Continue" carries on as if the files had matched; that is, it ignores the
420 mismatch. Any other output files for the same test will be compared before
421 moving on to the next test.
423 "Update & retry" copies the new file to the saved file, and reruns the test
424 after doing any further comparisons that may be necessary.
426 "Retry" does the same apart from the file copy.
428 Other circumstances give rise to other prompts. If a test generates output for
429 which there is no saved data, the prompt (after a message stating which file is
430 unexpectedly not empty) is:
432 Continue, Show, or Quit? [Q]
434 "Show" displays the data on the screen, and then you get the "Continue..."
435 prompt. If a test ends with an unexpected return code, the prompt is:
437 show stdErr, show stdOut, Continue (without file comparison), or Quit? [Q]
439 Typically in these cases there will be something interesting in the stderr
440 or stdout output. There is a similar prompt after the "server" auxiliary
444 OPENSSL AND GNUTLS ERROR MESSAGES
445 ---------------------------------
447 Some of the TLS tests deliberately cause errors to check how Exim handles them.
448 It has been observed that different releases of the OpenSSL and GnuTLS
449 libraries generate different error messages. This may cause the comparison with
450 the saved output to fail. Such errors can be ignored.
456 . Some of the tests are time-sensitive (e.g. when testing timeouts, as in test
457 461). These may fail if run on a host that is also running a lot of other
460 . Some versions of "ls" use a different format for times and dates. This can
461 cause test 345 to fail.
463 . Test 0142 tests open file descriptors; on some hosts the output may vary.
465 . Some tests may fail, for example 0022, because it says it uses cached data
466 when the expected output thinks it should not be in cache. Item #5 in the
467 Requirements section has:
468 "Exim must be built with its user and group specified at build time"
469 This means that you cannot use the "ref:username" in your Local/Makefile
470 when building the exim binary, in any of the following fields:
471 EXIM_USER EXIM_GROUP CONFIGURE_OWNER CONFIGURE_GROUP
473 . If the runtest script warns that the hostname is not a Fully Qualified
474 Domain Name (FQDN), expect that some tests will fail, for example 0036,
475 with an extra log line saying the hostname doesn't resolve. You must use a
476 FQDN for the hostname for proper test functionality.
478 . If you change your hostname to a FQDN, you must delete the test/dnszones
479 subdirectory. When you next run the runtest script, it will rebuild the
480 content to use the new hostname.
482 . If your hostname has an uppercase characters in it, expect that some tests
483 will fail, for example, 0036, because some log lines will have the hostname
484 in all lowercase. The regex which extracts the hostname from the log lines
485 will not match the lowercased version.
487 . Some tests may fail, for example 0015, with a cryptic error message:
488 Server return code 99
489 Due to security concerns, some specific files MUST have the group write bit
490 off. For the purposes of the test suite, some test/aux-fixed/* files MUST
491 have the group write bit off, so it's easier to just remove the group write
492 bit for all of them. If your umask is set to 002, the group write bit will
493 be on by default and you'll see this problem, so make sure your umask is
494 022 and re-checkout the test/ subdirectory.
496 . Some tests will fail if the username and group name are different. It does
497 not have to be the primary group, a secondary group is sufficient.
500 OTHER SCRIPTS AND PROGRAMS
501 --------------------------
503 There is a freestanding Perl script called "listtests" that scans the test
504 scripts and outputs a list of all the tests, with a short descriptive comment
505 for each one. Special requirements for groups of tests are also noted.
507 The main runtest script makes use of a second Perl script and some compiled C
510 patchexim A Perl script that makes a patched version of Exim (see the
511 next section for details).
513 bin/cf A text comparison program (see above).
515 bin/checkaccess A program that is run as root; it changes uid/gid to the
516 Exim user and group, and then checks that it can access
517 files in the test suite's directory.
519 bin/client A script-driven SMTP client simulation.
521 bin/client-gnutls A script-driven SMTP client simulation with GnuTLS support.
522 This is built only if GnuTLS support is detected on the host.
524 bin/client-ssl A script-driven SMTP client simulation with OpenSSL support.
525 This is built only if OpenSSL support is detected on the
528 bin/client-anytls A symlink to either client-ssl or client-gnutls, if
531 bin/fakens A fake "nameserver" for DNS tests (see below for details).
533 bin/fd A program that outputs details of open file descriptors.
535 bin/iefbr14 A program that does nothing, and returns 0. It's just like
536 the "true" command, but it is in a known place.
538 bin/loaded Some dynamically loaded functions for testing dlfunc support.
540 bin/mtpscript A script-driven SMTP/LMTP server simulation, on std{in,out}.
542 bin/server A script-driven SMTP server simulation, over a socket.
544 bin/showids Output the current uid, gid, euid, egid.
546 The runtest script also makes use of a number of ordinary commands such as
547 "cp", "kill", "more", and "rm", via the system() call. In some cases these are
548 run as root by means of sudo.
551 STANDARD SUBSTITUTIONS
552 ----------------------
554 In the following sections, there are several references to the "standard
555 substitutions". These make changes to some of the stored files when they are
556 used in a test. To save repetition, the substitutions themselves are documented
559 CALLER is replaced by the login name of the user running the tests
560 CALLERGROUP is replaced by the caller's group id
561 CALLER_GID is replaced by the caller's group id
562 CALLER_UID is replaced by the caller's user id
563 DIR is replaced by the name of the test-suite directory
564 EXIMGROUP is replaced by the name of the Exim group
565 EXIMUSER is replaced by the name of the Exim user
566 HOSTIPV4 is replaced by the local host's IPv4 address
567 HOSTIPV6 is replaced by the local host's IPv6 address
568 HOSTNAME is replaced by the local host's name
569 PORT_D is replaced by a port number for normal daemon use
570 PORT_N is replaced by a port number that should never respond
571 PORT_S is replaced by a port number for normal bin/server use
572 PORT_DYNAMIC is replaced by a port number allocated dynamically
573 TESTNUM is replaced by the current test number
574 V4NET is replaced by an IPv4 network number for testing
575 V6NET is replaced by an IPv6 network number for testing
577 PORT_D is currently hard-wired to 1225, PORT_N to 1223, and PORT_S to 1224.
578 V4NET is hardwired to 224 and V6NET to ff00. These networks are used for DNS
579 testing purposes, and for testing Exim with -bh. The only requirement is that
580 they are networks that can never be used for an IP address of a real host. I've
581 chosen two multicast networks for the moment.
583 PORT_DYNAMIC is allocated by hunting for a free port (starting at port
584 1024) a listener can bind to. This is done by runtest, for simulating
587 If the host has no IPv6 address, "<no IPv6 address found>" is substituted but
588 that does not matter because no IPv6 tests will be run. A similar substitution
589 is made if there is no IPv4 address, and again, tests that actually require a
590 running IPv4 interface should be skipped.
592 If the host has more than one IPv4 or IPv6 address, the first one that
593 "ifconfig" lists is used. If the only available address is 127.0.0.1 (or ::1
594 for IPv6) it is used, but another value is preferred if available.
596 In situations where a specific test is not being run (for example, when setting
597 up dynamic data files), TESTNUM is replaced by an empty string, but should not
598 in fact occur in such files.
604 Each numbered script runs Exim (sometimes several times) with its own Exim
605 configuration file. The configurations are stored in the "confs" directory,
606 and before running each test, a copy of the appropriate configuration, with the
607 standard substitutions, is made in the file test-config. The -C command line
608 option is used to tell Exim to use this configuration.
610 The -D option is used to pass the path of the Exim binary to the configuration.
611 This is not standardly substituted, because there are two possible binaries
612 that might be used in the same test (one setuid to root, the other to the exim
613 user). Some tests also make use of -D to vary the configuration for different
614 calls to the Exim binary.
616 Normally, of course, Exim gives up root privilege when -C and -D are used by
617 unprivileged users. We do not want this to happen when running the tests,
618 because we want to be able to test all aspects of Exim, including receiving
619 mail from unprivileged users. The way this is handled is as follows:
621 At the start of the runtest script, the patchexim script is run as root. This
622 script makes a copy of the Exim binary that is to be tested, patching it as it
623 does so. (This is a binary patch, not a source patch.) The patch causes the
624 binary, when run, to "know" that it is running in the test harness. It does not
625 give up root privilege when -C and -D are used, and in a few places it takes
626 other special actions, such as delaying when starting a subprocess to allow
627 debug output from the parent to be written first. If you want to know more,
628 grep the Exim source files for "running_in_test_harness".
630 The patched binary is placed in the directory eximdir/exim and given the normal
631 setuid root privilege. This is, of course, a dangerous binary to have lying
632 around, especially if there are unprivileged users on the system. To protect
633 it, the eximdir directory is created with the current user as owner, exim as
634 the group owner, and with access drwx--x---. Thus, only the user who is running
635 the tests (who is known to have access to root) and the exim user have access
636 to the modified Exim binary. When runtest terminates, the patched binary is
639 Each set of tests proceeds by interpreting its controlling script. The scripts
640 are in subdirectories of the "scripts" directory. They are split up according
641 to the requirements of the tests they contain, with the 0000-Basic directory
642 containing tests that can always be run. Run the "listtests" script to obtain a
649 Output from script runs is written to the files test-stdout and test-stderr.
650 When an Exim server is involved, test-stdout-server and test-stderr-server are
651 used for its output. Before being compared with the saved output, the
652 non-server and server files are concatenated, so a single saved file contains
655 A directory called spool is used for Exim's spool files, and for Exim logs.
656 These locations are specified in every test's configuration file.
658 When messages are delivered to files, the files are put in the test-mail
659 directory. Output from comparisons is written to test-cf.
661 Before comparisons are done, output texts are modified ("munged") to change or
662 remove parts that are expected to vary from run to run. The modified files all
663 end with the suffix "-munged". Thus, you will see test-stdout-munged,
664 test-mainlog-munged, test-mail-munged, and so on. Other files whose names start
665 with "test-" are created and used by some of the tests.
667 At the end of a successful test run, the spool directory and all the files
668 whose names begin with "test-" are removed. If the run ends unsuccessfully
669 (typically after a "Q" response to a prompt), the spool and test files are left
670 in existence so that the problem can be investigated.
676 Each test script consists of a list of commands, each optionally preceded by
677 comments (lines starting with #) and (also optionally) a line containing an
678 expected return code. Some of the commands are followed by data lines
679 terminated by a line of four asterisks.
681 The first line of each script must be a comment that briefly describes the
684 # -bS Use of HELO/RSET
686 A line consisting just of digits is interpreted as the expected return code
687 for the command that follows.
689 A line consisting of a tilde (~) followed by digits indicates a non-expected
690 return code for the command that follows.
692 The default expectation when neither such line exists
693 is a zero return code. For example, here is a complete test script, containing
696 # -bS Unexpected EOF in headers
699 mail from:<someone@some.where>
700 rcpt to:<blackhole@HOSTNAME>
705 The expected return code in this case is 1, and the data lines are passed to
706 Exim on its standard input. Both the command line and the data lines have the
707 standard substitutions applied to them. Thus, HOSTNAME in the example above will
708 be replaced by the local host's name. Long commands can be continued over
709 several lines by using \ as a continuation character. This does *not* apply to
712 A line with a leading number followed by a space and then an uppercase
713 word, equals character, value sets an expected return code as above
714 plus an environment variable. Example:
717 exim_msgdate -l -u -z -localhost_number=20 000000 1PANS3 ZZZZZZ
723 Here follows a list of supported commands. They can be divided into two groups:
726 Commands with no input
727 ----------------------
729 These commands are not followed by any input data, or by a line of asterisks.
732 ### This is a verbose comment
734 A line starting with three hashmarks and some space copies the following text to
735 both stdout and stderr file being written by the test.
737 dbmbuild <file1> <file1>
739 This command runs the exim_dbmbuild utility to build a DBM file. It is used
740 only when DBM support is available in Exim, and typically follows the use of a
741 "write" command (see below) that creates the input file.
746 This command runs the exim_dumpdb utility on the testing spool directory, using
747 the database name given, for example: "dumpdb retry".
752 The text is written to the screen; this is used to output comments from
756 exim_lock [options] <file name>
758 This command runs the exim_lock utility with the given options and file name.
759 The file remains locked for following commands until a non-daemon "exim"
765 This command runs the exinext utility with the given argument data.
770 This command runs the exigrep utility with the given data (the search pattern)
771 on the current mainlog file.
776 This command runs the exiqgrep utility with the given options
777 on the current spool directory.
782 This command is present at the start of all but one of the tests that use
783 GnuTLS. It copies a pre-existing parameter file into the spool directory, so
784 that Exim does not have to re-create the file each time. The first GnuTLS test
785 does not do this, in order to test that Exim can create the file.
790 This command must be given in any script that starts an Exim daemon, normally
791 at the end. It searches for the PID file in the spool directory, and sends a
792 SIGINT signal to the Exim daemon process whose PID it finds. See below for
793 comments about starting Exim daemons.
798 This command causes the script to sleep for m milliseconds. Nothing is output
804 This command requests custom munging of the test outputs. The munge names
805 used are coded in the runtest script (look for 'name of munge').
810 This command must be at the head of a script. If no IPv4 interface has been
811 found, the entire script is skipped, and a comment is output.
816 This command must be at the head of a script. If no IPv6 interface has been
817 found, the entire script is skipped, and a comment is output.
822 This command must be at the head of a script. If the Exim binary does not
823 support large files (off_t is <= 4), the entire script is skipped, and a
827 need_move_frozen_messages
829 This command must be at the head of a script. If the Exim binary does not have
830 support for moving frozen messages (which is an optional feature), the entire
831 script is skipped, and a comment is output.
836 If this command is encountered anywhere in the script, messages that are
837 delivered when the script runs are not compared with saved versions.
842 If this command is encountered anywhere in the script, message log files that
843 are still in existence at the end of the run (for messages that were not
844 delivered) are not compared with saved versions.
849 If this command is encountered anywhere in the script, the output is not
850 munged before it is compared with a saved version.
851 This option allows meaningful tests of the exim_msgdate utility;
852 without it all date comparison checks would succeed.
857 If this command is encountered anywhere in the script, the stderr output from
858 the run is not compared with a saved version.
863 If this command is encountered anywhere in the script, the stdout output from
864 the run is not compared with a saved version.
869 This command indicates that the script is for a certain type of filter test, in
870 which there are a lot of repetitive stdout lines that get in the way, because
871 filter tests output data about the sender and recipient. Such lines are removed
872 from the stdout output before comparing, for ease of human perusal.
877 This command causes the script to sleep for n seconds. If n is greater than
878 one, "sleep <n>" is output to the screen, followed by a dot for every second
884 This command causes special sorting to occur on the mainlog file before
885 comparison. Every sequence of contiguous delivery lines (lines containing the
886 => -> or *> flags) is sorted. This is necessary in some tests that use parallel
887 deliveries because on different systems the processes may terminate in a
891 A number of standard file management commands are also recognized. These are
892 cat, chmod, chown, cp, du, ln, ls, du, mkdir, mkfifo, rm, rmdir, and touch.
893 Some are run as root using "sudo".
899 The remaining commands are followed by data lines for their standard input,
900 terminated by four asterisks ("****"). Even if no data is required for the particular
901 usage, the asterisks must be given.
903 If the input line starts with ':<cmd>:', this prefix is removed and the
904 line is processed by the runtest script before sending. The following
905 commands are recognised:
907 - "eval": process the reset of the line with Perl's string eval()
908 function. This can be used to send arbitrary data by encoding it as
909 escape sequences (e.g. "\x41\101"). If you need a line ending, you have
910 to append it accordingly (e.g. "\r\n").
912 - "noeol": do not terminate the data sent to the application with an end
915 - "sleep": interpret the rest of the line as an integer and sleep for
916 that number of seconds before proceeding. No data will be output to
922 This command takes one script line and runs it in the background,
923 in parallel with following commands. For external daemons, eg. redis-server.
926 catwrite <file name> [nxm[=start-of-line-text]]*
928 This command operates like the "write" command, which is described below,
929 except that the data it generates is copied to the end of the test-stdout file
930 as well as to the named file.
933 client [<options>] <ip address> <port> [<outgoing interface>]
935 This command runs the auxiliary "client" program that simulates an SMTP client.
936 It is controlled by a script read from its standard input, details of which are
937 given below. There are two options. One is -t, which must be followed directly
938 by a number, to specify the command timeout in seconds (e.g. -t5). The default
939 timeout is 5 seconds. The other option is -tls-on-connect, which causes the
940 client to try to start up a TLS session as soon as it has connected, without
941 using the STARTTLS command. The client program connects to the given IP address
942 and port, using the specified interface, if one is given.
945 client-ssl [<options>] <ip address> <port> [<outgoing interface>] \
946 [<cert file>] [<key file>]
948 When OpenSSL is available on the host, an alternative version of the client
949 program is compiled, one that supports TLS using OpenSSL. The additional
950 arguments specify a certificate and key file when required for the connection.
951 There are two additional options: -tls-on-connect, that causes the client to
952 initiate TLS negotiation immediately on connection; -ocsp that causes the TLS
953 negotiation to include a certificate-status request. The latter takes a
954 filename argument, the CA info for verifying the stapled response.
957 client-gnutls [<options>] <ip address> <port> [<outgoing interface>] \
958 [<cert file>] [<key file>]
960 When GnuTLS is available on the host, an alternative version of the client
961 program is compiled, one that supports TLS using GnuTLS. The additional
962 arguments specify a certificate and key file when required. There is one
963 additional option, -tls-on-connect, that causes the client to initiate TLS
964 negotiation immediately on connection.
967 exim [<options>] [<arguments>]
969 This command runs the testing version of Exim. Any occurrence of "$msg1" in the
970 command line is replaced by the ID of the first (oldest) message in Exim's
971 (testing) spool. "$msg2" refers to the second, and so on. The name "exim" can
972 be preceded by an environment setting as in this example:
974 LDAPTLS_REQCERT=never exim -be
976 It can also be preceded by a number; this specifies a number of seconds to wait
977 before closing the stdout pipe to Exim, and is used for some timeout tests. For
982 Finally, "exim" can be preceded by "sudo", to run Exim as root. If more than
983 one of these prefixes is present, they must be in the above order.
985 If the options include "-DSERVER" but not "-DNOTDAEMON", the script waits for
986 Exim to start but then continues without waiting for it to terminate. Typically
987 this will be for a daemon-mode "-bd" operation. The daemon should be later
988 terminated using "killdaemon".
991 exim_exim [<options>] [<arguments>]
993 This runs an alternative version of Exim that is setuid to exim rather than to
997 server [<options>] <port or socket> [<connection count>]
999 This command runs the auxiliary "server" program that simulates an SMTP (or
1000 other) server. It is controlled by a script that is read from its standard
1001 input, details of which are given below. A number of options are implemented:
1003 -d causes the server to output debugging information
1005 -t <sec> sets a timeout (default 5) for when the server is
1006 awaiting an incoming connection. If negative, the
1007 absolute value is used and a timeout results in a
1008 nonfailure exit code
1010 -noipv4 causes the server not to set up an IPv4 socket
1012 -noipv6 causes the server not to set up an IPv6 socket
1014 -i <sec> sets an initial pause, to delay before creating the listen sockets
1016 By default, in an IPv6 environment, both kinds of socket are set up. However,
1017 the test script knows which interfaces actually exist on the host, and it adds
1018 -noipv4 or -noipv6 to the server command as required. An error occurs if both
1019 these options are given.
1021 The only required argument is either a port number or the path name of a Unix
1022 domain socket. The port is normally PORT_S, which is changed to an actual
1023 number by the standard substitutions. The optional final argument specifies the
1024 number of different connections to expect (default 1). These must happen
1025 serially (one at a time). There is no support for multiple simultaneous
1026 connections. Here are some example commands:
1029 server -t 10 PORT_S 3
1030 server /tmp/somesocket
1032 The following lines, up to a line of four asterisks, are the server's
1033 controlling standard input (described below). These lines are read and
1034 remembered; during the following commands, until a non-deamon "exim" command
1035 is reached, the server is run in parallel. Then the server termination
1039 write <file name> [nxm[=start-of-line-text]]*
1041 The "write" command is a way of creating files of specific sizes for buffering
1042 tests, or containing specific data lines. Being able to do this from within the
1043 script saves holding lots of little test files. The optional argument specifies
1044 n lines of length m. The lines consist of the letter "a". If start of line text
1045 is supplied, it replaces "a"s at the start of each line. Underscores in the
1046 start of line text are turned into spaces. The optional argument may be
1047 repeated. The data lines that follow a "write" command are split into two by a
1048 line of four plus signs. Any above the split are written before the
1049 fixed-length lines, and any below the split are written after. For example:
1051 write test-data 3x30=AB_ 1x50
1059 This command generates a file containing:
1063 AB aaaaaaaaaaaaaaaaaaaaaaaaaaa
1064 AB aaaaaaaaaaaaaaaaaaaaaaaaaaa
1065 AB aaaaaaaaaaaaaaaaaaaaaaaaaaa
1066 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
1070 If there are no fixed-length line specifiers, there is no need to split the
1071 data, and a line of plusses is not needed.
1076 This command runs Perl, with the data as its standard input, to allow arbitrary
1077 one-off things to be done.
1083 Lines in client scripts are of several kinds:
1085 (1) "??? ": If a line begins with three question marks and a space, the rest of the
1086 line defines the start of expected output from the server. If what is
1087 received does not match, the client bombs out with an error message.
1089 (2) "???*": If a line begins with three question marks and an asterisk, the server
1090 is expected to close the connection.
1092 (3) "????": If a line begins with four question marks, the rest of the line defines
1093 the start of one or more possible output lines from the server. When it
1094 matches, the client silently repeats the comparison using the next server
1095 line. When the match fails, the client silently proceeds to the next script
1096 line with the then-current server output unconsumed.
1098 (4) "+++ ": If a line starts with three plus signs followed by a space, the rest of the
1099 line specifies a number of seconds to sleep for before proceeding.
1101 (5) ">>> ": If a line begins with three '>' characters and a space, the rest of the
1102 line is input to be sent to the server. Backslash escaping is done as
1103 described below, but no trailing "\r\n" is sent.
1105 (6) "<<< ": If a line begin with three '<' characters and a space, the rest of the
1106 line is a filename; the content of the file is inserted into the script
1109 (7) Otherwise, the line is an input line line that is sent to the server. Any
1110 occurrences of \r and \n in the line are turned into carriage return and
1111 linefeed, respectively. This is used for testing PIPELINING.
1112 Any sequences of \x followed by two hex digits are converted to the equivalent
1113 byte value. Any other character following a \ is sent verbatim.
1114 The line is sent with a trailing "\r\n".
1116 Here is a simple example:
1118 client 127.0.0.1 PORT_D
1123 AUTH PLAIN AbdXi0AdnD2CVy
1129 In the case of client-gnutls and client-ssl, if a command is "starttls", this
1130 is remembered, and after a subsequent OK response, an attempt to move into TLS
1131 mode occurs. If a command is "starttls_wait", the client sends "starttls" but
1132 does not start up TLS; this is for testing timeouts. If a command is "stoptls",
1133 an existing TLS connection is shut down, but nothing is sent.
1139 The server program sleeps till a connection occurs or its timeout is reached,
1140 in which case it bombs out. The next set of command lines are interpreted. They
1141 are of the following kinds:
1143 (1) A line that starts with '>' or with a digit is an output line that is sent
1144 to the client. In the case of '>':
1146 (a) If the line starts with ">>", no terminating CRLF is sent.
1147 (b) If the line starts with ">CR>", just CR is sent at the end.
1148 (c) If the line starts with ">LF>", just LF is sent at the end.
1149 (d) If the line starts with ">*eof", nothing is sent and the connection
1152 The data that is sent starts after the initial '>' sequence. Within
1153 each line the sequence '\x' followed by two hex digits can be used
1154 to specify an arbitrary byte value. The sequence '\\' specifies a
1157 (2) A line that starts with "*sleep" specifies a number of seconds to wait
1160 (3) A line containing "*data" and a number specifies that the client is
1161 expected to send that many bytes; the server discards them
1163 (4) A line containing "*eof" specifies that the client is expected to close
1164 the connection at this point.
1166 (5) A line containing just '.' specifies that the client is expected to send
1167 many lines, terminated by one that contains just a dot.
1169 (6) Otherwise, the line defines the start of an input line that the client
1170 is expected to send. To allow for lines that start with digits, the line
1171 may start with '<', which is not taken as part of the input data. If the
1172 lines starts with '<<' then only the characters are expected; no return-
1173 linefeed terminator. If the input does not match, the server bombs out
1174 with an error message. Backslash-escape sequences may be used in the
1175 line content as for output lines.
1177 Here is a simple example of server use in a test script:
1195 After a "server" command in a test script, the server runs in parallel until an
1196 "exim" command is reached. The "exim" command attempts to deliver one or more
1197 messages to port PORT_S on the local host. When it has finished, the test
1198 script waits for the "server" process to finish.
1200 The "mtpscript" program is like "server", except that it uses stdin/stdout for
1201 its input and output instead of a script. However, it is not called from test
1202 scripts; instead it is used as the command for pipe transports in some
1203 configurations, to simulate non-socket LMTP servers.
1206 AUXILIARY DATA FILES
1207 --------------------
1209 Many of the tests make use of auxiliary data files. There are two types; those
1210 whose content is fixed, and those whose content needs to be varied according to
1211 the current environment. The former are kept in the directory aux-fixed. The
1212 latter are distributed in the directory aux-var-src, and copied with the
1213 standard substitutions into the directory aux-var at the start of each test
1214 run (with TESTNUM from the numeric prefix of the filename is there is one).
1216 Most of the auxiliary files have names that start with a test number,
1217 indicating that they are specific to that one test. A few fixed files (for
1218 example, some TLS certificates) are used by more than one test, and so their
1219 names are not of this form.
1221 There are also some auxiliary DNS zone files, which are described in the next
1225 DNS LOOKUPS AND GETHOSTBYNAME
1226 -----------------------------
1228 The original test suite required special testing zones to be loaded into a
1229 local nameserver. This is no longer a requirement for the new suite. Instead, a
1230 program called fakens is used to simulate a nameserver. When Exim is running in
1231 the test harness, instead of calling res_search() - the normal call to the DNS
1232 resolver - it calls a testing function. This handles a few special names itself
1233 (for compatibility with the old test suite), but otherwise passes the query to
1236 The fakens program consults "zone files" in the directory called dnszones, and
1237 returns data in the standard resource record format for Exim to process as if
1238 it came from the DNS. However, if the requested domain is not in any of the
1239 zones that fakens knows about, it returns a special code that causes Exim to
1240 pass the query on to res_search(). The zone files are:
1242 db.test.ex A zone for the domain test.ex.
1243 db.ip4.10 A zone for one special case in 10.250.0.0/16 (see below)
1244 db.ip4.V4NET A zone for the domain V4NET.in-addr.arpa.
1245 db.ip4.127 A zone for the domain 127.in-addr.arpa.
1246 db.ip6.V6NET A zone for the domain inverted(V6NET).ip6.arpa.
1247 db.ip6.0 A zone for the domain 0.ip6.arpa.
1249 V4NET and V6NET are substituted with the current testing networks (see above).
1250 In the case of V6NET, the network is four hex digits, and it is split and
1251 inverted appropriately when setting up the zone.
1253 These fake zone files are built dynamically from sources in the dnszones-src
1254 directory by applying the standard substitutions. The test suite also builds
1255 dynamic zone files for the name of the current host and its IP address(es). The
1256 idea is that there should not be any need to rely on an external DNS.
1258 The fakens program handles some names programmatically rather than using the
1259 fake zone files. These are:
1261 manyhome.test.ex This name is used for testing hosts with ridiculously large
1262 numbers of IP addresses; 2048 IP addresses are generated
1263 and returned. Doing it this way saves having to make the
1264 interface to fakens handle more records that can fit in the
1265 data block. The addresses that are generated are in the
1266 10.250.0.0/16 network.
1268 test.again.dns This always provokes a TRY_AGAIN response, for testing the
1269 handling of temporary DNS error. If the full domain name
1270 starts with digits, a delay of that many seconds occurs.
1272 test.fail.dns This always provokes a NO_RECOVERY response, for testing
1273 DNS server failures.
1275 The use of gethostbyname() and its IPv6 friends is also subverted when Exim is
1276 running in the test harness. The test code handles a few special names
1277 directly; for all the others it uses DNS lookups, which are then handled as
1278 just described. Thus, the use of /etc/hosts is completely bypassed. The names
1279 that are specially handled are:
1281 localhost Always returns 127.0.0.1 or ::1, for IPv4 and IPv6 lookups,
1284 <an IP address> If the IP address is of the correct form for the lookup
1285 type (IPv4 or IPv6), it is returned. Otherwise a panic-die
1288 The reverse zone db.ip4.10 is provided just for the manyhome.test.ex case. It
1289 contains a single wildcard resource record. It also contains the line
1293 Whenever fakens finds this line in a zone file, it returns PASS_ON instead of
1294 HOST_NOT_FOUND. This causes Exim to pass the query to res_search().