EXPORTABLE EXIM TEST SUITE -------------------------- This document last updated for: Test Suite Version: 4.87 Date: 30 January 2016 BACKGROUND ---------- For a long time, the Exim test suite was confined to Philip Hazel's workstation, because it relied on that particular environment. The problem is that an MTA such as Exim interacts a great deal with its environment, so if you run it somewhere else, the output will be different, which makes automatic checking difficult. Even in a single environment, things are not all that easy. For instance, if Exim delivers a message, the log line (which one would want to compare) contains a timestamp and an Exim message id that will be different each time. This issue is dealt with by a Perl script that munges the output by recognizing changing sequences and replacing them with fixed values before doing a comparison. Another problem with exporting the original test suite is that it assumes a version of Exim with more or less every optional feature enabled. This README describes a new test suite that is intended to be exportable and to run in a number of different environments. The tests themselves are in no particular order; they accumulated over the years as Exim was extended and modified. They vary greatly in size and complexity. Some were specifically constructed to test new features; others were made to demonstrate that a bug had been fixed. A few of the original tests have had to be omitted from this more general suite because differences in operating system behaviour make it impossible to generalize them. An example is a test that uses a version of Exim that is setuid to the Exim user rather than root, with the deliver_drop_privilege option set. In Linux, such a binary is able to deliver a message as the caller of Exim, because it can revert to the caller's uid. In FreeBSD this is not the case. REQUIREMENTS ------------ In order to run this test suite, the following requirements must be met: (1) You should run the tests on a matching version of Exim, because the suite is continuously updated to test the latest features and bug fixes. The version you test does not, however, have to be installed as the live version. You can of course try the tests on any version of Exim, but some may fail. In particular, the test suite will fall apart horrible with versions of Exim prior to 4.54. (2) You can use any non-root login to run the tests, but there must be access via "sudo" to root from this login. Privilege is required to override configuration change checks and for things like cleaning up spool files, but on the other hand, the tests themselves need to call Exim from a non-root process. The use of "sudo" is the easiest way to achieve all this. The test script uses "sudo" to do a number of things as root, so it is best if you set a sudo timeout so that you do not have to keep typing a password. For example, if you put Defaults timestamp_timeout=480 in /etc/sudoers, a password lasts for 8 hours (a working day). It is not permitted to run the tests as the Exim user because the test suite tracks the two users independently. Using the same user would result in false positives on some tests. Further, some tests invoke sudo in an environment where there might not be a TTY, so tickets should be global, not per-TTY. Taking this all together and assuming a user of "exim-build", you might have this in sudoers: Defaults:exim-build timestamp_timeout=480,!tty_tickets (3) The login under which you run the tests must have the exim group as a secondary so that it has access to logs, spool files, etc. However, it should have a different primary group (eg. "users" vs. "eximgroup"). The login should not be one of the names "userx", "usery", "userz", or a few other simple ones such as "abcd" and "xyz" and single letters that are used in the tests. The test suite expects the login to have a gecos name; I think it will now run if the gecos field is empty but there may be anomalies. The login must not contain a dash or an equal sign. (Otherwise some tests about local_from_{suffix,prefix} will fail.) (4) The directory into which you unpack the test suite must be accessible by the Exim user, so that code running as exim can access the files therein. This includes search-access on all path elements leading to it. A world-readable directory is fine. However, there may be problems if the path name of the directory is excessively long. This is because it sometimes appears in log lines or debug output, and if it is truncated, it is no longer recognized. (5) Exim must be built with its user and group specified at build time, and with certain minimum facilities, namely: Routers: accept, dnslookup, manualroute, redirect Transports: appendfile, autoreply, pipe, smtp Lookups: lsearch Authenticators: plaintext Most Exim binaries will have these included. (6) A C compiler is needed to build some test programs, and the test script is written in Perl, so you need that. (7) Some of the tests run Exim as a daemon, and others use a testing server (described below). These require TCP ports. In the configurations and scripts, the ports are parameterized, but at present, fixed values are written into the controlling script. These are ports 1224 to 1229. If these ports are not available for use, some of the tests will fail. (8) There is an underlying assumption that the host on which the tests are being run has an IPv4 address (which the test script seeks out). If there is also an IPv6 address, additional tests are run when the Exim binary contains IPv6 support. There are checks in the scripts for a running IPv4 interface; when one is not found, some tests are skipped (with a warning message). The local net may not be in 10.0/8 as that is used by the suite. (9) Exim must be built with TRUSTED_CONFIG_LIST support, so that the test configs can be placed into it. A suitable file location is .../exim/test/trusted_configs with content .../exim/test/test-config [fill out the ... to make full paths]. This file should be owner/group matching CONFIGURE_OWNER/GROUP, or root/root, and it has to be accessible for the login, under which you run the tests. The config files in .../exim/test/confs/ should be owner/group the same. DISABLE_D_OPTION must not be used. If ALT_CONFIG_PREFIX is used, it must contain the directory of the test-suite. WHITELIST_D_MACROS should contain: DIR:EXIM_PATH:AA:ACL:ACLRCPT:ACL_MAIL:ACL_PREDATA:ACL_RCPT:AFFIX:ALLOW:ARG1:ARG2:AUTHF:AUTHS:AUTH_ID_DOMAIN:BAD:BANNER:BB:BR:BRB:CERT:COM:COMMAND_USER:CONNECTCOND:CONTROL:CREQCIP:CREQMAC:CRL:CSS:D6:DATA:DCF:DDF:DEFAULTDWC:DELAY:DETAILS:DRATELIMIT:DYNAMIC_OPTION:ELI:ERROR_DETAILS:ERT:FAKE:FALLBACK:FILTER:FILTER_PREPEND_HOME:FORBID:FORBID_SMTP_CODE:FUSER:HAI:HAP:HARDLIMIT:HEADER_LINE_MAXSIZE:HEADER_MAXSIZE:HELO_MSG:HL:HOSTS:HOSTS_AVOID_TLS:HOSTS_MAX_TRY:HVH:IFACE:IGNORE_QUOTA:INC:INSERT:IP1:IP2:LAST:LDAPSERVERS:LENCHECK:LIMIT:LIST:LOG_SELECTOR:MAXNM:MESSAGE_LOGS:MSIZE:NOTDAEMON:ONCE:ONLY:OPT:OPTION:ORDER:PAH:PEX:PORT:PTBC:QDG:QOLL:QUOTA:QUOTA_FILECOUNT:QWM:RCPT_MSG:REMEMBER:REQUIRE:RETRY:RETRY1:RETRY2:RETURN:RETURN_ERROR_DETAILS:REWRITE:ROUTE_DATA:RRATELIMIT:SELECTOR:SELF:SERVER:SERVERS:SREQCIP:SREQMAC:SRV:STRICT:SUB:SUBMISSION_OPTIONS:TIMEOUTDEFER:TIMES:TRUSTED:TRYCLEAR:UL:USE_SENDER:UTF8:VALUE:WMF (10) Exim must *not* be built with USE_READLINE, as the test-suite's automation assumes the simpler I/O model. Exim must *not* be built with HEADERS_CHARSET set to UTF-8. OPTIONAL EXTRAS --------------- If the Exim binary that is being tested contains extra functionality in addition to the minimum specified above, additional tests are run to exercise the extra functionality, except for a few special cases such as the databases (MySQL, PostgreSQL, LDAP) where special data is needed for the tests. RUNNING THE TEST SUITE ---------------------- (1) Clone the git tree for Exim. This include both the Exim source and the testsuite. (2) cd into the test/ subdirectory (where this README lives). (3) Run "./configure" and then "make". This builds a few auxiliary programs that are written in C. (4) echo $PWD/test-config >> your_TRUSTED_CONFIG_LIST_filename Typically that is .../exim/test/trusted_configs (5) Run "./runtest" (a Perl script) as described below. (6) If you want to see what tests are available, run "./listtests". BREAKING OUT OF THE TEST SCRIPT ------------------------------- If you abandon the test run by typing ^C, the interrupt may be passed to a program that the script is running, or it may be passed to the script itself. In the former case, the script should detect that the program has ended abnormally. In both cases, the script tries to clean up everything, including killing any Exim daemons that it has started. However, there may be race conditions in which the clean up does not happen. If, after breaking out of a run, you see strange errors in the next run, look for any left-over Exim daemons, and kill them by hand. THE LISTTESTS SCRIPT -------------------- The individual test scripts are in subdirectories of the "scripts" directory. If you do not supply any arguments to ./listtests, it scans all the scripts in all the directories, and outputs the heading line from each script. The output is piped through "less", and begins like this: === 0000-Basic === Basic/0001 Basic configuration setting Basic/0002 Common string expansions Basic/0003 Caseless address blocking ... Lines that start === give the name of the subdirectory containing the test scripts that follow. If you supply an argument to ./listtests, it is used as a Perl pattern to match case-independently against the names of the subdirectories. Only those that match are scanned. For example, "./listtests ipv6" outputs this: === 1000-Basic-ipv6 === === Requires: support IPv6 Basic-ipv6/1000 -bh and non-canonical IPv6 addresses Basic-ipv6/1001 recognizing IPv6 address in HELO/EHLO === 2250-dnsdb-ipv6 === === Requires: support IPv6 lookup dnsdb dnsdb-ipv6/2250 dnsdb ipv6 lookup in string expansions If you supply a second argument to ./listtests, it is used as a Perl pattern to match case-independently against the individual script titles. For example, "./listtests . mx" lists all tests whose titles contain "mx", because "." matches all the subdirectory names. THE RUNTEST SCRIPT ------------------ If you do not supply any arguments to ./runtest, it searches for an Exim source tree at the same level as the test suite directory. A source tree is a source tree, if it contains a build-* directory. It then looks for an Exim binary in a "build" directory of that source tree. If there are several Exim source trees, it chooses the latest version of Exim. Consider the following example: $ ls -F /source/exim exim-4.60/ exim-4.62/ exim-testsuite-x.xx/ A simple ./runtest from within the test suite will use a 4.62 binary if it finds one, otherwise a 4.60 binary. If a binary cannot be found, the script prompts for one. Alternatively, you can supply the binary on the command line: ./runtest /usr/exim/bin/exim A matching test suite is released with each Exim release; if you use a test suite that does not match the binary, some tests may fail. The test suite uses some of the Exim utilities (such as exim_dbmbuild), and it expects to find them in the same directory as Exim itself. If they are not found, the tests that use them are omitted. A suitable comment is output. On the ./runtest command line, following the name of the binary, if present, there may be a number of options and then one or two numbers. The full syntax is as follows: ./runtest [binary name] [runtest options] [exim options] \ [first test] [last test] There are some options for the ./runtest script itself: -CONTINUE This will allow the script to move past some failing tests. It will write a simple failure line with the test number in a temporary logfile test/failed-summary.log. Unexpected exit codes will still stall the test execution and require interaction. -DEBUG This option is for debugging the test script. It causes some tracing information to be output. -DIFF By default, file comparisons are done using a private compare command called "cf", which is built from source that is provided in the src directory. This is a command I've had for nearly 20 years - look at the source comments for its history - whose output I prefer. However, if you want to use "diff" instead, give -DIFF as a runtest option. In that case, "diff -u" is used for comparisons. (If it turns out that most people prefer to use diff, I'll change the default.) -FLAVOR -FLAVOUR This allows "overrides" for the test results. It's intended use is to deal with distro specific differences in the test output. The default flavour is "FOO" if autodetection fails. (Autodetection is possible for known flavours only. Known flavours are computed after file name extensions in stdout/* and stderr/*.) If during the test run differences between the current and the expected output are found and no flavour file exists already, you may update the "common" expected output or you may create a flavour file. If a flavour file already exists, any updates will go into that flavour file! -KEEP Normally, after a successful run, the test output files are deleted. This option prevents this. It is useful when running a single test, in order to look at the actual output before it is modified for comparison with saved output. -NOIPV4 Pretend that an IPv4 interface was not found. This is useful for testing that the test suite correctly skips tests that require a running IPv4 interface. -NOIPV6 Pretend that an IPv6 interface was not found. This is useful for testing that the test suite correctly skips tests that require a running IPv6 interface. -UPDATE If this option is set, any detected changes in test output are automatically accepted and used to update the stored copies of the output. It is a dangerous option, but it useful for the test suite maintainer after making a change to the code that affects a lot of tests (for example, the wording of a message). -SLOW For very slow hosts that appear to have Heisenbugs, delay before comparing output files from a testcase The options for ./runtest must be given first (but after the name of the binary, if present). Any further options, that is, items on the command line that start with a hyphen, are passed to the Exim binary when it is run as part of a test. The only sensible use of this is to pass "-d" in order to run a test with debugging enabled. Any other options are likely to conflict with options that are set in the tests. Some tests are already set up to run with debugging. In these cases, -d on the command line overrides their own debug settings. The final two arguments specify the range of tests to be run. Test numbers lie in the range 1 to 9999. If no numbers are given, the defaults are 1 and 8999 (sic). Tests with higher numbers (9000 upwards) are not run automatically because they require specific data (such as a particular MySQL table) that is unlikely to be generally available. Tests that require certain optional features of Exim are grouped by number, so in any given range, not all the tests will exist. Non-existent tests are just skipped, but if there are no tests at all in the given range, a message is output. If you give only one number, just that test is run (if it exists). Instead of a second number, you can give the character "+", which is interpreted as "to the end". Normally this is 8999; if the starting number is 9000 or higher, "+" is interpreted as 9999. Examples: ./runtest 1300 ./runtest 1400 1699 ./runtest /usr/sbin/exim 5000 + ./runtest -DIFF -d 81 When the script starts up, the first thing it does is to check that you have sudo access to root. Then it outputs the version number of the Exim binary that it is testing, and also information about the optional facilities that are present (obtained from "exim -bV"). This is followed by some environmental information, including the current login id and the hosts's IP address. The script checks that the current user is in the Exim group, and that the Exim user has access to the test suite directory. The script outputs the list of tests requested, and a list of tests that will be omitted because the relevant optional facilities are not in the binary. You are then invited to press Return to start the tests running. TEST OUTPUT ----------- When all goes well, the only permanent output is the identity of the tests as they are run, and "Script completed" for each test script, for example: Basic/0001 Basic configuration setting Script completed Basic/0002 Basic string expansions Script completed Basic/0003 Caseless address blocking Script completed Basic/0004 Caseful address blocking Script completed Basic/0005 -bs to simple local delivery ... While a script is running, it shows "Test n" on the screen, for each of the Exim tests within the script. There may also be comments from some tests when a delay is expected, for example, if there is a "sleep" while testing a timeout. Before each set of optional tests, an extra identifying line is output. For example: >>> The following tests require: authenticator cram_md5 CRAM-MD5/2500 CRAM-MD5 server tests Script completed CRAM-MD5/2501 CRAM-MD5 client tests Script completed If a test fails, you are shown the output of the text comparison that failed, and prompted as to what to do next. The output is shown using the "less" command, or "more" if "less" is not available. The options for "less" are set to that it automatically exits if there is less that a screenful of output. By default, the output is from the "cf" program, and might look like this: DBM/1300 DBM files and exim_dbmbuild =============== Lines 7-9 of "test-stdout-munged" do not match lines 7-11 of "stdout/1300". ---------- exim_dbmbuild exit code = 1 Continued set of lines is too long: max permitted length is 99999 exim_dbmbuild exit code = 1 ---------- dbmbuild abandoned exim_dbmbuild exit code = 2 Continued set of lines is too long: max permitted length is 99999 dbmbuild abandoned exim_dbmbuild exit code = 2 =============== 1 difference found. "test-stdout-munged" contains 16 lines; "stdout/1300" contains 18 lines. Continue, Retry, Update & retry, Quit? [Q] This example was generated by running the test with a version of Exim that had a bug in the exim_dbmbuild utility (the bug was fixed at release 4.53). See "How the tests work" below for a description of the files that are used. In this case, the standard output differed from what was expected. The reply to the prompt must either be empty, in which case it takes the default that is given in brackets (in this case Q), or a single letter, in upper or lower case (in this case, one of C, R, U, or Q). If you type anything else, the prompt is repeated. "Continue" carries on as if the files had matched; that is, it ignores the mismatch. Any other output files for the same test will be compared before moving on to the next test. "Update & retry" copies the new file to the saved file, and reruns the test after doing any further comparisons that may be necessary. "Retry" does the same apart from the file copy. Other circumstances give rise to other prompts. If a test generates output for which there is no saved data, the prompt (after a message stating which file is unexpectedly not empty) is: Continue, Show, or Quit? [Q] "Show" displays the data on the screen, and then you get the "Continue..." prompt. If a test ends with an unexpected return code, the prompt is: show stdErr, show stdOut, Continue (without file comparison), or Quit? [Q] Typically in these cases there will be something interesting in the stderr or stdout output. There is a similar prompt after the "server" auxiliary program fails. OPENSSL AND GNUTLS ERROR MESSAGES --------------------------------- Some of the TLS tests deliberately cause errors to check how Exim handles them. It has been observed that different releases of the OpenSSL and GnuTLS libraries generate different error messages. This may cause the comparison with the saved output to fail. Such errors can be ignored. OTHER ISSUES ------------ . Some of the tests are time-sensitive (e.g. when testing timeouts, as in test 461). These may fail if run on a host that is also running a lot of other processes. . Some versions of "ls" use a different format for times and dates. This can cause test 345 to fail. . Test 0142 tests open file descriptors; on some hosts the output may vary. . Some tests may fail, for example 0022, because it says it uses cached data when the expected output thinks it should not be in cache. Item #5 in the Requirements section has: "Exim must be built with its user and group specified at build time" This means that you cannot use the "ref:username" in your Local/Makefile when building the exim binary, in any of the following fields: EXIM_USER EXIM_GROUP CONFIGURE_OWNER CONFIGURE_GROUP . If the runtest script warns that the hostname is not a Fully Qualified Domain Name (FQDN), expect that some tests will fail, for example 0036, with an extra log line saying the hostname doesn't resolve. You must use a FQDN for the hostname for proper test functionality. . If you change your hostname to a FQDN, you must delete the test/dnszones subdirectory. When you next run the runtest script, it will rebuild the content to use the new hostname. . If your hostname has an uppercase characters in it, expect that some tests will fail, for example, 0036, because some log lines will have the hostname in all lowercase. The regex which extracts the hostname from the log lines will not match the lowercased version. . Some tests may fail, for example 0015, with a cryptic error message: Server return code 99 Due to security concerns, some specific files MUST have the group write bit off. For the purposes of the test suite, some test/aux-fixed/* files MUST have the group write bit off, so it's easier to just remove the group write bit for all of them. If your umask is set to 002, the group write bit will be on by default and you'll see this problem, so make sure your umask is 022 and re-checkout the test/ subdirectory. . Some tests will fail if the username and group name are different. It does not have to be the primary group, a secondary group is sufficient. OTHER SCRIPTS AND PROGRAMS -------------------------- There is a freestanding Perl script called "listtests" that scans the test scripts and outputs a list of all the tests, with a short descriptive comment for each one. Special requirements for groups of tests are also noted. The main runtest script makes use of a second Perl script and some compiled C programs. These are: patchexim A Perl script that makes a patched version of Exim (see the next section for details). bin/cf A text comparison program (see above). bin/checkaccess A program that is run as root; it changes uid/gid to the Exim user and group, and then checks that it can access files in the test suite's directory. bin/client A script-driven SMTP client simulation. bin/client-gnutls A script-driven SMTP client simulation with GnuTLS support. This is built only if GnuTLS support is detected on the host. bin/client-ssl A script-driven SMTP client simulation with OpenSSL support. This is built only if OpenSSL support is detected on the host. bin/fakens A fake "nameserver" for DNS tests (see below for details). bin/fd A program that outputs details of open file descriptors. bin/iefbr14 A program that does nothing, and returns 0. It's just like the "true" command, but it is in a known place. bin/loaded Some dynamically loaded functions for testing dlfunc support. bin/mtpscript A script-driven SMTP/LMTP server simulation, on std{in,out}. bin/server A script-driven SMTP server simulation, over a socket. bin/showids Output the current uid, gid, euid, egid. The runtest script also makes use of a number of ordinary commands such as "cp", "kill", "more", and "rm", via the system() call. In some cases these are run as root by means of sudo. STANDARD SUBSTITUTIONS ---------------------- In the following sections, there are several references to the "standard substitutions". These make changes to some of the stored files when they are used in a test. To save repetition, the substitutions themselves are documented here: CALLER is replaced by the login name of the user running the tests CALLERGROUP is replaced by the caller's group id CALLER_GID is replaced by the caller's group id CALLER_UID is replaced by the caller's user id DIR is replaced by the name of the test-suite directory EXIMGROUP is replaced by the name of the Exim group EXIMUSER is replaced by the name of the Exim user HOSTIPV4 is replaced by the local host's IPv4 address HOSTIPV6 is replaced by the local host's IPv6 address HOSTNAME is replaced by the local host's name PORT_D is replaced by a port number for normal daemon use PORT_N is replaced by a port number that should never respond PORT_S is replaced by a port number for normal bin/server use PORT_DYNAMIC is replaced by a port number allocated dynamically TESTNUM is replaced by the current test number V4NET is replaced by an IPv4 network number for testing V6NET is replaced by an IPv6 network number for testing PORT_D is currently hard-wired to 1225, PORT_N to 1223, and PORT_S to 1224. V4NET is hardwired to 224 and V6NET to ff00. These networks are used for DNS testing purposes, and for testing Exim with -bh. The only requirement is that they are networks that can never be used for an IP address of a real host. I've chosen two multicast networks for the moment. PORT_DYNAMIC is allocated by hunting for a free port (starting at port 1024) a listener can bind to. This is done by runtest, for simulating inetd operations. If the host has no IPv6 address, "" is substituted but that does not matter because no IPv6 tests will be run. A similar substitution is made if there is no IPv4 address, and again, tests that actually require a running IPv4 interface should be skipped. If the host has more than one IPv4 or IPv6 address, the first one that "ifconfig" lists is used. If the only available address is 127.0.0.1 (or ::1 for IPv6) it is used, but another value is preferred if available. In situations where a specific test is not being run (for example, when setting up dynamic data files), TESTNUM is replaced by an empty string, but should not in fact occur in such files. HOW THE TESTS WORK ------------------ Each numbered script runs Exim (sometimes several times) with its own Exim configuration file. The configurations are stored in the "confs" directory, and before running each test, a copy of the appropriate configuration, with the standard substitutions, is made in the file test-config. The -C command line option is used to tell Exim to use this configuration. The -D option is used to pass the path of the Exim binary to the configuration. This is not standardly substituted, because there are two possible binaries that might be used in the same test (one setuid to root, the other to the exim user). Some tests also make use of -D to vary the configuration for different calls to the Exim binary. Normally, of course, Exim gives up root privilege when -C and -D are used by unprivileged users. We do not want this to happen when running the tests, because we want to be able to test all aspects of Exim, including receiving mail from unprivileged users. The way this is handled is as follows: At the start of the runtest script, the patchexim script is run as root. This script makes a copy of the Exim binary that is to be tested, patching it as it does so. (This is a binary patch, not a source patch.) The patch causes the binary, when run, to "know" that it is running in the test harness. It does not give up root privilege when -C and -D are used, and in a few places it takes other special actions, such as delaying when starting a subprocess to allow debug output from the parent to be written first. If you want to know more, grep the Exim source files for "running_in_test_harness". The patched binary is placed in the directory eximdir/exim and given the normal setuid root privilege. This is, of course, a dangerous binary to have lying around, especially if there are unprivileged users on the system. To protect it, the eximdir directory is created with the current user as owner, exim as the group owner, and with access drwx--x---. Thus, only the user who is running the tests (who is known to have access to root) and the exim user have access to the modified Exim binary. When runtest terminates, the patched binary is removed. Each set of tests proceeds by interpreting its controlling script. The scripts are in subdirectories of the "scripts" directory. They are split up according to the requirements of the tests they contain, with the 0000-Basic directory containing tests that can always be run. Run the "listtests" script to obtain a list of tests. TEST OUTPUT ----------- Output from script runs is written to the files test-stdout and test-stderr. When an Exim server is involved, test-stdout-server and test-stderr-server are used for its output. Before being compared with the saved output, the non-server and server files are concatenated, so a single saved file contains both. A directory called spool is used for Exim's spool files, and for Exim logs. These locations are specified in every test's configuration file. When messages are delivered to files, the files are put in the test-mail directory. Output from comparisons is written to test-cf. Before comparisons are done, output texts are modified ("munged") to change or remove parts that are expected to vary from run to run. The modified files all end with the suffix "-munged". Thus, you will see test-stdout-munged, test-mainlog-munged, test-mail-munged, and so on. Other files whose names start with "test-" are created and used by some of the tests. At the end of a successful test run, the spool directory and all the files whose names begin with "test-" are removed. If the run ends unsuccessfully (typically after a "Q" response to a prompt), the spool and test files are left in existence so that the problem can be investigated. TEST COMMANDS ------------- Each test script consists of a list of commands, each optionally preceded by comments (lines starting with #) and (also optionally) a line containing an expected return code. Some of the commands are followed by data lines terminated by a line of four asterisks. The first line of each script must be a comment that briefly describes the script. For example: # -bS Use of HELO/RSET A line consisting just of digits is interpreted as the expected return code for the command that follows. The default expectation when no such line exists is a zero return code. For example, here is a complete test script, containing just one command: # -bS Unexpected EOF in headers 1 exim -bS -odi mail from: rcpt to: data from: me **** The expected return code in this case is 1, and the data lines are passed to Exim on its standard input. Both the command line and the data lines have the standard substitutions applied to them. Thus, HOSTNAME in the example above will be replaced by the local host's name. Long commands can be continued over several lines by using \ as a continuation character. This does *not* apply to data lines. Here follows a list of supported commands. They can be divided into two groups: Commands with no input ---------------------- These commands are not followed by any input data, or by a line of asterisks. dbmbuild This command runs the exim_dbmbuild utility to build a DBM file. It is used only when DBM support is available in Exim, and typically follows the use of a "write" command (see below) that creates the input file. dump This command runs the exim_dumpdb utility on the testing spool directory, using the database name given, for example: "dumpdb retry". echo The text is written to the screen; this is used to output comments from scripts. exim_lock [options] This command runs the exim_lock utility with the given options and file name. The file remains locked with the following command (normally exim) is obeyed. exinext This command runs the exinext utility with the given argument data. exigrep This command runs the exigrep utility with the given data (the search pattern) on the current mainlog file. gnutls This command is present at the start of all but one of the tests that use GnuTLS. It copies a pre-existing parameter file into the spool directory, so that Exim does not have to re-create the file each time. The first GnuTLS test does not do this, in order to test that Exim can create the file. killdaemon This command must be given in any script that starts an Exim daemon, normally at the end. It searches for the PID file in the spool directory, and sends a SIGINT signal to the Exim daemon process whose PID it finds. See below for comments about starting Exim daemons. millisleep This command causes the script to sleep for m milliseconds. Nothing is output to the screen. munge This command requests custom munging of the test outputs. The munge names used are coded in the runtest script (look for 'name of munge'). need_ipv4 This command must be at the head of a script. If no IPv4 interface has been found, the entire script is skipped, and a comment is output. need_ipv6 This command must be at the head of a script. If no IPv6 interface has been found, the entire script is skipped, and a comment is output. need_largefiles This command must be at the head of a script. If the Exim binary does not support large files (off_t is <= 4), the entire script is skipped, and a comment is output. need_move_frozen_messages This command must be at the head of a script. If the Exim binary does not have support for moving frozen messages (which is an optional feature), the entire script is skipped, and a comment is output. no_message_check If this command is encountered anywhere in the script, messages that are delivered when the script runs are not compared with saved versions. no_msglog_check If this command is encountered anywhere in the script, message log files that are still in existence at the end of the run (for messages that were not delivered) are not compared with saved versions. no_stderr_check If this command is encountered anywhere in the script, the stderr output from the run is not compared with a saved version. no_stdout_check If this command is encountered anywhere in the script, the stdout output from the run is not compared with a saved version. rmfiltertest This command indicates that the script is for a certain type of filter test, in which there are a lot of repetitive stdout lines that get in the way, because filter tests output data about the sender and recipient. Such lines are removed from the stdout output before comparing, for ease of human perusal. sleep This command causes the script to sleep for n seconds. If n is greater than one, "sleep " is output to the screen, followed by a dot for every second that passes. sortlog This command causes special sorting to occur on the mainlog file before comparison. Every sequence of contiguous delivery lines (lines containing the => -> or *> flags) is sorted. This is necessary in some tests that use parallel deliveries because on different systems the processes may terminate in a different order. A number of standard file management commands are also recognized. These are cat, chmod, chown, cp, du, ln, ls, du, mkdir, mkfifo, rm, rmdir, and touch. Some are run as root using "sudo". Commands with input ------------------- The remaining commands are followed by data lines for their standard input, terminated by four asterisks. Even if no data is required for the particular usage, the asterisks must be given. background This command takes one script line and runs it in the background, in parallel with following commands. For external daemons, eg. redis-server. catwrite [nxm[=start-of-line-text]]* This command operates like the "write" command, which is described below, except that the data it generates is copied to the end of the test-stdout file as well as to the named file. client [] [] This command runs the auxiliary "client" program that simulates an SMTP client. It is controlled by a script read from its standard input, details of which are given below. There are two options. One is -t, which must be followed directly by a number, to specify the command timeout in seconds (e.g. -t5). The default timeout is 5 seconds. The other option is -tls-on-connect, which causes the client to try to start up a TLS session as soon as it has connected, without using the STARTTLS command. The client program connects to the given IP address and port, using the specified interface, if one is given. client-ssl [] [] \ [] [] When OpenSSL is available on the host, an alternative version of the client program is compiled, one that supports TLS using OpenSSL. The additional arguments specify a certificate and key file when required for the connection. There are two additional options: -tls-on-connect, that causes the client to initiate TLS negotiation immediately on connection; -ocsp that causes the TLS negotiation to include a certificate-status request. The latter takes a filename argument, the CA info for verifying the stapled response. client-gnutls [] [] \ [] [] When GnuTLS is available on the host, an alternative version of the client program is compiled, one that supports TLS using GnuTLS. The additional arguments specify a certificate and key file when required. There is one additional option, -tls-on-connect, that causes the client to initiate TLS negotiation immediately on connection. exim [] [] This command runs the testing version of Exim. Any occurrence of "$msg1" in the command line is replaced by the ID of the first (oldest) message in Exim's (testing) spool. "$msg2" refers to the second, and so on. The name "exim" can be preceded by an environment setting as in this example: LDAPTLS_REQCERT=never exim -be It can also be preceded by a number; this specifies a number of seconds to wait before closing the stdout pipe to Exim, and is used for some timeout tests. For example: 3 exim -bs Finally, "exim" can be preceded by "sudo", to run Exim as root. If more than one of these prefixes is present, they must be in the above order. If the options include "-DSERVER" but not "-DNOTDAEMON", the script waits for Exim to start but then continues without waiting for it to terminate. Typically this will be for a daemon-mode "-bd" operation. The daemon should be later terminated using "killdaemon". exim_exim [] [] This runs an alternative version of Exim that is setuid to exim rather than to root. server [] [] This command runs the auxiliary "server" program that simulates an SMTP (or other) server. It is controlled by a script that is read from its standard input, details of which are given below. A number of options are implemented: -d causes the server to output debugging information -t sets a timeout (default 5) for when the server is awaiting an incoming connection. If negative, the absolute value is used and a timeout results in a nonfailure exit code -noipv4 causes the server not to set up an IPv4 socket -noipv6 causes the server not to set up an IPv6 socket -i sets an initial pause, to delay before creating the listen sockets By default, in an IPv6 environment, both kinds of socket are set up. However, the test script knows which interfaces actually exist on the host, and it adds -noipv4 or -noipv6 to the server command as required. An error occurs if both these options are given. The only required argument is either a port number or the path name of a Unix domain socket. The port is normally PORT_S, which is changed to an actual number by the standard substitutions. The optional final argument specifies the number of different connections to expect (default 1). These must happen serially (one at a time). There is no support for multiple simultaneous connections. Here are some example commands: server PORT_S server -t 10 PORT_S 3 server /tmp/somesocket The following lines, up to a line of four asterisks, are the server's controlling standard input (described below). These lines are read and remembered; during the following commands, until an "exim" command is reached, the server is run in parallel. write [nxm[=start-of-line-text]]* The "write" command is a way of creating files of specific sizes for buffering tests, or containing specific data lines. Being able to do this from within the script saves holding lots of little test files. The optional argument specifies n lines of length m. The lines consist of the letter "a". If start of line text is supplied, it replaces "a"s at the start of each line. Underscores in the start of line text are turned into spaces. The optional argument may be repeated. The data lines that follow a "write" command are split into two by a line of four plus signs. Any above the split are written before the fixed-length lines, and any below the split are written after. For example: write test-data 3x30=AB_ 1x50 Pre-data lines ++++ Post-data lines **** This command generates a file containing: Pre-data lines AB aaaaaaaaaaaaaaaaaaaaaaaaaaa AB aaaaaaaaaaaaaaaaaaaaaaaaaaa AB aaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa Post-data lines If there are no fixed-length line specifiers, there is no need to split the data, and a line of plusses is not needed. [sudo] perl This command runs Perl, with the data as its standard input, to allow arbitrary one-off things to be done. CLIENT SCRIPTS -------------- Lines in client scripts are of several kinds: (1) If a line begins with three question marks and a space, the rest of the line defines the start of expected output from the server. If what is received does not match, the client bombs out with an error message. (2) If a line starts with three plus signs followed by a space, the rest of the line specifies a number of seconds to sleep for before proceeding. (3) If a line begins with three '>' characters and a space, the rest of the line is input to be sent to the server. Backslash escaping is done as described below, but no trailing "\r\n" is sent. (4) If a line begin with three '<' characters and a space, the rest of the line is a filename; the content of the file is inserted intto the script at this point. (5) Otherwise, the line is an input line line that is sent to the server. Any occurrences of \r and \n in the line are turned into carriage return and linefeed, respectively. This is used for testing PIPELINING. Any sequences of \x followed by two hex digits are converted to the equvalent byte value. Any other character following a \ is sent verbatim. The line is sent with a trailing "\r\n". Here is a simple example: client 127.0.0.1 PORT_D ??? 220 EHLO xxx ??? 250- ??? 250 AUTH PLAIN AbdXi0AdnD2CVy ??? 535 quit ??? 221 **** In the case of client-gnutls and client-ssl, if a command is "starttls", this is remembered, and after a subsequent OK response, an attempt to move into TLS mode occurs. If a command is "starttls_wait", the client sends "starttls" but does not start up TLS; this is for testing timeouts. If a command is "stoptls", an existing TLS connection is shut down, but nothing is sent. SERVER SCRIPTS -------------- The server program sleeps till a connection occurs or its timeout is reached, in which case it bombs out. The next set of command lines are interpreted. They are of the following kinds: (1) A line that starts with '>' or with a digit is an output line that is sent to the client. In the case of '>': (a) If the line starts with ">>", no terminating CRLF is sent. (b) If the line starts with ">CR>", just CR is sent at the end. (c) If the line starts with ">LF>", just LF is sent at the end. (d) If the line starts with ">*eof", nothing is sent and the connection is closed. The data that is sent starts after the initial '>' sequence. Within each line the sequence '\x' followed by two hex digits can be used to specify an arbitrary byte value. The sequence '\\' specifies a single backslash. (2) A line that starts with "*sleep" specifies a number of seconds to wait before proceeding. (3) A line containing "*eof" specifies that the client is expected to close the connection at this point. (4) A line containing just '.' specifies that the client is expected to send many lines, terminated by one that contains just a dot. (5) Otherwise, the line defines the start of an input line that the client is expected to send. To allow for lines that start with digits, the line may start with '<', which is not taken as part of the input data. If the lines starts with '<<' then only the characters are expected; no return- linefeed terminator. If the input does not match, the server bombs out with an error message. Backslash-escape sequences may be used in the line content as for output lines. Here is a simple example of server use in a test script: server PORT_S 220 Greetings EHLO 250 Hello there MAIL FROM 250 OK RCPT TO 250 OK DATA 354 Send it! . 250 OK QUIT 225 OK **** After a "server" command in a test script, the server runs in parallel until an "exim" command is reached. The "exim" command attempts to deliver one or more messages to port PORT_S on the local host. When it has finished, the test script waits for the "server" process to finish. The "mtpscript" program is like "server", except that it uses stdin/stdout for its input and output instead of a script. However, it is not called from test scripts; instead it is used as the command for pipe transports in some configurations, to simulate non-socket LMTP servers. AUXILIARY DATA FILES -------------------- Many of the tests make use of auxiliary data files. There are two types; those whose content is fixed, and those whose content needs to be varied according to the current environment. The former are kept in the directory aux-fixed. The latter are distributed in the directory aux-var-src, and copied with the standard substitutions into the directory aux-var at the start of each test run. Most of the auxiliary files have names that start with a test number, indicating that they are specific to that one test. A few fixed files (for example, some TLS certificates) are used by more than one test, and so their names are not of this form. There are also some auxiliary DNS zone files, which are described in the next section. DNS LOOKUPS AND GETHOSTBYNAME ----------------------------- The original test suite required special testing zones to be loaded into a local nameserver. This is no longer a requirement for the new suite. Instead, a program called fakens is used to simulate a nameserver. When Exim is running in the test harness, instead of calling res_search() - the normal call to the DNS resolver - it calls a testing function. This handles a few special names itself (for compatibility with the old test suite), but otherwise passes the query to the fakens program. The fakens program consults "zone files" in the directory called dnszones, and returns data in the standard resource record format for Exim to process as if it came from the DNS. However, if the requested domain is not in any of the zones that fakens knows about, it returns a special code that causes Exim to pass the query on to res_search(). The zone files are: db.test.ex A zone for the domain test.ex. db.ip4.10 A zone for one special case in 10.250.0.0/16 (see below) db.ip4.V4NET A zone for the domain V4NET.in-addr.arpa. db.ip4.127 A zone for the domain 127.in-addr.arpa. db.ip6.V6NET A zone for the domain inverted(V6NET).ip6.arpa. db.ip6.0 A zone for the domain 0.ip6.arpa. V4NET and V6NET are substituted with the current testing networks (see above). In the case of V6NET, the network is four hex digits, and it is split and inverted appropriately when setting up the zone. These fake zone files are built dynamically from sources in the dnszones-src directory by applying the standard substitutions. The test suite also builds dynamic zone files for the name of the current host and its IP address(es). The idea is that there should not be any need to rely on an external DNS. The fakens program handles some names programmatically rather than using the fake zone files. These are: manyhome.test.ex This name is used for testing hosts with ridiculously large numbers of IP addresses; 2048 IP addresses are generated and returned. Doing it this way saves having to make the interface to fakens handle more records that can fit in the data block. The addresses that are generated are in the 10.250.0.0/16 network. test.again.dns This always provokes a TRY_AGAIN response, for testing the handling of temporary DNS error. If the full domain name starts with digits, a delay of that many seconds occurs. test.fail.dns This always provokes a NO_RECOVERY response, for testing DNS server failures. The use of gethostbyname() and its IPv6 friends is also subverted when Exim is running in the test harness. The test code handles a few special names directly; for all the others it uses DNS lookups, which are then handled as just described. Thus, the use of /etc/hosts is completely bypassed. The names that are specially handled are: localhost Always returns 127.0.0.1 or ::1, for IPv4 and IPv6 lookups, respectively. If the IP address is of the correct form for the lookup type (IPv4 or IPv6), it is returned. Otherwise a panic-die error occurs. The reverse zone db.ip4.10 is provided just for the manyhome.test.ex case. It contains a single wildcard resource record. It also contains the line PASS ON NOT FOUND Whenever fakens finds this line in a zone file, it returns PASS_ON instead of HOST_NOT_FOUND. This causes Exim to pass the query to res_search(). ****