GNU Automake 版本(version 1.16.1, 26 February 2018)
Permission is granted to copy, distribute and/or modify this
document under the terms of the GNU Free Documentation License,
Version 1.3 or any later version published by the Free Software
Foundation; with no Invariant Sections, with no Front-Cover texts,
and with no Back-Cover Texts. A copy of the license is included in
the section entitled “GNU Free Documentation License.”
15 Support for test suites
Automake can generate code to handle two kinds of test suites. One is
based on integration with the ‘dejagnu’ framework. The other (and most
used) form is based on the use of generic test scripts, and its
activation is triggered by the definition of the special ‘TESTS’
variable. This second form allows for various degrees of sophistication
and customization; in particular, it allows for concurrent execution of
test scripts, use of established test protocols such as TAP, and
definition of custom test drivers and test runners.
In either case, the testsuite is invoked via ‘make check’.
15.1 Generalities about Testing
The purpose of testing is to determine whether a program or system
behaves as expected (e.g., known inputs produce the expected outputs,
error conditions are correctly handled or reported, and older bugs do
not resurface).
The minimal unit of testing is usually called test case, or simply
test. How a test case is defined or delimited, and even what exactly
constitutes a test case, depends heavily on the testing paradigm
and/or framework in use, so we won’t attempt any more precise
definition. The set of the test cases for a given program or system
constitutes its testsuite.
A test harness (also testsuite harness) is a program or software
component that executes all (or part of) the defined test cases,
analyzes their outcomes, and report or register these outcomes
appropriately. Again, the details of how this is accomplished (and how
the developer and user can influence it or interface with it) varies
wildly, and we’ll attempt no precise definition.
A test is said to pass when it can determine that the condition or
behaviour it means to verify holds, and is said to fail when it can
determine that such condition of behaviour does not hold.
Sometimes, tests can rely on non-portable tools or prerequisites, or
simply make no sense on a given system (for example, a test checking a
Windows-specific feature makes no sense on a GNU/Linux system). In this
case, accordingly to the definition above, the tests can neither be
considered passed nor failed; instead, they are skipped – i.e., they
are not run, or their result is anyway ignored for what concerns the
count of failures an successes. Skips are usually explicitly reported
though, so that the user will be aware that not all of the testsuite has
really run.
It’s not uncommon, especially during early development stages, that
some tests fail for known reasons, and that the developer doesn’t want
to tackle these failures immediately (this is especially true when the
failing tests deal with corner cases). In this situation, the better
policy is to declare that each of those failures is an expected
failure (or xfail). In case a test that is expected to fail ends up
passing instead, many testing environments will flag the result as a
special kind of failure called unexpected pass (or xpass).
Many testing environments and frameworks distinguish between test
failures and hard errors. As we’ve seen, a test failure happens when
some invariant or expected behaviour of the software under test is not
met. An hard error happens when e.g., the set-up of a test case
scenario fails, or when some other unexpected or highly undesirable
condition is encountered (for example, the program under test
experiences a segmentation fault).
15.2 Simple Tests
15.2.1 Scripts-based Testsuites
If the special variable ‘TESTS’ is defined, its value is taken to be a
list of programs or scripts to run in order to do the testing. Under
the appropriate circumstances, it’s possible for ‘TESTS’ to list also
data files to be passed to one or more test scripts defined by different
means (the so-called “log compilers”, *note Parallel Test Harness::).
Test scripts can be executed serially or concurrently. Automake
supports both these kinds of test execution, with the parallel test
harness being the default. The concurrent test harness relies on the
concurrence capabilities (if any) offered by the underlying ‘make’
implementation, and can thus only be as good as those are.
By default, only the exit statuses of the test scripts are considered
when determining the testsuite outcome. But Automake allows also the
use of more complex test protocols, either standard (*note Using the TAP
test protocol::) or custom (*note Custom Test Drivers::). Note that you
can’t enable such protocols when the serial harness is used, though. In
the rest of this section we are going to concentrate mostly on
protocol-less tests, since we cover test protocols in a later section
(again, *note Custom Test Drivers::).
When no test protocol is in use, an exit status of 0 from a test
script will denote a success, an exit status of 77 a skipped test, an
exit status of 99 an hard error, and any other exit status will denote a
failure.
You may define the variable ‘XFAIL_TESTS’ to a list of tests (usually
a subset of ‘TESTS’) that are expected to fail; this will effectively
reverse the result of those tests (with the provision that skips and
hard errors remain untouched). You may also instruct the testsuite
harness to treat hard errors like simple failures, by defining the
‘DISABLE_HARD_ERRORS’ make variable to a nonempty value.
Note however that, for tests based on more complex test protocols,
the exact effects of ‘XFAIL_TESTS’ and ‘DISABLE_HARD_ERRORS’ might
change, or they might even have no effect at all (for example, in tests
using TAP, there is no way to disable hard errors, and the
‘DISABLE_HARD_ERRORS’ variable has no effect on them).
The result of each test case run by the scripts in ‘TESTS’ will be
printed on standard output, along with the test name. For test
protocols that allow more test cases per test script (such as TAP), a
number, identifier and/or brief description specific for the single test
case is expected to be printed in addition to the name of the test
script. The possible results (whose meanings should be clear from the
previous *note Generalities about Testing::) are ‘PASS’, ‘FAIL’, ‘SKIP’,
‘XFAIL’, ‘XPASS’ and ‘ERROR’. Here is an example of output from an
hypothetical testsuite that uses both plain and TAP tests:
PASS: foo.sh
PASS: zardoz.tap 1 - Daemon started
PASS: zardoz.tap 2 - Daemon responding
SKIP: zardoz.tap 3 - Daemon uses /proc # SKIP /proc is not mounted
PASS: zardoz.tap 4 - Daemon stopped
SKIP: bar.sh
PASS: mu.tap 1
XFAIL: mu.tap 2 # TODO frobnication not yet implemented
A testsuite summary (expected to report at least the number of run,
skipped and failed tests) will be printed at the end of the testsuite
run.
If the standard output is connected to a capable terminal, then the
test results and the summary are colored appropriately. The developer
and the user can disable colored output by setting the ‘make’ variable
‘AM_COLOR_TESTS=no’; the user can in addition force colored output even
without a connecting terminal with ‘AM_COLOR_TESTS=always’. It’s also
worth noting that some ‘make’ implementations, when used in parallel
mode, have slightly different semantics (*note (autoconf)Parallel
make::), which can break the automatic detection of a connection to a
capable terminal. If this is the case, the user will have to resort to
the use of ‘AM_COLOR_TESTS=always’ in order to have the testsuite output
colorized.
Test programs that need data files should look for them in ‘srcdir’
(which is both a make variable and an environment variable made
available to the tests), so that they work when building in a separate
directory (*note Build Directories: (autoconf)Build Directories.), and
in particular for the ‘distcheck’ rule (*note Checking the
Distribution::).
The ‘AM_TESTS_ENVIRONMENT’ and ‘TESTS_ENVIRONMENT’ variables can be
used to run initialization code and set environment variables for the
test scripts. The former variable is developer-reserved, and can be
defined in the ‘Makefile.am’, while the latter is reserved for the user,
which can employ it to extend or override the settings in the former;
for this to work portably, however, the contents of a non-empty
‘AM_TESTS_ENVIRONMENT’ must be terminated by a semicolon.
The ‘AM_TESTS_FD_REDIRECT’ variable can be used to define file
descriptor redirections for the test scripts. One might think that
‘AM_TESTS_ENVIRONMENT’ could be used for this purpose, but experience
has shown that doing so portably is practically impossible. The main
hurdle is constituted by Korn shells, which usually set the
close-on-exec flag on file descriptors opened with the ‘exec’ builtin,
thus rendering an idiom like ‘AM_TESTS_ENVIRONMENT = exec 9>&2;’
ineffectual. This issue also affects some Bourne shells, such as the
HP-UX’s ‘/bin/sh’,
AM_TESTS_ENVIRONMENT = \
## Some environment initializations are kept in a separate shell
## file 'tests-env.sh', which can make it easier to also run tests
## from the command line.
. $(srcdir)/tests-env.sh; \
## On Solaris, prefer more POSIX-compliant versions of the standard
## tools by default.
if test -d /usr/xpg4/bin; then \
PATH=/usr/xpg4/bin:$$PATH; export PATH; \
fi;
## With this, the test scripts will be able to print diagnostic
## messages to the original standard error stream, even if the test
## driver redirects the stderr of the test scripts to a log file
## before executing them.
AM_TESTS_FD_REDIRECT = 9>&2
Note however that ‘AM_TESTS_ENVIRONMENT’ is, for historical and
implementation reasons, not supported by the serial harness (*note
Serial Test Harness::).
Automake ensures that each file listed in ‘TESTS’ is built before it
is run; you can list both source and derived programs (or scripts) in
‘TESTS’; the generated rule will look both in ‘srcdir’ and ‘.’. For
instance, you might want to run a C program as a test. To do this you
would list its name in ‘TESTS’ and also in ‘check_PROGRAMS’, and then
specify it as you would any other program.
Programs listed in ‘check_PROGRAMS’ (and ‘check_LIBRARIES’,
‘check_LTLIBRARIES’…) are only built during ‘make check’, not during
‘make all’. You should list there any program needed by your tests that
does not need to be built by ‘make all’. Note that ‘check_PROGRAMS’ are
not automatically added to ‘TESTS’ because ‘check_PROGRAMS’ usually
lists programs used by the tests, not the tests themselves. Of course
you can set ‘TESTS = $(check_PROGRAMS)’ if all your programs are test
cases.
15.2.2 Older (and discouraged) serial test harness
First, note that today the use of this harness is strongly discouraged
in favour of the parallel test harness (*note Parallel Test Harness::).
Still, there are few situations when the advantages offered by the
parallel harness are irrelevant, and when test concurrency can even
cause tricky problems. In those cases, it might make sense to still use
the serial harness, for simplicity and reliability (we still suggest
trying to give the parallel harness a shot though).
The serial test harness is enabled by the Automake option
‘serial-tests’. It operates by simply running the tests serially, one
at the time, without any I/O redirection. It’s up to the user to
implement logging of tests’ output, if that’s required or desired.
For historical and implementation reasons, the ‘AM_TESTS_ENVIRONMENT’
variable is not supported by this harness (it will be silently ignored
if defined); only ‘TESTS_ENVIRONMENT’ is, and it is to be considered a
developer-reserved variable. This is done so that, when using the
serial harness, ‘TESTS_ENVIRONMENT’ can be defined to an invocation of
an interpreter through which the tests are to be run. For instance, the
following setup may be used to run tests with Perl:
TESTS_ENVIRONMENT = $(PERL) -Mstrict -w
TESTS = foo.pl bar.pl baz.pl
It’s important to note that the use of ‘TESTS_ENVIRONMENT’ endorsed here
would be invalid with the parallel harness. That harness provides a
more elegant way to achieve the same effect, with the further benefit of
freeing the ‘TESTS_ENVIRONMENT’ variable for the user (*note Parallel
Test Harness::).
Another, less serious limit of the serial harness is that it doesn’t
really distinguish between simple failures and hard errors; this is due
to historical reasons only, and might be fixed in future Automake
versions.
15.2.3 Parallel Test Harness
By default, Automake generated a parallel (concurrent) test harness. It
features automatic collection of the test scripts output in ‘.log’
files, concurrent execution of tests with ‘make -j’, specification of
inter-test dependencies, lazy reruns of tests that have not completed in
a prior run, and hard errors for exceptional failures.
The parallel test harness operates by defining a set of ‘make’ rules
that run the test scripts listed in ‘TESTS’, and, for each such script,
save its output in a corresponding ‘.log’ file and its results (and
other “metadata”, *note API for Custom Test Drivers::) in a
corresponding ‘.trs’ (as in Test ReSults) file. The ‘.log’ file will
contain all the output emitted by the test on its standard output and
its standard error. The ‘.trs’ file will contain, among the other
things, the results of the test cases run by the script.
The parallel test harness will also create a summary log file,
‘TEST_SUITE_LOG’, which defaults to ‘test-suite.log’ and requires a
‘.log’ suffix. This file depends upon all the ‘.log’ and ‘.trs’ files
created for the test scripts listed in ‘TESTS’.
As with the serial harness above, by default one status line is
printed per completed test, and a short summary after the suite has
completed. However, standard output and standard error of the test are
redirected to a per-test log file, so that parallel execution does not
produce intermingled output. The output from failed tests is collected
in the ‘test-suite.log’ file. If the variable ‘VERBOSE’ is set, this
file is output after the summary.
Each couple of ‘.log’ and ‘.trs’ files is created when the
corresponding test has completed. The set of log files is listed in the
read-only variable ‘TEST_LOGS’, and defaults to ‘TESTS’, with the
executable extension if any (*note EXEEXT::), as well as any suffix
listed in ‘TEST_EXTENSIONS’ removed, and ‘.log’ appended. Results are
undefined if a test file name ends in several concatenated suffixes.
‘TEST_EXTENSIONS’ defaults to ‘.test’; it can be overridden by the user,
in which case any extension listed in it must be constituted by a dot,
followed by a non-digit alphabetic character, followed by any number of
alphabetic characters. For example, ‘.sh’, ‘.T’ and ‘.t1’ are valid
extensions, while ‘.x-y’, ‘.6c’ and ‘.t.1’ are not.
It is important to note that, due to current limitations (unlikely to
be lifted), configure substitutions in the definition of ‘TESTS’ can
only work if they will expand to a list of tests that have a suffix
listed in ‘TEST_EXTENSIONS’.
For tests that match an extension ‘.EXT’ listed in ‘TEST_EXTENSIONS’,
you can provide a custom “test runner” using the variable
‘EXT_LOG_COMPILER’ (note the upper-case extension) and pass options in
‘AM_EXT_LOG_FLAGS’ and allow the user to pass options in
‘EXT_LOG_FLAGS’. It will cause all tests with this extension to be
called with this runner. For all tests without a registered extension,
the variables ‘LOG_COMPILER’, ‘AM_LOG_FLAGS’, and ‘LOG_FLAGS’ may be
used. For example,
TESTS = foo.pl bar.py baz
TEST_EXTENSIONS = .pl .py
PL_LOG_COMPILER = $(PERL)
AM_PL_LOG_FLAGS = -w
PY_LOG_COMPILER = $(PYTHON)
AM_PY_LOG_FLAGS = -v
LOG_COMPILER = ./wrapper-script
AM_LOG_FLAGS = -d
will invoke ‘$(PERL) -w foo.pl’, ‘$(PYTHON) -v bar.py’, and
‘./wrapper-script -d baz’ to produce ‘foo.log’, ‘bar.log’, and
‘baz.log’, respectively. The ‘foo.trs’, ‘bar.trs’ and ‘baz.trs’ files
will be automatically produced as a side-effect.
It’s important to note that, differently from what we’ve seen for the
serial test harness (*note Serial Test Harness::), the
‘AM_TESTS_ENVIRONMENT’ and ‘TESTS_ENVIRONMENT’ variables cannot be
used to define a custom test runner; the ‘LOG_COMPILER’ and ‘LOG_FLAGS’
(or their extension-specific counterparts) should be used instead:
## This is WRONG!
AM_TESTS_ENVIRONMENT = PERL5LIB='$(srcdir)/lib' $(PERL) -Mstrict -w
## Do this instead.
AM_TESTS_ENVIRONMENT = PERL5LIB='$(srcdir)/lib'; export PERL5LIB;
LOG_COMPILER = $(PERL)
AM_LOG_FLAGS = -Mstrict -w
By default, the test suite harness will run all tests, but there are
several ways to limit the set of tests that are run:
• You can set the ‘TESTS’ variable. For example, you can use a
command like this to run only a subset of the tests:
env TESTS="foo.test bar.test" make -e check
Note however that the command above will unconditionally overwrite
the ‘test-suite.log’ file, thus clobbering the recorded results of
any previous testsuite run. This might be undesirable for packages
whose testsuite takes long time to execute. Luckily, this problem
can easily be avoided by overriding also ‘TEST_SUITE_LOG’ at
runtime; for example,
env TEST_SUITE_LOG=partial.log TESTS="..." make -e check
will write the result of the partial testsuite runs to the
‘partial.log’, without touching ‘test-suite.log’.
• You can set the ‘TEST_LOGS’ variable. By default, this variable is
computed at ‘make’ run time from the value of ‘TESTS’ as described
above. For example, you can use the following:
set x subset*.log; shift
env TEST_LOGS="foo.log $*" make -e check
The comments made above about ‘TEST_SUITE_LOG’ overriding applies
here too.
• By default, the test harness removes all old per-test ‘.log’ and
‘.trs’ files before it starts running tests to regenerate them.
The variable ‘RECHECK_LOGS’ contains the set of ‘.log’ (and, by
implication, ‘.trs’) files which are removed. ‘RECHECK_LOGS’
defaults to ‘TEST_LOGS’, which means all tests need to be
rechecked. By overriding this variable, you can choose which tests
need to be reconsidered. For example, you can lazily rerun only
those tests which are outdated, i.e., older than their prerequisite
test files, by setting this variable to the empty value:
env RECHECK_LOGS= make -e check
• You can ensure that all tests are rerun which have failed or passed
unexpectedly, by running ‘make recheck’ in the test directory.
This convenience target will set ‘RECHECK_LOGS’ appropriately
before invoking the main test harness.
In order to guarantee an ordering between tests even with ‘make -jN’,
dependencies between the corresponding ‘.log’ files may be specified
through usual ‘make’ dependencies. For example, the following snippet
lets the test named ‘foo-execute.test’ depend upon completion of the
test ‘foo-compile.test’:
TESTS = foo-compile.test foo-execute.test
foo-execute.log: foo-compile.log
Please note that this ordering ignores the results of required tests,
thus the test ‘foo-execute.test’ is run even if the test
‘foo-compile.test’ failed or was skipped beforehand. Further, please
note that specifying such dependencies currently works only for tests
that end in one of the suffixes listed in ‘TEST_EXTENSIONS’.
Tests without such specified dependencies may be run concurrently
with parallel ‘make -jN’, so be sure they are prepared for concurrent
execution.
The combination of lazy test execution and correct dependencies
between tests and their sources may be exploited for efficient unit
testing during development. To further speed up the edit-compile-test
cycle, it may even be useful to specify compiled programs in
‘EXTRA_PROGRAMS’ instead of with ‘check_PROGRAMS’, as the former allows
intertwined compilation and test execution (but note that
‘EXTRA_PROGRAMS’ are not cleaned automatically, *note Uniform::).
The variables ‘TESTS’ and ‘XFAIL_TESTS’ may contain conditional parts
as well as configure substitutions. In the latter case, however,
certain restrictions apply: substituted test names must end with a
nonempty test suffix like ‘.test’, so that one of the inference rules
generated by ‘automake’ can apply. For literal test names, ‘automake’
can generate per-target rules to avoid this limitation.
Please note that it is currently not possible to use ‘$(srcdir)/’ or
‘$(top_srcdir)/’ in the ‘TESTS’ variable. This technical limitation is
necessary to avoid generating test logs in the source tree and has the
unfortunate consequence that it is not possible to specify distributed
tests that are themselves generated by means of explicit rules, in a way
that is portable to all ‘make’ implementations (*note (autoconf)Make
Target Lookup::, the semantics of FreeBSD and OpenBSD ‘make’ conflict
with this). In case of doubt you may want to require to use GNU ‘make’,
or work around the issue with inference rules to generate the tests.
15.3 Custom Test Drivers
15.3.1 Overview of Custom Test Drivers Support
Starting from Automake version 1.12, the parallel test harness allows
the package authors to use third-party custom test drivers, in case the
default ones are inadequate for their purposes, or do not support their
testing protocol of choice.
A custom test driver is expected to properly run the test programs
passed to it (including the command-line arguments passed to those
programs, if any), to analyze their execution and outcome, to create the
‘.log’ and ‘.trs’ files associated to these test runs, and to display
the test results on the console. It is responsibility of the author of
the test driver to ensure that it implements all the above steps
meaningfully and correctly; Automake isn’t and can’t be of any help
here. On the other hand, the Automake-provided code for testsuite
summary generation offers support for test drivers allowing several test
results per test script, if they take care to register such results
properly (*note Log files generation and test results recording::).
The exact details of how test scripts’ results are to be determined
and analyzed is left to the individual drivers. Some drivers might only
consider the test script exit status (this is done for example by the
default test driver used by the parallel test harness, described in the
previous section). Other drivers might implement more complex and
advanced test protocols, which might require them to parse and
interpreter the output emitted by the test script they’re running
(examples of such protocols are TAP and SubUnit).
It’s very important to note that, even when using custom test
drivers, most of the infrastructure described in the previous section
about the parallel harness remains in place; this includes:
• list of test scripts defined in ‘TESTS’, and overridable at runtime
through the redefinition of ‘TESTS’ or ‘TEST_LOGS’;
• concurrency through the use of ‘make’’s option ‘-j’;
• per-test ‘.log’ and ‘.trs’ files, and generation of a summary
‘.log’ file from them;
• ‘recheck’ target, ‘RECHECK_LOGS’ variable, and lazy reruns of
tests;
• inter-test dependencies;
• support for ‘check_*’ variables (‘check_PROGRAMS’,
‘check_LIBRARIES’, …);
• use of ‘VERBOSE’ environment variable to get verbose output on
testsuite failures;
• definition and honoring of ‘TESTS_ENVIRONMENT’,
‘AM_TESTS_ENVIRONMENT’ and ‘AM_TESTS_FD_REDIRECT’ variables;
• definition of generic and extension-specific ‘LOG_COMPILER’ and
‘LOG_FLAGS’ variables.
On the other hand, the exact semantics of how (and if) testsuite output
colorization, ‘XFAIL_TESTS’, and hard errors are supported and handled
is left to the individual test drivers.
15.3.2 Declaring Custom Test Drivers
Custom testsuite drivers are declared by defining the make variables
‘LOG_DRIVER’ or ‘EXT_LOG_DRIVER’ (where EXT must be declared in
‘TEST_EXTENSIONS’). They must be defined to programs or scripts that
will be used to drive the execution, logging, and outcome report of the
tests with corresponding extensions (or of those with no registered
extension in the case of ‘LOG_DRIVER’). Clearly, multiple distinct test
drivers can be declared in the same ‘Makefile.am’. Note moreover that
the ‘LOG_DRIVER’ variables are not a substitute for the ‘LOG_COMPILER’
variables: the two sets of variables can, and often do, usefully and
legitimately coexist.
The developer-reserved variable ‘AM_LOG_DRIVER_FLAGS’ and the
user-reserved variable ‘LOG_DRIVER_FLAGS’ can be used to define flags
that will be passed to each invocation of ‘LOG_DRIVER’, with the
user-defined flags obviously taking precedence over the
developer-reserved ones. Similarly, for each extension EXT declared in
‘TEST_EXTENSIONS’, flags listed in ‘AM_EXT_LOG_DRIVER_FLAGS’ and
‘EXT_LOG_DRIVER_FLAGS’ will be passed to invocations of
‘EXT_LOG_DRIVER’.
15.3.3 API for Custom Test Drivers
Note that the APIs described here are still highly experimental, and
will very likely undergo tightenings and likely also extensive changes
in the future, to accommodate for 新特性 or to satisfy additional
portability requirements.
The main characteristic of these APIs is that they are designed to
share as much infrastructure, semantics, and implementation details as
possible with the parallel test harness and its default driver.
15.3.3.1 Command-line arguments for test drivers
…………………………………………
A custom driver can rely on various command-line options and arguments
being passed to it automatically by the Automake-generated test harness.
It is mandatory that it understands all of them (even if the exact
interpretation of the associated semantics can legitimately change
between a test driver and another, and even be a no-op in some drivers).
Here is the list of options:
‘–test-name=NAME’
The name of the test, with VPATH prefix (if any) removed. This can
have a suffix and a directory component (as in e.g.,
‘sub/foo.test’), and is mostly meant to be used in console reports
about testsuite advancements and results (*note Testsuite progress
output::).
‘–log-file=PATH.log’
The ‘.log’ file the test driver must create (*note Basics of test
metadata::). If it has a directory component (as in e.g.,
‘sub/foo.log’), the test harness will ensure that such directory
exists before the test driver is called.
‘–trs-file=PATH.trs’
The ‘.trs’ file the test driver must create (*note Basics of test
metadata::). If it has a directory component (as in e.g.,
‘sub/foo.trs’), the test harness will ensure that such directory
exists before the test driver is called.
‘–color-tests={yes|no}’
Whether the console output should be colorized or not (*note Simple
tests and color-tests::, to learn when this option gets activated
and when it doesn’t).
‘–expect-failure={yes|no}’
Whether the tested program is expected to fail.
‘–enable-hard-errors={yes|no}’
Whether “hard errors” in the tested program should be treated
differently from normal failures or not (the default should be
‘yes’). The exact meaning of “hard error” is highly dependent from
the test protocols or conventions in use.
‘–’
Explicitly terminate the list of options.
The first non-option argument passed to the test driver is the program
to be run, and all the following ones are command-line options and
arguments for this program.
Note that the exact semantics attached to the ‘–color-tests’,
‘–expect-failure’ and ‘–enable-hard-errors’ options are left up to the
individual test drivers. Still, having a behaviour compatible or at
least similar to that provided by the default driver is advised, as that
would offer a better consistency and a more pleasant user experience.
15.3.3.2 Log files generation and test results recording
………………………………………………..
The test driver must correctly generate the files specified by the
‘–log-file’ and ‘–trs-file’ option (even when the tested program fails
or crashes).
The ‘.log’ file should ideally contain all the output produced by the
tested program, plus optionally other information that might facilitate
debugging or analysis of bug reports. Apart from that, its format is
basically free.
The ‘.trs’ file is used to register some metadata through the use of
custom reStructuredText fields. This metadata is expected to be
employed in various ways by the parallel test harness; for example, to
count the test results when printing the testsuite summary, or to decide
which tests to re-run upon ‘make recheck’. Unrecognized metadata in a
‘.trs’ file is currently ignored by the harness, but this might change
in the future. The list of currently recognized metadata follows.
‘:test-result:’
The test driver must use this field to register the results of
each test case run by a test script file. Several
‘:test-result:’ fields can be present in the same ‘.trs’ file; this
is done in order to support test protocols that allow a single test
script to run more test cases.
The only recognized test results are currently ‘PASS’, ‘XFAIL’,
‘SKIP’, ‘FAIL’, ‘XPASS’ and ‘ERROR’. These results, when declared
with ‘:test-result:’, can be optionally followed by text holding
the name and/or a brief description of the corresponding test; the
harness will ignore such extra text when generating
‘test-suite.log’ and preparing the testsuite summary.
‘:recheck:’
If this field is present and defined to ‘no’, then the
corresponding test script will not be run upon a ‘make recheck’.
What happens when two or more ‘:recheck:’ fields are present in the
same ‘.trs’ file is undefined behaviour.
‘:copy-in-global-log:’
If this field is present and defined to ‘no’, then the content of
the ‘.log’ file will not be copied into the global
‘test-suite.log’. We allow to forsake such copying because, while
it can be useful in debugging and analysis of bug report, it can
also be just a waste of space in normal situations, e.g., when a
test script is successful. What happens when two or more
‘:copy-in-global-log:’ fields are present in the same ‘.trs’ file
is undefined behaviour.
‘:test-global-result:’
This is used to declare the “global result” of the script.
Currently, the value of this field is needed only to be reported
(more or less verbatim) in the generated global log file
‘$(TEST_SUITE_LOG)’, so it’s quite free-form. For example, a test
script which run 10 test cases, 6 of which pass and 4 of which are
skipped, could reasonably have a ‘PASS/SKIP’ value for this field,
while a test script which run 19 successful tests and one failed
test could have an ‘ALMOST PASSED’ value. What happens when two or
more ‘:test-global-result:’ fields are present in the same ‘.trs’
file is undefined behaviour.
Let’s see a small example. Assume a ‘.trs’ file contains the following
lines:
:test-result: PASS server starts
:global-log-copy: no
:test-result: PASS HTTP/1.1 request
:test-result: FAIL HTTP/1.0 request
:recheck: yes
:test-result: SKIP HTTPS request (TLS library wasn't available)
:test-result: PASS server stops
Then the corresponding test script will be re-run by ‘make check’, will
contribute with five test results to the testsuite summary (three of
these tests being successful, one failed, and one skipped), and the
content of the corresponding ‘.log’ file will not be copied in the
global log file ‘test-suite.log’.
15.3.3.3 Testsuite progress output
…………………………….
A custom test driver also has the task of displaying, on the standard
output, the test results as soon as they become available. Depending on
the protocol in use, it can also display the reasons for failures and
skips, and, more generally, any useful diagnostic output (but remember
that each line on the screen is precious, so that cluttering the screen
with overly verbose information is bad idea). The exact format of this
progress output is left up to the test driver; in fact, a custom test
driver might theoretically even decide not to do any such report,
leaving it all to the testsuite summary (that would be a very lousy
idea, of course, and serves only to illustrate the flexibility that is
granted here).
Remember that consistency is good; so, if possible, try to be
consistent with the output of the built-in Automake test drivers,
providing a similar “look & feel”. In particular, the testsuite
progress output should be colorized when the ‘–color-tests’ is passed
to the driver. On the other end, if you are using a known and
widespread test protocol with well-established implementations, being
consistent with those implementations’ output might be a good idea too.
15.4 Using the TAP test protocol
15.4.1 Introduction to TAP
TAP, the Test Anything Protocol, is a simple text-based interface
between testing modules or programs and a test harness. The tests (also
called “TAP producers” in this context) write test results in a simple
format on standard output; a test harness (also called “TAP consumer”)
will parse and interpret these results, and properly present them to the
user, and/or register them for later analysis. The exact details of how
this is accomplished can vary among different test harnesses. The
Automake harness will present the results on the console in the usual
fashion (*note Testsuite progress on console::), and will use the ‘.trs’
files (*note Basics of test metadata::) to store the test results and
related metadata. Apart from that, it will try to remain as much
compatible as possible with pre-existing and widespread utilities, such
as the ‘prove’ utility
(http://search.cpan.org/~andya/Test-Harness/bin/prove), at least for the
simpler usages.
TAP started its life as part of the test harness for Perl, but today
it has been (mostly) standardized, and has various independent
implementations in different languages; among them, C, C++, Perl,
Python, PHP, and Java. For a semi-official specification of the TAP
protocol, please refer to the documentation of ‘Test::Harness::TAP’
(http://search.cpan.org/~petdance/Test-Harness/lib/Test/Harness/TAP.pod).
The most relevant real-world usages of TAP are obviously in the
testsuites of ‘perl’ and of many perl modules. Still, also other
important third-party packages, such as ‘git’ (http://git-scm.com/), use
TAP in their testsuite.
15.4.2 Use TAP with the Automake test harness
Currently, the TAP driver that comes with Automake requires some by-hand
steps on the developer’s part (this situation should hopefully be
improved in future Automake versions). You’ll have to grab the
‘tap-driver.sh’ script from the Automake distribution by hand, copy it
in your source tree, and use the Automake support for third-party test
drivers to instruct the harness to use the ‘tap-driver.sh’ script and
the awk program found by ‘AM_INIT_AUTOMAKE’ to run your TAP-producing
tests. See the example below for clarification.
Apart from the options common to all the Automake test drivers (*note
Command-line arguments for test drivers::), the ‘tap-driver.sh’ supports
the following options, whose names are chosen for enhanced compatibility
with the ‘prove’ utility.
‘–ignore-exit’
Causes the test driver to ignore the exit status of the test
scripts; by default, the driver will report an error if the script
exits with a non-zero status. This option has effect also on
non-zero exit statuses due to termination by a signal.
‘–comments’
Instruct the test driver to display TAP diagnostic (i.e., lines
beginning with the ‘#’ character) in the testsuite progress output
too; by default, TAP diagnostic is only copied to the ‘.log’ file.
‘–no-comments’
Revert the effects of ‘–comments’.
‘–merge’
Instruct the test driver to merge the test scripts’ standard error
into their standard output. This is necessary if you want to
ensure that diagnostics from the test scripts are displayed in the
correct order relative to test results; this can be of great help
in debugging (especially if your test scripts are shell scripts run
with shell tracing active). As a downside, this option might cause
the test harness to get confused if anything that appears on
standard error looks like a test result.
‘–no-merge’
Revert the effects of ‘–merge’.
‘–diagnostic-string=STRING’
Change the string that introduces TAP diagnostic from the default
value of “‘#’” to ‘STRING’. This can be useful if your TAP-based
test scripts produce verbose output on which they have limited
control (because, say, the output comes from other tools invoked in
the scripts), and it might contain text that gets spuriously
interpreted as TAP diagnostic: such an issue can be solved by
redefining the string that activates TAP diagnostic to a value you
know won’t appear by chance in the tests’ output. Note however
that this feature is non-standard, as the “official” TAP protocol
does not allow for such a customization; so don’t use it if you can
avoid it.
Here is an example of how the TAP driver can be set up and used.
% cat configure.ac
AC_INIT([GNU Try Tap], [1.0], [bug-automake@gnu.org])
AC_CONFIG_AUX_DIR([build-aux])
AM_INIT_AUTOMAKE([foreign -Wall -Werror])
AC_CONFIG_FILES([Makefile])
AC_REQUIRE_AUX_FILE([tap-driver.sh])
AC_OUTPUT
% cat Makefile.am
TEST_LOG_DRIVER = env AM_TAP_AWK='$(AWK)' $(SHELL) \
$(top_srcdir)/build-aux/tap-driver.sh
TESTS = foo.test bar.test baz.test
EXTRA_DIST = $(TESTS)
% cat foo.test
#!/bin/sh
echo 1..4 # Number of tests to be executed.
echo 'ok 1 - Swallows fly'
echo 'not ok 2 - Caterpillars fly # TODO metamorphosis in progress'
echo 'ok 3 - Pigs fly # SKIP not enough acid'
echo '# I just love word plays ...'
echo 'ok 4 - Flies fly too :-)'
% cat bar.test
#!/bin/sh
echo 1..3
echo 'not ok 1 - Bummer, this test has failed.'
echo 'ok 2 - This passed though.'
echo 'Bail out! Ennui kicking in, sorry...'
echo 'ok 3 - This will not be seen.'
% cat baz.test
#!/bin/sh
echo 1..1
echo ok 1
# Exit with error, even if all the tests have been successful.
exit 7
% cp PREFIX/share/automake-APIVERSION/tap-driver.sh .
% autoreconf -vi && ./configure && make check
...
PASS: foo.test 1 - Swallows fly
XFAIL: foo.test 2 - Caterpillars fly # TODO metamorphosis in progress
SKIP: foo.test 3 - Pigs fly # SKIP not enough acid
PASS: foo.test 4 - Flies fly too :-)
FAIL: bar.test 1 - Bummer, this test has failed.
PASS: bar.test 2 - This passed though.
ERROR: bar.test - Bail out! Ennui kicking in, sorry...
PASS: baz.test 1
ERROR: baz.test - exited with status 7
...
Please report to bug-automake@gnu.org
...
% echo exit status: $?
exit status: 1
% env TEST_LOG_DRIVER_FLAGS='--comments --ignore-exit' \
TESTS='foo.test baz.test' make -e check
...
PASS: foo.test 1 - Swallows fly
XFAIL: foo.test 2 - Caterpillars fly # TODO metamorphosis in progress
SKIP: foo.test 3 - Pigs fly # SKIP not enough acid
# foo.test: I just love word plays...
PASS: foo.test 4 - Flies fly too :-)
PASS: baz.test 1
...
% echo exit status: $?
exit status: 0
15.4.3 Incompatibilities with other TAP parsers and drivers
For implementation or historical reasons, the TAP driver and harness as
implemented by Automake have some minors incompatibilities with the
mainstream versions, which you should be aware of.
• A ‘Bail out!’ directive doesn’t stop the whole testsuite, but only
the test script it occurs in. This doesn’t follow TAP
specifications, but on the other hand it maximizes compatibility
(and code sharing) with the “hard error” concept of the default
testsuite driver.
• The ‘version’ and ‘pragma’ directives are not supported.
• The ‘–diagnostic-string’ option of our driver allows to modify the
string that introduces TAP diagnostic from the default value of
“‘#’”. The standard TAP protocol has currently no way to allow
this, so if you use it your diagnostic will be lost to more
compliant tools like ‘prove’ and ‘Test::Harness’
• And there are probably some other small and yet undiscovered
incompatibilities, especially in corner cases or with rare usages.
15.4.4 Links and external resources on TAP
Here are some links to more extensive official or third-party
documentation and resources about the TAP protocol and related tools and
libraries.
• ‘Test::Harness::TAP’
(http://search.cpan.org/~petdance/Test-Harness/lib/Test/Harness/TAP.pod),
the (mostly) official documentation about the TAP format and
protocol.
• ‘prove’ (http://search.cpan.org/~andya/Test-Harness/bin/prove),
the most famous command-line TAP test driver, included in the
distribution of ‘perl’ and ‘Test::Harness’
(http://search.cpan.org/~andya/Test-Harness/lib/Test/Harness.pm).
• The TAP wiki (http://testanything.org/wiki/index.php/Main_Page).
• A “gentle introduction” to testing for perl coders:
‘Test::Tutorial’
(http://search.cpan.org/dist/Test-Simple/lib/Test/Tutorial.pod).
• ‘Test::Simple’
(http://search.cpan.org/~mschwern/Test-Simple/lib/Test/Simple.pm)
and ‘Test::More’
(http://search.cpan.org/~mschwern/Test-Simple/lib/Test/More.pm),
the standard perl testing libraries, which are based on TAP.
• C TAP Harness
(http://www.eyrie.org/~eagle/software/c-tap-harness/), a C-based
project implementing both a TAP producer and a TAP consumer.
• tap4j (http://www.tap4j.org/), a Java-based project implementing
both a TAP producer and a TAP consumer.
15.5 DejaGnu Tests
If ‘dejagnu’ (https://ftp.gnu.org/gnu/dejagnu/) appears in
‘AUTOMAKE_OPTIONS’, then a ‘dejagnu’-based test suite is assumed. The
variable ‘DEJATOOL’ is a list of names that are passed, one at a time,
as the ‘–tool’ argument to ‘runtest’ invocations; it defaults to the
name of the package.
The variable ‘RUNTESTDEFAULTFLAGS’ holds the ‘–tool’ and ‘–srcdir’
flags that are passed to dejagnu by default; this can be overridden if
necessary.
The variables ‘EXPECT’ and ‘RUNTEST’ can also be overridden to
provide project-specific values. For instance, you will need to do this
if you are testing a compiler toolchain, because the default values do
not take into account host and target names.
The contents of the variable ‘RUNTESTFLAGS’ are passed to the
‘runtest’ invocation. This is considered a “user variable” (*note User
Variables::). If you need to set ‘runtest’ flags in ‘Makefile.am’, you
can use ‘AM_RUNTESTFLAGS’ instead.
Automake will generate rules to create a local ‘site.exp’ file,
defining various variables detected by ‘configure’. This file is
automatically read by DejaGnu. It is OK for the user of a package to
edit this file in order to tune the test suite. However this is not the
place where the test suite author should define new variables: this
should be done elsewhere in the real test suite code. Especially,
‘site.exp’ should not be distributed.
Still, if the package author has legitimate reasons to extend
‘site.exp’ at ‘make’ time, he can do so by defining the variable
‘EXTRA_DEJAGNU_SITE_CONFIG’; the files listed there will be considered
‘site.exp’ prerequisites, and their content will be appended to it (in
the same order in which they appear in ‘EXTRA_DEJAGNU_SITE_CONFIG’).
Note that files are not distributed by default.
For more information regarding DejaGnu test suites, see *note
(dejagnu)Top::.
15.6 Install Tests
The ‘installcheck’ target is available to the user as a way to run any
tests after the package has been installed. You can add tests to this
by writing an ‘installcheck-local’ rule.
