I’m thinking about enhancing library(plunit) in a few ways … what do you think of these possibilities? (And please add suggestions of your own)
The changes I’m proposing appear to be easy to add, but I might have read the code too optimistically.
At the level of a test suite:
random(Bool) - if true, the tests are run in a random order. (I want this because I’m seeing some memory-access errors in foreign functions that depend on the order that blobs are garbage collected). If any tests fail, this would also output the ordering of the tests (see option order).
order(List) - a list of test names, causing the tests to run in that specific order rather than in sequential order (the default) or random order (if random(true) is specified).
repeat(Number) - runs the entire test suite this many times (default is 1) or until first test failure. For individual tests, this can be done by the option forall(between(1,N,_)). Repeated execution until a failure (for finding rare but flakey tests) can be done by repeat, \+ run_tests.
test_setup(Goal) and test_cleanup(Goal) - adds a setup(Goal) and cleanup(Goal) to each test within a test suite - the test suite’s test_setup runs before the individual test’s setup (if any) and the test suite’s test_cleanup runs after the individual test’s cleanup (if any). (Currently, I use term-expansion to do this, which is a bit ugly.)
As an option to set_test_options/1:
verbose(Bool) - if true, output a message for each test, not just the ones that fail. This is useful when adding some print debug to code. Note that there is already a predicate set_test_options(silen(true)).
For an individual test:
multiple true options. Currently, if I have two things I want to check, my choices are:
Some other tests print the random seed in case of errors (they first explicitly seed the random generator using set_random/1. Some tests also produce random amounts of garbage (atoms) in situations like this. As explained offline, this is not going to solve the library(archive) as the randomness is based on the IO stream table.
I typically solve that using
?- repeat, \+ run_tests(Spec).
That surely covers a common use case.
I think that should be part of set_test_options/1 instead of the test suite. It is something one may want to easily figure out what happens in each test without counting the dots
Outputting the random state when running the tests in random order is a good idea – for rare bugs, this would help in reproducing the bug. (I almost always find that I’ve forgotten to output something that helps reproduce a rare bug.)
Nice trick! (It took me a bit of thinking to realize that it meant “re-run the test suite until a test fails”.)
True; but if I had run the tests in random order, I would have found the bug sooner.
here I am for my yearly foray in the PLUnit library.
I like @peter.ludemann proposal, and I like the recent (last year?) addition of the summary(Report) option, which does something like what I previously proposed.
I have a few things I’d like to see in the library, which, every year I use it, shows some inconsistent behavior, which I do not understand.
Output
Output messages in setup options do not appear anywhere.
Sometimes, output messages in test body are dropped inexplicably.
If there is a special stream/log to send output, it is not documented.
Test Dependencies
I would like to run some tests only if some previous one succeeded. There is no apparent way to list such dependencies. If a test is not run, it should be recorded as not_run or even failed. Something along the lines of
From the current documentation it is not clear whether timeout clauses can be used with begin_tests and/or test. I use it with set_test_options/1 but I wonder if there is a more fine grained control over it.
In any case, timeouts still fail in unfathomable ways is some cases. if I remember correctly it is because they do not operate as interrupts or the interrupt handlers do not really run asynchronously, but need to be run only in specific points in the code.
If these could be fixed (at least the timeout), I’d be a happy camper.
I don’t have this clear in my mind. There is a summary that gives the counts for the various possible results of the tests. Would you like a list of the failed tests at the end, so you do not need to scroll back? Something else?
That is a bug. Like normal messages they should by default be shown if something fails.
By default output of succeeding tests is hidden. You can enable output using the output(always) option, which also shows the output of setup code. This is similar (and inspired by ctest).
Why do you want this? Because the next test will fail as well? Or because you want to use the side effects of the first test? In the first case it might be better to offer an option to abort the block as soon as some test fails. Making tests depend on each other for side-effects smells dubious to me. Note that if you want to test progress inside the test, you can use assertion/1 as
test(...) :-
do some work
assertion(X==1),
do some more
assertion(Y == 3),
...
As all options where it makes sense, there is a cascade. First we look at the test itself, than the block in which it appears, then the global options and finally defaults. So, yes, you can add timeout(Time) to a test or a block (making it effective for each test in the block, i.e., not the test time for the entire block).
They run as interrupts, but SWI-Prolog interrupts are synchronous. Normal Prolog code should act quickly, but foreign code needs to call PL_handle_signals() and return with failure if this returns -1. If foreign code does not do so, no processes are handled. Then there are blocking system calls. Most handle interrupts, but it depends on the OS and involved system call.
