Plunit and individual test setup/cleanup

Help! Tools
1

Is there any way to tell plunit to run a goal before every test or after every test? This is useful for tests that implicitly affect global state, for example when testing predicates that cache results (library(pcre) is one example, where some predicates cache intermediate results using assertz/1).

There’s a setup(Goal) option in begin_tests/2, but it runs before any of the tests. (Similarly, begin_tests/2 option cleanup(Goal) runs after all the tests.)
I want a way of specifying a goal to be run for each test, and not have to specify setup(Goal) for each test. (Some other language’s unit test frameworks have both global and per-test setup/cleanuyp; SWI-Prolog’s plunit only has global setup/cleanup.)

Do the tests run sequentially or can they run in parallel? I presume that they run sequentially, but this isn’t documented AFAICT. If I want to test that things work properly in a multi-threaded environment (e.g., that caching is done on a per-thread basis or globally), is there any test infrastructure to help this kind of test? I noticed that the code for library(plunit) has set_test_options([concurrent(true]) but this doesn’t appear in the documentation for set_test_options/1; also, it’s not clear how to specify these options – do I simply add :- set_test_options([concurrent(true]). to the file?

2

Individual tests have the same setup and cleanup options (which may share
variables with the test). For repetitive similar tests this may get a bit of verbose. Next to the forall construct in the test head you can term expand tests. Something like this:

term_expansion(re(...), (test(...) :- Body)) :-
    ...

re(...).

That makes your tests concise and creates nice independent tests that correctly report line numbers if one fails.

Tests are run sequentially. The overall SWI-Prolog test suite defines them in over 70 bundles that are executed in concurrent Prolog instances if you run ctest -j tasks. You can tweak a bit with the numbers to get the shortest total time depending on your hardware. On my desktop I use ctest -j 16 run the whole thing in 4.2 seconds.

The concurrent option was experimental. I don’t really recall the status. Not all system tests are designed to run concurrent, for example because they test whether all threads have died or whether atom garbage collection has been executed. Another problem is that output needs to be considered carefully not to make a mess.

1 Like
3

I suppose I could do something like this (untested):

term_expansion((re_test(Name) :- Body),
                  (test(Name, []) :- Body)).
term_expansion((re_test(Name, Options) :- Body), 
                  (test(Name, Options2) :- Body)) :-
    expand_options(Options, Options2).
expand_options([], Options2) =>
    Options2 = [setup(my_setup)].
expand_options([O1|O2], Options2) =>
    Options2 = [setup(my_setup),O1|O2].
expand_options(Option, Options2) => % e.g.: re_test(t123, X==[1,2,3]) :- pred(123, X).
    Options2 = [setup(my_setup),Option].

Have I forgotten a pattern? TBH, it’s easier to just add the my_setup goal when I write each test, or add them retroactively using a simple Emacs keyboard macro.

But it would be nicer if this were done for me as part of plunit:expand_test/4, by just writing something like begin_tests(my_tests, test_setup(my_setup)]) .

4

Term expansion is not applied recursively, so the first clause will not work.

You can do that, but it means something different. It runs the setup and cleanup once for the entire block of tests, not for each individual test.

Just term expanding to get a setup handler is probably not worthwhile. The term expansion trick is mostly useful if you have a lot of tests that are closely related such that you can write them in some dedicated concise syntax.

5

begin_tests(my_tests,setup(my_setup)]) behaves the way you describe. I was suggesting new option test_setup to begin_tests/2.

FWIW, other test frameworks I’ve used (in other languages) allow specifying both global setup/cleanup (done once for a set of tests) and every-test setup/cleanup – which is why I’m suggesting adding this to plunit.

6

BTW, the term_expansion I gave isn’t quite right – it needs to look for any setup option and combine with its goal. (It appears that specifying setup multiple times results in only the first one being used and the others are ignored.)

7

Thanks for pointing that out – which shows how tricky it can be to get even simple term expansion right. (I got confused with goal_expansion, which does apply recursively to a fix-point)

8

Have you tried using a wrapper?

I remember originally noting this with wrappers but IIRC a Prolog wrapper can not pass stateful information from the before predicate to the after predicate. Never checked if that was allowed or not.

I also have to agree that this is often the case

other test frameworks allow specifying both global setup/cleanup and every-test setup/cleanup

9

Using a wrapper would work but requires knowing the details of how test/2 is expanded and could break if details of plunit change in the future.

As it happens, plunit term expansion produces two clauses: 'unit test'/4 and `‘unit body’/2, which are “interpreted” by the plunit framework.
E.g.:

test(match1, [Sub == re_match{0:"aap"}]) :-
    re_compile("a+p", Re, []),
    re_matchsub(Re, "aapenootjes", Sub, []).

expands to this (inside :-begin_tests(pcre)):

plunit_pcre:'unit test'(match1,
                        76,
                        [true(A==re_match{0:"aap"})],
                        plunit_pcre:'unit body'('match1@line 76', vars(A))).

plunit_pcre:'unit body'('match1@line 76', vars(A)) :-
    !,
    re_compile("a+p", B, []),
    re_matchsub(B, "aapenootjes", A, []).
1 Like