Ways to segregate test from code, feedback desired

Once code gets sizable test cases are often added.

During the early stages of development, it might be beneficial to put the test cases after the code using :- begin_tests(<Name>). and :- end_tests(<Name>).

When preparing the code for an initial release moving the test cases into a separate file is common.

Currently I know of two common ways to do this with Prolog.

  1. Use a plt file type. (ref)
  2. Wrap the test cases with a module and add a predicate to call run_tests/1. This is prevelent in the SWI-Prolog Tests directory on GitHub

Example code (ref)

:- module(test_list,
	  [ test_list/0
	  ]).
:- use_module(library(plunit)).
:- use_module(library(terms)).
:- use_module(library(apply)).

test_list :-
	run_tests([ memberchk
		  ]).

:- begin_tests(memberchk, []).

test(memberchk, X == y) :-		% Verify unbinding after failing PL_unify()
	memberchk(f(X,a), [f(x,b), f(y,a)]).

:- end_tests(memberchk).

I suspect that wrapping the test cases with a module has a side benefit for automation and continuous integration; I am not sure of the details of how the test cases are activated during the CI check.

The way it is done with SWI-Prolog on GitHub is the way I plan to go unless there is some serious problem with that pattern.

Just looking for ideas and pros and cons of the various ways for publishing code with test cases before I am deeply down a path that I don’t want to have to spend time redoing if I choose wrong.

EDIT

After trying both ways and reading Controlling the test suite in the documentation, am considering that using plt files is the better way to go, but as Jan W. mentions

Possibly we should have a library that implements this running all tests from some directory.

which should be relatively easy to implement.

Granted it will quite some time before I have so much code to publish for one project that it needs multiple directories.


Two specific reasons for using plt files

The test files can be easily loaded if

:- load_test_files([]).

is added to the source code file for the predicates.

Then when the source code file for the predicates is consulted, the test are automatically loaded and then run_tests/0 works the same as if the test were in the same file as the the predicates, e.g.

Welcome to SWI-Prolog (threaded, 64 bits, version 8.3.4-19-gefb65916d)

?- working_directory(_,'C:/Users/Groot/Documents').
true.

?- [rfc5234].
true.

?- run_tests.
% PL-Unit: rfc5234 ...... done
% All 6 tests passed
true.

While I have not tried this, this should apply

set_test_options(+Options)

Option: load(+Load)

Finally, when using the default value normal , tests are loaded if the code is not compiled with optimisation turned on.


One reason why it is organized as is is that there was a time plunit did not exist. There has a time before that that there was a single test file, which still exists as src/test.pl. As that became a mess, I started adding subdirectories with test files. The simple automation was:

  • A test file is a file XYZ,pl (nowadays typically test_XYZ.pl)
  • It defines a module equal to the file base name
  • exports a predicate with the same name and arity 0
  • The test succeeds iff this predicate succeeds.

Combining tests in a module allows to load and run them in the same Prolog process, which seriously improves the testing performance as you need to start only one process. If there is unexpected interaction is typically causes more tests to fail, which is good as well.

Then plunit came about and most new test files used the structure you describe. I’m still fairly happy with that. It is easy to add files using no or some different test driver. Currently the directories are individually declared as ctest units, so testing proceeds concurrently. Possibly we should have a library that implements this running all tests from some directory.

1 Like