SWI Prolog 9.0.4 Linux: run_tests runs only the first file

General
1

I’m running SWI Prolog unit tests from the command line as per the documentation.

swipl -g run_tests -t halt <file1.plt> <file2.plt> ...

The documentation claims that multiple test files can be provided on the command line: “If you want to test multiple files, you can pass multiple …pl files.”

That’s not what I’m experiencing. SWI Prolog runs the tests in the first file but ignores the rest of them. I can work around this by running a script like below instead, but still, the reality doesn’t seem to match the documentation.

Is this a bug?

find . -name \*.plt | xargs swipl -g run_tests -t halt
2

Gentle ping. Any thoughts? Is this a bug, or is it intentional?

3

This thread was definitely not sooo successful… :sweat_smile: sorry, but you need the answer of one of the big ones :man_shrugging:

4

If someone can point me to the right source file or directory, I’ll be happy to take a look for myself.

5

Have you tried doing something like this:

:- module(test_cpp, [test_cpp/0]).

test_cpp :-
    run_tests([ cpp,
                cpp_atommap,
                cpp_map_str_str
	      ]).

and then specify the goal test_cpp.

(This is from package(cpp), hence the names)

In the past I’ve had only the first test run but couldn’t reproduce it. There might be some kind of race condition, to do with the order that modules get loaded (plunit does use reflection to figure what to run).

The sources for plunit are here: GitHub - SWI-Prolog/packages-plunit: The SWI-Prolog Unit Testing library

6

Thanks for the suggestion. Unfortunately, it’s undesirable, because it would mean an additional maintenance point for the team, and it has a severe failure mode: if one forgets to add their test file to this list, those tests won’t run. (Of course, one could forget to write the tests in the first place, but that is more easily caught in code reviews and less likely in the first place.)

I run the tests using swipl, so I think the culprit here is the swipl executable itself. Could you point me to the source code for that?

swipl -g run_tests -t halt <file1.plt> <file2.plt> ...
7

I changed the cpp test code to this (that is, removed the list of tests to run):

test_cpp :-
    run_tests.

and it ran all the tests using this:

swipl -g test_cpp -t halt cpp/test_cpp.pl

However, specifying the goal run_tests also worked.

I also tried running the tests from two packages, and that also worked:

swipl -g run_tests -t halt cpp/test_cpp.pl pcre/test_pcre.pl

swipl is a small executable that loads the main Prolog engine from libswipl.so. I don’t think your answer is there, but somewhere in plunit’s code that figures out which tests to run.

8

Thanks for troubleshooting with me, but please note that my issue is with test files, not tests. All the tests in the first test file are executed, none of the tests in the remaining files are. Does that change the direction of your troubleshooting?

9

It says you can have multiple *.pl files. The *.plt are designed as test files associated with normal Prolog files. The idea is that you can have some application and test the application using e.g.

  swipl -g 'load_test_files([]),run_tests' -t halt -l myprog.pl

If your test files are not necessarily linked to Prolog files, just name than *.pl and you can load them. Alternatively you can use -l f.plt -l f2.plt .... Except for a .plt file being related to a .pl file and loaded by load_test_files/1 if the matching .pl file is already loaded, it is just a normal Prolog file.

Most of the internal testing using files called test_xyz.pl that define the module test_xyz and export text_xyz/0. This may do whatever it wants, often calling run_tests/1 as @peter.ludemann shows.

10

Stringing multiple -l files together brings in all sorts of issues.

swipl -g run_tests -t halt -l component.plt -l component2.plt

ERROR:    load_files/2: No permission to load source `'component2.plt'' (Non-module file already loaded into module plunit_comms; trying to load into plunit_testing)

Warning: component2.plt:2:
Warning:    Goal (directive) failed: plunit_testing:ensure_loaded(testing)

And then a bunch of these:

test my_pred: received error: plunit_testing:'unit body'/2: Unknown procedure: plunit_testing:my_pred/2

My actual files (component.pl) are not yet modules. But the test files (component.plt) are.

Do I need to switch my files to modules? If yes, can I use the same module name in the test file and the component file? (I believe I tested this a couple of weeks ago and the answer was yes.)

11

You cannot load a non-module file into multiple modules. Well, you can do so using include/1, which replicates the code in all modules into which you include the file (and is typically not what you want).

I guess because the file-to-test is not loaded into the second module.

If you want a file loaded into multiple modules, yes. A good rule of thumb is to use modules everywhere or nowhere. A bit more subtle, a bunch of non-module files may load each other and load module files, but a bunch of module files cannot load the same non-module file twice. Except for very small playground files all my files are module files.

SWI-Prolog (and all Prolog systems AFAIK) have a flat module space, so no two files that are loaded into the same instance may define the same module. I’d use test_component, etc. for the test modules. As you do not use the association between .plt and .pl file, I’d create a directory test, put all my tests there as test_componentN.pl and now you can run

swipl -g run_tests -t halt test/test*.pl

P.s. Note that gxref/0 can be executed into the loaded program to reveal all dependencies of the non-modular program and generate the module headers (interactively).

12

I see, I will give this a try at some point and report back with any issues. Thank you for the tireless responses.