# Enhancements to plunit **URL:** \ **Category:** Discussion\ **Tags:** software\ **Created:** [June 3, 2022, 7:11pm UTC](https://swi-prolog.discourse.group/t/enhancements-to-plunit/5459 "2022-06-03T19:11:44Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![peter.ludemann](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/peter.ludemann/32/48_2.png) [@peter.ludemann](https://swi-prolog.discourse.group/u/peter.ludemann)\ **Post date:** [June 3, 2022, 7:11pm UTC](https://swi-prolog.discourse.group/t/enhancements-to-plunit/5459/1 "2022-06-03T19:11:44Z")
I’m thinking about enhancing [library(plunit)](https://www.swi-prolog.org/pldoc/doc_for?object=section(%27packages/plunit.html%27)) in a few ways … what do you think of these possibilities? (And please add suggestions of your own) The changes I’m proposing appear to be easy to add, but I might have read the code too optimistically. At the level of a test suite: - `random(Bool)` - if `true`, the tests are run in a random order. (I want this because I’m seeing some memory-access errors in foreign functions that depend on the order that blobs are garbage collected). If any tests fail, this would also output the ordering of the tests (see option `order`). - `order(List)` - a list of test names, causing the tests to run in that specific order rather than in sequential order (the default) or random order (if `random(true)` is specified). - ~~`repeat(Number)` - runs the entire test suite this many times (default is 1) or until first test failure.~~ For individual tests, this can be done by the option `forall(between(1,N,_))`. Repeated execution until a failure (for finding rare but flakey tests) can be done by `repeat, \+ run_tests`. - `test_setup(Goal)` and `test_cleanup(Goal)` - adds a `setup(Goal)` and `cleanup(Goal)` to each test within a test suite - the test suite’s `test_setup` runs before the individual test’s `setup` (if any) and the test suite’s `test_cleanup` runs after the individual test’s `cleanup` (if any). (Currently, I use term-expansion to do this, which is a bit ugly.) As an option to set\_test\_options/1: - `verbose(Bool)` - if `true`, output a message for each test, not just the ones that fail. This is useful when adding some print debug to code. Note that there is already a predicate `set_test_options(silen(true))`. For an individual test: - multiple `true` options. Currently, if I have two things I want to check, my choices are: ```prolog test(a_test) :- do_something(A, B), assertion(A == expected_value_A), assertion(B == expected_value_B). ``` or ```prolog test(a_test, [A,B] == [expected_value_A, expected_value_B]) :- do_something(A, B). ``` and I’m proposing to allow this: ```prolog test(a_test, [A == expected_value_A, B == expectedValue_B]) :- do_something(A, B). ``` _EDITS_ - option `verbose` for test suite - option `order` for test suite - changed `multiple` to `repeat` and clarified what happens when test fails - added comment to `repeat` and removed it. ---
**Author:** ![peter.ludemann](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/peter.ludemann/32/48_2.png) [@peter.ludemann](https://swi-prolog.discourse.group/u/peter.ludemann)\ **Post date:** [June 3, 2022, 8:24pm UTC](https://swi-prolog.discourse.group/t/enhancements-to-plunit/5459/2 "2022-06-03T20:24:37Z")
For those who use Discourse via email, I’ll be editing my original post as ideas occur to me (or are suggested by others). ---
**Author:** ![jan](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/jan/32/4_2.png) [@jan](https://swi-prolog.discourse.group/u/jan)\ **Post date:** [June 7, 2022, 1:04pm UTC](https://swi-prolog.discourse.group/t/enhancements-to-plunit/5459/3 "2022-06-07T13:04:13Z")
> [@peter.ludemann](#): > > - `random(Bool)` - if `true`, the tests are run in a random order. (I want this because I’m seeing some memory-access errors in foreign functions that depend on the order that blobs are garbage collected). If any tests fail, this would also output the ordering of the tests (see option `order`). > - `order(List)` - a list of test names, causing the tests to run in that specific order rather than in sequential order (the default) or random order (if `random(true)` is specified). Some other tests print the random seed in case of errors (they first explicitly seed the random generator using set\_random/1. Some tests also produce random amounts of garbage (atoms) in situations like this. As explained offline, this is not going to solve the library(archive) as the randomness is based on the IO stream table. > [@peter.ludemann](#): > > `repeat(Number)` - runs the entire test suite this many times (default is 1) or until first test failure. Note that for individual tests, this can be done by the option `forall(between(1,N,_))`. I typically solve that using ``` ?- repeat, \+ run_tests(Spec). ``` > [@peter.ludemann](#): > > `test_setup(Goal)` and `test_cleanup(Goal)` - adds a `setup(Goal)` and `cleanup(Goal)` … That surely covers a common use case. > [@peter.ludemann](#): > > `verbose(Bool)` - if `true`, output a message for each test, not just the ones that fail. This is useful when adding some print debug to code. Note that there is already a predicate `set_test_options(silent(true))`. I think that should be part of `set_test_options/1` instead of the test suite. It is something one may want to easily figure out what happens in each test without counting the dots 🙂 ---
**Author:** ![peter.ludemann](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/peter.ludemann/32/48_2.png) [@peter.ludemann](https://swi-prolog.discourse.group/u/peter.ludemann)\ **Post date:** [June 7, 2022, 2:47pm UTC](https://swi-prolog.discourse.group/t/enhancements-to-plunit/5459/4 "2022-06-07T14:47:10Z")
> [@jan](#): > > Some other tests print the random seed in case of errors (they first explicitly seed the random generator using [set\_random/1](https://www.swi-prolog.org/pldoc/doc_for?object=set_random/1). Outputting the random state when running the tests in random order is a good idea – for rare bugs, this would help in reproducing the bug. (I almost always find that I’ve forgotten to output something that helps reproduce a rare bug.) > [@jan](#): > > ```prolog > ?- repeat, \+ run_tests(Spec). > > ``` Nice trick! (It took me a bit of thinking to realize that it meant “re-run the test suite until a test fails”.) > [@jan](#): > > this is not going to solve the library(archive) as the randomness is based on the IO stream table. True; but if I had run the tests in random order, I would have found the bug sooner. > [@jan](#): > > I think that [`verbose(Bool)`] should be part of `set_test_options/1` instead of the test suite. I’ve edited my original post accordingly. ---
**Author:** ![marcoxa](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/marcoxa/32/78_2.png) [@marcoxa](https://swi-prolog.discourse.group/u/marcoxa)\ **Post date:** [March 10, 2024, 3:15pm UTC](https://swi-prolog.discourse.group/t/enhancements-to-plunit/5459/5 "2024-03-10T15:15:00Z")
Hi here I am for my yearly foray in the `PLUnit` library. I like @peter.ludemann proposal, and I like the recent (last year?) addition of the `summary(Report)` option, which does something like what I previously proposed. I have a few things I’d like to see in the library, which, every year I use it, shows some inconsistent behavior, which I do not understand. # Output - Output messages in `setup` options do not appear anywhere. - Sometimes, output messages in `test` body are dropped inexplicably. If there is a special stream/log to send output, it is not documented. # Test Dependencies I would like to run some tests only if some previous one succeeded. There is no apparent way to list such dependencies. If a test is not run, it should be recorded as `not_run` or even `failed`. Something along the lines of ```prolog test(first_test) :- ... test(second_test, [depends_on([first_test])]) :- ... ``` # Timeouts From the current documentation it is not clear whether `timeout` clauses can be used with `begin_tests` and/or `test`. I use it with `set_test_options/1` but I wonder if there is a more fine grained control over it. In any case, `timeout`s still fail in unfathomable ways is some cases. if I remember correctly it is because they do not operate as interrupts or the interrupt handlers do not really run asynchronously, but need to be run only in specific points in the code. If these could be fixed (at least the `timeout`), I’d be a happy camper. All the best Marco ---
**Author:** ![jan](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/jan/32/4_2.png) [@jan](https://swi-prolog.discourse.group/u/jan)\ **Post date:** [March 10, 2024, 4:45pm UTC](https://swi-prolog.discourse.group/t/enhancements-to-plunit/5459/6 "2024-03-10T16:45:45Z")
Thanks for the feedback > [@marcoxa](#): > > I like @peter.ludemann proposal, and I like the recent (last year?) addition of the `summary(Report)` option, which does something like what I previously proposed. I don’t have this clear in my mind. There is a summary that gives the counts for the various possible results of the tests. Would you like a list of the failed tests at the end, so you do not need to scroll back? Something else? > [@marcoxa](#): > > Output messages in `setup` options do not appear anywhere. That is a bug. Like normal messages they should by default be shown if something fails. > [@marcoxa](#): > > Sometimes, output messages in `test` body are dropped inexplicably By default output of succeeding tests is hidden. You can enable output using the `output(always)` option, which also shows the output of setup code. This is similar (and inspired by `ctest`). > [@marcoxa](#): > > `test(second_test, [depends_on([first_test])]) :-` Why do you want this? Because the next test will fail as well? Or because you want to use the side effects of the first test? In the first case it might be better to offer an option to abort the block as soon as some test fails. Making tests depend on each other for side-effects smells dubious to me. Note that if you want to test progress inside the test, you can use assertion/1 as ```prolog test(...) :- do some work assertion(X==1), do some more assertion(Y == 3), ... ``` > [@marcoxa](#): > > From the current documentation it is not clear whether `timeout` clauses can be used with `begin_tests` and/or `test`. I use it with `set_test_options/1` but I wonder if there is a more fine grained control over it. As all options where it makes sense, there is a cascade. First we look at the test itself, than the block in which it appears, then the global options and finally defaults. So, yes, you can add timeout(Time) to a test or a block (making it effective for each test in the block, i.e., _not_ the test time for the entire block). > [@marcoxa](#): > > if I remember correctly it is because they do not operate as interrupts or the interrupt handlers do not really run asynchronously, but need to be run only in specific points in the code. They run as interrupts, but SWI-Prolog interrupts are _synchronous_. Normal Prolog code should act quickly, but foreign code needs to call `PL_handle_signals()` and return with failure if this returns `-1`. If foreign code does not do so, no processes are handled. Then there are blocking system calls. Most handle interrupts, but it depends on the OS and involved system call. ---
**Author:** ![jan](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/jan/32/4_2.png) [@jan](https://swi-prolog.discourse.group/u/jan)\ **Post date:** [March 13, 2024, 5:30pm UTC](https://swi-prolog.discourse.group/t/enhancements-to-plunit/5459/7 "2024-03-13T17:30:54Z")
> [@marcoxa](#): > > Output messages in `setup` options do not appear anywhere. Pushed a fix for that, so we get e.g. ```prolog [2/2] setup:nosetup .................................. **FAILED (0.000 sec) ERROR: /home/janw/src/prolog/plunit/test_setup.pl:30: ERROR: test setup:nosetup: test setup goal raised error: //2: Arithmetic: evaluation error: `zero_divisor' This is setup ``` The test setup prints `This is setup` and divides by zero.