Plunit fixme and assertion/1

Help! Tools
1

When I specify fixme in a unit test, it appears that calls to assertion/1 are recorded as successful even when they fail. Is this a bug or a feature?

Here’s an example of two tests that differ only in having calls to assertion/1:

test(cached_compile_1b, fixme(issue7)) :- 
    re_flush,
    assertion(   re_match('b', "ABC", [caseless(true)])),
    assertion(\+ re_match('b', "ABC", [caseless(false)])).
test(cached_compile_1bx, fixme(issue7)) :- % as cached_compile_1b but without the assertions.
    re_flush,
                 re_match('b', "ABC", [caseless(true)]),
              \+ re_match('b', "ABC", [caseless(false)]).

The output shows the error messages for both of these:

ERROR: /home/peter/src/swipl-devel/packages/pcre/test_pcre.pl:401:
	test cached_compile_1b: assertion failed
	Assertion: plunit_pcre:re_match(b,"ABC",[caseless(true)])
+-
ERROR: /home/peter/src/swipl-devel/packages/pcre/test_pcre.pl:409:
	test cached_compile_1c: assertion failed
	Assertion: plunit_pcre:re_match(b,"ABC",[caseless(true)])

but the overall report shows something different (also the summary report counts these FIXME tests as passing):

% FIXME: /home/peter/src/swipl-devel/packages/pcre/test_pcre.pl:405: issue7
% FIXME: /home/peter/src/swipl-devel/packages/pcre/test_pcre.pl:409: passed issue7
2

Looks like assertions are not (always) correctly handled with the fixme option. Passing is intentional. The fixme and blocked options are similar and mean: “I won’t fix this now, but I want to be remembered”. Here the fixme runs the test and the blocked does not try to run the test. The status approach doesn’t really really work for the system tests as we use the --on-error=status option that still causes the test to be reported as a failure.

So, for releases fixme tests are not allowed. blocked tests are ok AFAIK.