Little testing tip

Nice to know
1

From a conversation with @pmoura elsewhere, it occurs to me that this is something that is widely-known, yet widely not followed.

When testing we should be favouring ==/2 over =/2 or binding in the head.

Some examples. Assuming that foo/1 asserts something and bar/1 extracts it (like a CHR constraint or some configuration setting):

my_test :- TestValue = 100, foo(TestValue), bar(TestValue).

This seems pretty elegant on the face of it, but if bar/1 fails to bind something in its body, the test will still succeed, even though it’s likely it shouldn’t. This is, of course, the implicit version of this code:

my_test :- TestValue = 100, foo(TestValue), bar(TestValueOut), TestValue = TestValueOut.

When this is written thusly it’s a bit more obvious what the little hole can be because it’s more visible what happens if bar/1 fails to bind something. In most cases the correct way to code this test is:

my_test :- TestValue = 100, foo(TestValue), bar(TestValueOut), TestValue == TestValueOut.

It turns out that unification isn’t always the right test. Who’d’ve thunk it?

This may sound “obvious”, and it is. But in the context of the conversation it was pointed out how often people make this very mistake. (@pmoura had been talking about his discoveries as he coded his linter.) And indeed in the course of that conversation I’d awoken to the horrible knowledge that not only had I fallen for this mistake, I’d moved from code equivalent to the third test to something equivalent to the first test because it was so damned “elegant”.

Elegance is nice, but it can be our enemy. Double-check your tests.

8 Likes
2

It is obliquely mentioned in the documentation; but not easy to find:
https://www.swi-prolog.org/pldoc/man?section=unitbox
“Cmp is typically one of =/2, ==/2, =:=/2 or =@=/2, …” and most of the examples use ==/2.

3 Likes
3

Not long ago I got hit by the exact same thinko. I even wrote it up, but did not spell out the underlying problem as you did.

My take-home message from it was that those are two distinct test cases and they might deserve two separate unit tests.

Test case 1: the “out” argument has the expected value.

Test case 2: the “out” argument is not a free variable, or is ground, or whatever it is you really expect from it.

Your suggestion is to make sure that the one test case you write covers both cases, which is a valid suggestion.

1 Like
4

Well, my specific suggestion is to make sure you know what you’re actually testing. :smiley:

In my specific test scenario (testing the value of registers in a simulated machine) a free variable as output from a register is nonsensical and should not ever pass, hence my not even bothering with a free test and a bound test. But in the move to something “elegant” I lost that vital feature without noticing because I was unifying in the head which is semantically identical to using =/2, but doesn’t LOOK that way so it didn’t hit my alarm bells.

If you have a valid use case for a free variable in the “out” side of the column, then of course the test case would be different.

1 Like
5

Another predicate that can come handy is subsumes_term/2, notably for verifying complex terms such as error terms where you typically only want to make sure part of the structure has a given shape and do not care about e.g., the second argument of the error(Formal,Context) term. In most cases =@= (variant) has my preference or == if the output should contain some specific variable.

5 Likes