Update version of coverage analysis

Resources
1

I’ve pushed an update to the coverage analysis tool, show_coverage/1,2. Using the option list we now get a detailed annotated program back. For example (using the good old CHAT80 program, available as add-on chat80):

?- show_coverage(test_chat, [dir(cov)]).

Results in annotates files. A fragment is below, showing most of the possible annotations.

image
image754×794 104 KB

The relevant doc is

  • dir(+Dir)
    Dump the annotations in the given directory. If not given, the annotated files are created in the same directory as the source file. Each clause that is related to a physical line in the file is annotated with one of:

    Annotation Description
    ### Clause was never executed.
    ++N Clause was entered N times and always succeeded
    –N Clause was entered N times and never succeeded
    +N-M Clause has succeeded N times and failed M times
    +N*M Clause was entered N times and succeeded M times

    All call sites are annotated using the same conventions, except that --- is used to annotate subgoals that were never called.

One of the issues is that we monitor entering a clause (the SWI-Prolog unify port), the call port to find call sites (calls to this predicate) and the exit port to find successes. That tells us a predicate was called N times at a specific call site and exited M times from the same call site. The number of exits may exceed the number of calls due to non-determinism. Given the VM instrumentation distinguishing the first answer from alternative answers is not easy and basically unreliable :frowning: The result is though that a predicate may have been called 10 times, failed 5 times and produced 5 times two answers and we cannot distinguish this from it simply succeeding always. The other issue is performance. Running on top of the debugger callbacks makes the code rather slow. Eventually we should do something similar to the profiler and collect the data in C. The way it is done now is much simpler though and allows us to fine the desirable behavior.

Comments welcome.

4 Likes
2

The latest git version moved the low-level data collection to the core. This avoids the debugger callback and makes the bookkeeping a lot more efficient. Coverage analysis shares with the debugger to disable last-call optimization and turns calls to predicates that are normally handled inside the VM to normal predicate calls. This, together with the admin, causes a slow-down of about 2 times on tests using s(CASP). Note that disabling last-call optimization may make programs run out of memory that would normally run fine.

That is quite a bit better than the 20-50 times it used to be :slight_smile: Seems we can now claim to have a pretty much state-of-the-art coverage analysis tool!

3 Likes
3

Using SWI-Prolog version 9.1.21, looks like this library is now called “test_cover”? use_module(library(prolog_coverage)) fails, but use_module(library(test_cover)) works.

(Looks like the module name in the file /plunit/test_cover.pl should be updated too.)

And another question: what is a preferred editor/viewer for opening the “.cov” files?

4

The coverage stuff has been moved from the plunit extension to the core. The library was renamed from library(test_cover) to library(prolog_coverage). The predicates are largely compatible. They still accept the old common usage, but as deprecated behavior.

To the user this should all be transparent as show_coverage/1,2 is an autoload predicate, so they work without using use_module/1.

That is a bit of an issue. The files use ANSI terminal escapes for the colors. You can add color(false) to stop that. If you have the less utility, less -r typically does the job to show them with paging on the terminal. less is standard on Linux and MacOS. I guess there are Windows ports. Alternatively, there are a large number of utilities around that translate ansi color codes to HTML.

I guess it would be better to emit HTML … That could be combined with the highlighting HTML coloring library that is part of PlDoc. With a bit of JavaScript this could be turned into a nice tool :slight_smile: Anyone?