Update version of coverage analysis

I’ve pushed an update to the coverage analysis tool, show_coverage/1,2. Using the option list we now get a detailed annotated program back. For example (using the good old CHAT80 program, available as add-on chat80):

?- show_coverage(test_chat, [dir(cov)]).

Results in annotates files. A fragment is below, showing most of the possible annotations.

The relevant doc is

  • dir(+Dir)
    Dump the annotations in the given directory. If not given, the annotated files are created in the same directory as the source file. Each clause that is related to a physical line in the file is annotated with one of:

    Annotation Description
    ### Clause was never executed.
    ++N Clause was entered N times and always succeeded
    –N Clause was entered N times and never succeeded
    +N-M Clause has succeeded N times and failed M times
    +N*M Clause was entered N times and succeeded M times

    All call sites are annotated using the same conventions, except that --- is used to annotate subgoals that were never called.

One of the issues is that we monitor entering a clause (the SWI-Prolog unify port), the call port to find call sites (calls to this predicate) and the exit port to find successes. That tells us a predicate was called N times at a specific call site and exited M times from the same call site. The number of exits may exceed the number of calls due to non-determinism. Given the VM instrumentation distinguishing the first answer from alternative answers is not easy and basically unreliable :frowning: The result is though that a predicate may have been called 10 times, failed 5 times and produced 5 times two answers and we cannot distinguish this from it simply succeeding always. The other issue is performance. Running on top of the debugger callbacks makes the code rather slow. Eventually we should do something similar to the profiler and collect the data in C. The way it is done now is much simpler though and allows us to fine the desirable behavior.

Comments welcome.

3 Likes

The latest git version moved the low-level data collection to the core. This avoids the debugger callback and makes the bookkeeping a lot more efficient. Coverage analysis shares with the debugger to disable last-call optimization and turns calls to predicates that are normally handled inside the VM to normal predicate calls. This, together with the admin, causes a slow-down of about 2 times on tests using s(CASP). Note that disabling last-call optimization may make programs run out of memory that would normally run fine.

That is quite a bit better than the 20-50 times it used to be :slight_smile: Seems we can now claim to have a pretty much state-of-the-art coverage analysis tool!

2 Likes