It’s a long story, but I’m keen to find a programmatic way way to see what parts of a knowledge base were used while proving a query: like using trace but for programmatic access (via pyswip). For example with the rules:
b(X):-a(X).
a(x).
earth_spins.
When proving b(x) I would like to know that the first two rules where used but not the third.
Thanks greatly,
Rob
(I’ve gone as far as whipping up a toy Prolog in Python that does this, but its spiralling into a time sink!)
See the following blog post:
https://logtalk.org/2019/12/02/generating-code-coverage-reports.html
By simply wrapping your clauses in an object and defining your goals as tests, you get a report (see the example linked in the blog post) detailing which clauses (by index) were used when proving the goals. Using your example database:
:- object(clauses).
:- public(b/1).
b(X) :- a(X).
a(x).
earth_spins.
:- end_object.
:- object(tests,
extends(lgtunit)).
cover(clauses).
test(t1) :-
clauses::b(_).
:- end_object.
Using a driver file for the tests:
:- initialization((
set_logtalk_flag(report, warnings),
logtalk_load(lgtunit(loader)),
logtalk_load(clauses, [source_data(on), debug(on)]),
logtalk_load(tests, [hook(lgtunit)]),
tests::run
)).
We get:
$ swilgt
...
?- {tester}.
%
% tests started at 2020-02-14, 17:41:43
%
% running tests from object tests
% file: /Users/pmoura/Desktop/cc/tests.lgt
%
% t1: success
%
% 1 tests: 0 skipped, 1 passed, 0 failed
% completed tests from object tests
%
%
% clause coverage ratio and covered clauses per entity predicate
%
% clauses: a/1 - 1/1 - (all)
% clauses: b/1 - 1/1 - (all)
% clauses: earth_spins/0 - 0/1 - []
% clauses: 2 out of 3 clauses covered, 66.666667% coverage
%
% 1 entity declared as covered containing 3 clauses
% 1 out of 1 entity covered, 100.000000% entity coverage
% 2 out of 3 clauses covered, 66.666667% clause coverage
%
% tests ended at 2020-02-14, 17:41:43
%
true.
The printed data can be intercepted if you want to make the coverage data available for further processing.
Thanks for very quick help @pmoura. Will try this out.
I haven’t tried Logtalk yet. Will I be able to call it from pyswip? I’m assuming not? If not, is there a similar approach in swiprolog without Logtalk?
Either way looking forward to trying this out – Rob
Should not be an issue. The most simple route could be to install its pack and then load it at startup. The queries are:
?- pack_install(logtalk).
...
?- use_module(library(logtalk)).
Brilliant, thank you! Will report back. – Rob
Note that if you already have it installed, you might need to do this to get the latest version:
?- pack_remove(logtalk).
?- pack_install(logtalk).
The upgrade option for pack_install has some limitations, so pack_install(logtalk, [update(true)]) might not do what you want.
If you want to programmatically manage goals within prolog, allowing you to print and even control their execution , you may want to write what is called a meta-interpreter. It is not complicated, but you need to wrap your brain around the concepts.
See https://www.metalevel.at/acomip/ for examples and information.
And, native SWI-Prolog, there is show_coverage/1,2 which runs the debugger in non-interactive mode to obtain these details. It is all written in Prolog and you can use the library implementation as a basis to do exactly what you want.
How? With a ccpl.pl Prolog file with the three clauses in the original question, we get:
?- [ccpl].
true.
?- use_module(library(test_cover)).
true.
?- show_coverage(b(X)).
==============================================================================
Coverage by File
==============================================================================
File Clauses %Cov %Fail
==============================================================================
/Users/pmoura/Desktop/ccpl.pl 3 66.7 0.0
/Users/pmoura/lib/swipl/library/test_cover.pl 32 6.2 3.1
==============================================================================
X = x.
But the OP asked about which clauses are used and which are not used, not about which percentage of the total clauses are used (information that lgtunit code coverage provides as illustrated in my answer). Am I missing something? The SWI-Prolog test_cover library documentation states “this library currently only shows some rough quality measure for test-suite. Later versions should provide a report to the developer identifying which clauses are covered, not covered and always failed.”
I can find nothing for show_coverage/2, no documentation on it in the docs or source code in the main repository at GitHub.
True. Considering it can give a percentage one can derive the implementation does collect the used clauses. Only the reporting abstracts this to a percentage. As the OP wants programmatic access, he must do a little work. The code tracks the unify and exit ports and gets the involved clause. From there it is really easy.
The code has docs. The online docs come from LaTeX though. Added.
%! show_coverage(:Goal) is semidet.
%! show_coverage(:Goal, +Modules:list(atom)) is semidet.
%
% Report on coverage by Goal. Goal is executed as in once/1. Report
% the details of the uncovered clauses for each module in the list
% Modules
EDIT by EricGT
test_cover.pl – Clause cover analysis
Located by using SWI-Prolog documentation search
with keyword show_coverage which will then present the options as a drop down.
Indeed. As I wrote in my initial reply:
The printed data can be intercepted if you want to make the coverage data available for further processing.
In my solution, the message to be intercepted is:
entity_predicate_coverage(Entity,Predicate,Covered,Total,Percentage,Clauses)
The interception code would be something like:
:- category(code_coverage_interception).
:- multifile(logtalk::message_hook/4).
:- dynamic(logtalk::message_hook/4).
logtalk::message_hook(entity_predicate_coverage(Entity,Predicate,Covered,Total,Percentage, Clauses), _, lgtunit, _) :-
% process the coverage data
...
:- end_cateogry.
Thanks for the ongoing discussion guys! Am away from my actual computer but have been fiddling with swish a bit. There are some good options above that would work. Alternatively…
Access to the trace of a solution would probably include everything I need and details that might be useful later too. From tracehook I spotted this magic:
prolog_trace_interception(Port, Frame, _PC, continue) :-
prolog_frame_attribute(Frame, goal, Goal),
prolog_frame_attribute(Frame, level, Level),
recordz(trace, trace(Port, Level, Goal)).
Not sure it’s working (when called with trace on) inside SWISH as replacing continue with skip has no effect. Also recorded reports being sandboxed. Will try when I have access to proper computer.
My apologies for asking the question when I can’t try things out properly and reply!. —rob
At first glance, I think the following solves my problem,
kb.pl:
prolog_trace_interception(Port, Frame, _PC, continue) :-
prolog_frame_attribute(Frame, goal, Goal),
prolog_frame_attribute(Frame, level, Level),
recordz(trace, trace(Port, Level, Goal)).
c(X):-b(X).
b(X):-a(X).
a(x).
earth_spins.
Call:
?- trace, c(x), notrace.
true.
Read the trace:
?- recorded(trace, Value, _).
Value = trace(call, 9, c(x)) ;
Value = trace(call, 10, b(x)) ;
Value = trace(call, 11, a(x)) ;
Value = trace(exit, 11, a(x)) ;
Value = trace(exit, 10, b(x)) ;
Value = trace(exit, 9, c(x)).
And clear for the next run:
?- recorded(trace, _, Reference), erase(Reference).
Reference = <record>(0x7fafabfa3f20) ;
Reference = <record>(0x7fafabfa3f80) ;
Reference = <record>(0x7fafabfa3fe0) ;
Reference = <record>(0x7fafabfa4040) ;
Reference = <record>(0x7fafabfa40a0) ;
Reference = <record>(0x7fafabfa4100).
Just need to wrap this up after reading about meta-predicates! Also will need to process the resulting tree. Thanks for your help guys, will try out some of the other solutions above when I get a chance.
Thanks, this looks super interesting.
Forgot to mention, if your clauses are in a plain Prolog file, e.g. clauses.pl, no need to change it. You can simply use:
:- object(clauses).
:- public(b/1).
:- include('clauses.pl').
:- end_object.Thanks @pmoura. My application (at least unless I go full prolog which might happen yet!) is calling out from python into prolog do some reasoning; so I’m more likely to be loading and unloading clauses into the db and making calls into it than loading from a file.
I may end up using modules as namespaces in which case Logtalk objects might be a better option, but am not there yet! Also, I’m guessing that running with trace on might slow things down giving me another reason to switch to the Logtalk solution if performance becomes an issue.
Thanks for the awesome support!
P.S. I wrapped up the trace recording and reading as record_trace and pop_trace_recording :
prolog_trace_interception(Port, Frame, _PC, continue) :-
prolog_frame_attribute(Frame, goal, Goal),
prolog_frame_attribute(Frame, level, Level),
recordz(trace, trace(Port, Level, Goal)).
:- meta_predicate record_trace(?).
record_trace(Goal):- trace, Goal, notrace.
pop_trace_recording(Value):-
recorded(trace, Value, Reference), erase(Reference).
With kb:
b(X):-d(X).
b(X):-a(X).
a(x).
d(y).
earth_spins.
Query with:
[debug] ?- record_trace(b(x)).
true.
[debug] ?- pop_trace_recording(V).
V = trace(call, 9, b(x)) ;
V = trace(call, 10, d(x)) ;
V = trace(fail, 10, d(x)) ;
V = trace(redo(0), 9, b(x)) ;
V = trace(call, 10, a(x)) ;
V = trace(exit, 10, a(x)) ;
V = trace(exit, 9, b(x)).Thanks @jan. Had a good look in test_cover.pl .
In the solution I suggested, that would mean dynamically creating an object instead. For example:
?- create_object(
clauses,
[],
[set_logtalk_flag(source_data,on), set_logtalk_flag(debug,on), public(b/1)],
[(b(X) :- a(X)), a(x), earth_spins]
).
true.
We still get the indexes of the clauses that are used in a query but not the total number of the clauses (as the predicate are dynamic):
% clause coverage ratio and covered clauses per entity predicate
%
% clauses: a/1 - 1/0 - [1]
% clauses: b/1 - 1/0 - [1]
% clauses: earth_spins/0 - 0/0 - (all)
% clauses: 2 out of 2 clauses covered, 100.000000% coverage
Unloading would mean either retracting the clauses in the dynamic object and asserting the new ones or simply calling abolish_object/1 and recreating it as above.
The solutions suggested by me and Jan use trace/debugging events that are intercepted to collect the data. Their implementation is quite different, however (the Logtalk one is portable while the SWI-Prolog one is native, working at a lower level). Both solutions come with a performance overhead. Still, I would expect both to be faster than a meta-interpreter solution that was also suggested.