Does PLDoc have a way to incorporate unit tests akin to Python's doctest?

A few years ago when I was helping others learn Python I used Runestone Academy which hosts interactive text books for learning Python among other things. The site had many more types of widgets than I have seen in any other type of online textbook. Again I am not asking that these be added to SWISH, but just showing them as an example of what can be done by extending the usefulness of an online notebook with HTML and CSS.

1 Like

Hi Eric

Sadly, only converted Prolog enthusiasts like us are going to make the effort to play with Swish, find examples in git, and search the discussion forums. Most people are going to turn to the official documentation as their first port of call – and it will be their last if it doesn’t answer their question.

While you can include examples in the documentation using markdown’s

example(Arg1,...) :-
   ....

the age old problem remains that coders don’t enjoy writing documentation and put minimal effort into it.

A trend I’ve noted is the more a language puts an effort into automating its documentation and making it useful, the longer it lives and prospers.

Relying on people to keep cutting and pasting examples from separate unit test files inevitably results in either no illustrative examples in the documentation, or out-of-date examples.

2 Likes

I partly agree with that and partly do not agree with that. :slightly_smiling_face:

A person who has to accomplish a task with Prolog and is just learning, e.g. student, as you note, will likely turn to the documentation first, but then many of them will turn to Google at which point they may/will find many similar answers at StackOverflow, and then may or may not ask a question there. I do agree that the documentation should be the first place with the best reference style examples, but that How To examples for more complex task should not be in the documentation next to a predicate but somewhere else.

I think some do, but like me, it is not writing the documentation that is the problem, the problem is keeping it in sync with the code, especially if it has a nice graphic which I like to do for the more complex concepts. The other problem with documentation for code is, when do you do the documentation? I often make so many changes at the start that if I took the time to write the documentation as I write the code it would take me 3x or more before completing the code because the I would also be updating the documentation.

That one I will disagree with because there are many languages that have had long life and that don’t/didn’t have automated documentation. When I started code it was on key punch machines and languages like C, Fortran and Basic existed and are still used commonly today, but other languages also existed and seem to have fallen out of favor. And yet there are some modern languages that have had automated documentation and are dying, e.g. Objective-C

There are many factors that influence if a programming language will live and prosper, and IMHO the backing of a major company like Google, Microsoft, or Oracle has more to do with it than the documentation.

One trick I have learned over the years for some of the more basic Prolog predicates to learn them is to just run the most general query. Just a few days ago I needed to understand sub_string/5 and while the docs kind of made sense, the part that had me confused was the type of values for each parameter, which ones were string and which ones were integer. So using this query

?- sub_string("abc",Before,Length,After,Substring).
Before = Length, Length = 0,
After = 3,
Substring = "" ;
Before = 0,
Length = 1,
After = 2,
Substring = "a" ;
Before = 0,
Length = 2,
After = 1,
Substring = "ab" ;
Before = After, After = 0,
Length = 3,
Substring = "abc" ;
Before = 1,
Length = 0,
After = 2,
Substring = "" ;
Before = Length, Length = After, After = 1,
Substring = "b" ;
Before = 1,
Length = 2,
After = 0,
Substring = "bc" ;
Before = 2,
Length = 0,
After = 1,
Substring = "" ;
Before = 2,
Length = 1,
After = 0,
Substring = "c" ;
Before = 3,
Length = After, After = 0,
Substring = "".

and then a quick reformat of the output with an added note gave me

Before Length  After  Substring
0      0       3      ""
0      1       2      "a"
0      2       1      "ab"
0      3       0      "abc"
1      0       2      ""
1      1       1      "b"
1      2       0      "bc"
2      0       1      ""
2      1       0      "c"
3      0       0      ""

Before + Length + After = length(String)

and then in made perfect sense.

Maybe one of the first things that new people, (students) need to learn when learning Prolog is how to run the most general query and then make sense of what the result(s) means to help them understand the predicate.

I personally know that if I take time to write a note down rather than just reading it, or as many young people do today of just taking a picture with their phone, I remember it much more, and recall it faster and in more detail.

Maybe the problem is not to teach people, but how to better teach virtual assistants (Alexa, Siri, Google Assistant, Cortona), as that seems to be where the newer generation goes for answers. :slightly_smiling_face: Not that I agree with that.

2 Likes

That’s a great idea. Plus, the ```test … ``` notation would likely do it without touching PLDoc, for example:

%% sort(+List, -Sorted)
% True if Sorted can be unified ...
%
% ```test
% ?- sort([4,3,1,2], Sorted),
% Sorted = [1, 2, 3, 4].
% ```
sort(In, Out) :- magic(In, Out).

If we run PLDoc on that it should just see the test case as a code block and print it in the docs. PLDoc has nothing to run, it’s not executable, it’s just a printed example.

What we’d want then is something that could parse these test cases out of the doc-string,
consult the file in question, and then call the test cases. It could be quite distinct from
PLUnit and even SWI-Prolog. But parsing is my biggest blind-spot in CS…

1 Like

When I worked on IBM Prolog, we had a script that took the system documentation, extracted all the examples, and ran them to verify that they all worked. I don’t recall anything special in the markup language. But this wasn’t for literate programming(*) - it was for verifying that the documentation was correct.

I also found this:
http://mirrors.ibiblio.org/CTAN/macros/latex/contrib/gene/pl/pl.pdf
https://ctan.org/tex-archive/macros/latex/contrib/gene/pl
and @pmoura has a bit to say here: https://dtai.cs.kuleuven.be/projects/ALP/newsletter/archive_93_96/net/techniques/lit.html

(*) I once had a 100-line script (written by someone at Quintus - perhaps Tim Lindholm) that generated nice LaTex from code that used special comments. But it didn’t do anything special with test cases – they were just like any other code. [Literate Programming for Prolog is a lot easier than other languages because Prolog allows predicates to be defined in almost any order (directives can be a bit of a problem).

1 Like

A key thing Jan said in an earlier post:

The most promising part I see is that it is now not uncommon for examples in the documentation to be incorrect, often even simply containing syntax errors (because the IDE doesn’t check syntax inside comments). If PlDoc would know an example is supposed to be valid and complete it could evaluate it and complain.

I think there is a lot of value in checking that examples in documentation don’t have typos and actually run. Combining illustrative examples and documentation would lead to good habits in unit tests, improving development besides the quality of documentation.

But am I going to be the person to actually implement this? Uhm, I’m just a hobbyist programer, journalist by trade…

OK, so I’ve started playing around with a program that could possibly do this. First I tried to do it in SWI-Prolog, but my attempts at reading a file as plain text and parsing out code blocks in comments were just miserable. If someone can provide a DCG or something for this then I’m happy to continue it in SWI-Prolog. Again, I’m not good at parsing!

In the meantime, I’m building a prototype in Python3. My approach is this:

  • Read in the file
  • Parse out the code blocks in comments marked ```test using Regex
  • Write out a .plt file
  • Call run_tests and let SWI-Prolog do the testing
  • Delete .plt file by default

It’s not perfect, it’s not robust, it’s not in SWI-Prolog (so adds a dependency). But it’s enough to raise a question:

PLUnit uses options in the tests (e.g. test(foo, [nondet]) :- bar.), these should be included in the code comment block, so how do we like:

% ```test([Options]) and the program names the test with the line in the file
Or
% ```test(foo, [Options]) and we just put that into the test file as the head

I really like the second one, it would mirror PLUnit so closely, we’d just have to shift things round a bit. But % ```test renders fine in PLDoc, as soon as we add a space or bracket into that line we’ll need to change PLDoc to handle it.

An alternative proposal, we don’t provide the full power of PLUnit assuming this usecase is for examples only that will succeed and unify all their test arguments in the code-block so the user can see everything clearly. In this case we can parse:

%% sort(+List, -Sorted) is nondet.
% True if Sorted can be unified, either
% by magic or science
%
% ```example
% ?- sort([4,3,1,2], Sorted),
% Sorted = [1, 2, 3, 4] ;
% false.
% ```
sort(In, Out) :- magic(In, Out) ; science(In, Out).

Grabbing a nondet option from the first line and assuming no options otherwise. Nothing to change in PLDoc and closer to the queries typically used in the docs.

1 Like

I don’t know why it doesn’t show in the docs, but there is xref_comment/[3,4] which parses the comment for you.

If you have the source code like this:

%% sort(+List, -Sorted) is nondet.
% True if Sorted can be unified, either
% by magic or science
%
% ```example
% ?- sort([4,3,1,2], Sorted),
% Sorted = [1, 2, 3, 4] ;
% false.
% ```
sort(In, Out) :- magic(In, Out) ; science(In, Out).

in a file called /tmp/s.pl, you can run the following query:

1 ?- S='/tmp/s.pl', xref_source(S), xref_comment(S, Head, Summary, Comment).
S = '/tmp/s.pl',
Head = user:sort(_13402, _13404),
Summary = "True if Sorted can be unified, either by magic or science.",
Comment = "%% sort(+List, -Sorted) is nondet.\n% True if Sorted can be unified
, either\n% by magic or science\n%\n% ```example\n% ?- sort([4,3,1,2], Sorted)
,\n% Sorted = [1, 2, 3, 4] ;\n% false.\n% ```" .

So the comment is returned as a string which you can parse with DCGs or with regular expressions ( library(pcre) ).

2 Likes

Python is unfortunately a strong argument against my idea: it has supported doctest since way back but hardly anyone uses it because test driven development has to be drummed into programers from an early age.

Ideally getting working illustrative examples into documentation has to be part of the basic development cycle. And it can easily be done now by cutting and pasting relevant tests from the *.plt files, but the existing documentation shows this involves too much admin for many developers.

Unfortunately, creating a Python script to do this will be no easier than cutting and pasting, so I’m a bit of a “doomster and gloomster” (to quote Boris Johnson) about it getting used much.

Sweet! Thank you swi! I reckon I can handle that in SWI-Prolog!

1 Like

2 posts were split to a new topic: Add unit test to PLDoc

Just some remarks. The PlDoc system allows extracting the comments from loaded code using doc_comment/4 or as found through the cross referencer using xref_comment for not-loaded code. The doc_wiki library translates the documentation to a Prolog term, from which you can extract the right subterm.

The code for the LPN (Learn Prolog Now) proxy contains a lot of useful stuff to analyse code fragments.

3 Likes

In looking at the LPN proxy code and thinking about one of the goals of the original question,

it is just a small intuitive leap to realize that the same concept of extracting source code of LPN to generate working SWISH consoles could be applied to the SWI-Prolog documentation. So instead of just adding example source code to the SWI-Prolog documentation which would be read as static code, instead it could have the gear icon added and then a user could click on the gear to launch a SWISH console and play with the predicate. That would be nice.


Details for those not up on this concept.

Originally there was a paper for learning Prolog called Learn Prolog Now.

That evolved into a poplar web site: LPN

Then an enhanced version was created (Notice) that scans the LPN HTML pages for Prolog source code and converts the Prolog source code into an active component with a link, shown as a gear

image

that converts the source code into a SWISH console.

The code that scans the HTML is in lpn-swish-proxy


Example from enhanced LPN, 5.3 Arithmetic and Lists

So this Prolog code in the original LPN site

len([],0).
len([_|T],N) :- len(T,X), N is X+1.

becomes this in the enhanced web site

image

and when the gear is clicked

the Prolog source code is loaded into a SWISH console ready to be run.

2 Likes

I worked 10 years at Google, which heavily emphasizes testing (from unit tests through system tests), and doctest is not used – but an extended form of Python unittest is heavily used. I think that a major reason for not using doctest is that often some test setup/teardown is needed, and doctest is not very good for that. (At Google, the same test framework also supports C++ and Java, and for those, doctest is even less useful.)

It would be nice to have a tool that validates documentation examples. The extraction part should be quite easy; but integrating with plunit seems non-trivial (I recall a moderate amount of effort was put into such a tool for documenting IBM Prolog).

2 Likes

I’m just looking at sub_string/5 in the docs, which as an example could become:

name_value(String, Name, Value) :-
        sub_string(String, Before, _, After, "="), !,
        sub_string(String, 0, Before, _, NameString),
        atom_string(Name, NameString),
        sub_string(String, _, After, 0, Value).

?- name_value("foo=bar", foo, "bar").

More challenging to parse and very useful for examples, which leads me to the conclusion that Eric is onto a better idea with adapting the LPN SWISH thing to the docs.

2 Likes

doc_comment/4 looks interesting. If only it had some illustrative examples of how to use it :slightly_smiling_face:

One of the paradoxes I’ve noticed is the worst documented part of most programing languages is how to do documentation.

This is why I’m a fan of MIT’s How to design Programs philosophy: When people learn a new programing language, instead of getting taught “Hello World” they should start with a documentation template to get clear in their head what the purpose of their predicate/function they are writing is, what it will consume, what it will produce, with some examples which initially act as stubs but in due course (possibly after many auxilary functions have been written), do the job.

That’s a really old reference (by Internet standards) :slight_smile: But the idea remains the same: in Logtalk, all data relevant for documentation (and other tools) is saved at compile time (assuming the source_data flag is on) and made available using the reflection API. This includes the predicate examples if any (no parsing required). It’s thus quite easy to programmatically access the examples and construct test queries. But this is a bit tricky to automate as there might be a need to e.g. setup a query context or clean up after it (I think that was already mention elsewhere in this thread). The examples may also be included in e.g. a protocol or category documentation instead of an object documentation.

P.S. Are the IBM Prolog manuals and other documentation publicly available? My understanding is that it’s a discontinued system.

It is all not that different from SWI-Prolog’s infrastructure. Documentation is comment, so if you do nothing it is lost. If you use :- doc_collect(true). it collects the structured comment while loading the sources. That parses the mode declaration and makes these available through library(pldoc/doc_modes). It associates the remainder of the documentation as a string with the documented predicates, making this raw material available through doc_comment/4. Next, library(pldoc/doc_wiki) can be used to turn this into a structured Prolog term. In the normal documentation setup, library(pldoc/doc_html) turns this into HTML and library(pldoc/doc_latex) into a LaTeX document.

The whole things is pretty transparent. Most functionality is only documented using reference style documentation as PlDoc in the sources. So, you need a little look around in the source to find the relevant APIs.

Best way to get examples on how to use a predicate is to do something like this:

$ cd <swi-prolog source dir>
$ grep -r -A 3 -B 3  'doc_comment' .                 
1 Like

I tried finding the manuals but they seem to have disappeared. It is a long-discontinued system. (A long sad story … amongst other things, IBM Marketing pushed some inferior products; there were a number of bad product decisions due to IBM SAA AD/Cycle and PS/2; etc., etc.)

1 Like