Does PLDoc have something akin to Python’s doctest, ie a way to include simple examples when describing what a function does, which double as unit tests?
In prolog, this might look something like this
%% sort(+List, -Sorted)
% True if Sorted can be unified ...
%
% ?- sort([4,3,1,2], Sorted).
% Sorted = [1, 2, 3, 4].
% ....
The example in the comment would be ignored unless an option (in Python’s case -v) is used when the script is run.
I think having a feature like this is nifty for at least three reasons:
It encourages test driven development, leading to better work methods when creating brand new code.
Once the code is done and hopefully used by other people, it leaves helpful examples of how to use it in its wake. (I’m sure I’m not the only one who finds the lack of examples in SWI Prolog’s documentation frustrating).
When the code inevitably has to get refactored, the documentation acts as a test framework to ensure old stuff hasn’t been broken while introducing new stuff.
A problem with PLUnit is it isn’t integrated with PLDoc (at least not to my knowledge). Literate programming and test driven development go hand-in-hand because examples make good documentation, and SWI Prolog suffers from a lack of examples in its documentation.
That is known and some of us would like to change that. I tried to add more documentation here with Discourse but ran into a parade of problems with using raw HTML on Discourse. Mostly I like to use links to SVG and Discouse does not like the links and does not work with embedded SVG with the SVG tag more than 250 characters from the start of the HTML.
One of the best places I have found to find examples is in the SWI-Prolog source code on GitHub. If you look you can find test cases in with the code, and some of them are done in an older style that have not be upgraded.
The way I look at it is that I get to use SWI-Prolog for free, it works great, and Jan is very helpful with free and fast support. Try writing a Prolog interpreter on your own and then you will realize how many man years you have saved by using SWI-Prolog.
I don’t really see the full picture. Of course there are code blocks in plDoc comments. These comments are not necessarily sensible test cases though. Many are incomplete, so they do not run without additional context. Of course we could do something like ```test to indicate that a particular block is also a test.
Then there are tests, let’s say using plUnit. If you look at them (there are plenty in src/Tests in the source distribution), you’ll see that many of them are quite meaningless as examples.
The most promising part I see is that it is now not uncommon for examples in the documentation to be incorrect, often even simply containing syntax errors (because the IDE doesn’t check syntax inside comments). If PlDoc would know an example is supposed to be valid and complete it could evaluate it and complain.
I’m not sure this is worth the trouble, but if someone creates a PR I’m ok with it. Note that this probably requires sandboxing as well such that we can still run PlDoc over untrusted sources
Ideas to get more examples are welcome though. One might be a much better annotation facility on the website. Some people do already add examples as annotations. If this would have a good discussion option such as the Discourse platform, it might work. In fact, Discourse has an API and thus we could remove the existing annotation and use Discourse for page annotation. Not sure how hard that will be, but it might turn out not to be too hard. Leaves @EricGT’s complaints for doing advanced Prolog and SVG stuff on this platform. We could also link documentation pages to a SWISH notebook.
I’ve no idea how difficult it would be to include something similar in PLDoc, but think it would improve SWI Prolog’s “educability” for novice programers.
While Python seems to encourage the HTDP philosophy with doctest, I haven’t seen it widely taught or used, which is a pitty.
I accept that trying to get examples in the documentation to replace unit tests is unrealistic, but I’ve personally found following the HTDP philosophy of starting by thinking up illustrative examples and then coding has helped me a lot, and also improved my documentation (especially for myself when I return to decipher my own code later) a lot.
As Jan noted, we could also link documentation pages (Discourse) to a SWISH notebook and in another topic that SWISH is a long term asset for SWI-Prolog. So writing documentation examples and How To examples in SWISH then using links from Discourse to SWISH is the next path I plan to check out.
However SWISH does not support unit test (Example), because of the sandbox. I do not plan to ask that SWISH support unit test at present.
Since I create HTML pages as raw HTML, I am use to them not being liked by many online sites that accept markdown and occasional raw HTML, but Jan noted in another topic that such changes to SWISH to accept such HTML and CSS would be considered.
A few years ago when I was helping others learn Python I used Runestone Academy which hosts interactive text books for learning Python among other things. The site had many more types of widgets than I have seen in any other type of online textbook. Again I am not asking that these be added to SWISH, but just showing them as an example of what can be done by extending the usefulness of an online notebook with HTML and CSS.
Sadly, only converted Prolog enthusiasts like us are going to make the effort to play with Swish, find examples in git, and search the discussion forums. Most people are going to turn to the official documentation as their first port of call – and it will be their last if it doesn’t answer their question.
While you can include examples in the documentation using markdown’s
example(Arg1,...) :-
....
the age old problem remains that coders don’t enjoy writing documentation and put minimal effort into it.
A trend I’ve noted is the more a language puts an effort into automating its documentation and making it useful, the longer it lives and prospers.
Relying on people to keep cutting and pasting examples from separate unit test files inevitably results in either no illustrative examples in the documentation, or out-of-date examples.
I partly agree with that and partly do not agree with that.
A person who has to accomplish a task with Prolog and is just learning, e.g. student, as you note, will likely turn to the documentation first, but then many of them will turn to Google at which point they may/will find many similar answers at StackOverflow, and then may or may not ask a question there. I do agree that the documentation should be the first place with the best reference style examples, but that How To examples for more complex task should not be in the documentation next to a predicate but somewhere else.
I think some do, but like me, it is not writing the documentation that is the problem, the problem is keeping it in sync with the code, especially if it has a nice graphic which I like to do for the more complex concepts. The other problem with documentation for code is, when do you do the documentation? I often make so many changes at the start that if I took the time to write the documentation as I write the code it would take me 3x or more before completing the code because the I would also be updating the documentation.
That one I will disagree with because there are many languages that have had long life and that don’t/didn’t have automated documentation. When I started code it was on key punch machines and languages like C, Fortran and Basic existed and are still used commonly today, but other languages also existed and seem to have fallen out of favor. And yet there are some modern languages that have had automated documentation and are dying, e.g. Objective-C
There are many factors that influence if a programming language will live and prosper, and IMHO the backing of a major company like Google, Microsoft, or Oracle has more to do with it than the documentation.
One trick I have learned over the years for some of the more basic Prolog predicates to learn them is to just run the most general query. Just a few days ago I needed to understand sub_string/5 and while the docs kind of made sense, the part that had me confused was the type of values for each parameter, which ones were string and which ones were integer. So using this query
?- sub_string("abc",Before,Length,After,Substring).
Before = Length, Length = 0,
After = 3,
Substring = "" ;
Before = 0,
Length = 1,
After = 2,
Substring = "a" ;
Before = 0,
Length = 2,
After = 1,
Substring = "ab" ;
Before = After, After = 0,
Length = 3,
Substring = "abc" ;
Before = 1,
Length = 0,
After = 2,
Substring = "" ;
Before = Length, Length = After, After = 1,
Substring = "b" ;
Before = 1,
Length = 2,
After = 0,
Substring = "bc" ;
Before = 2,
Length = 0,
After = 1,
Substring = "" ;
Before = 2,
Length = 1,
After = 0,
Substring = "c" ;
Before = 3,
Length = After, After = 0,
Substring = "".
and then a quick reformat of the output with an added note gave me
Maybe one of the first things that new people, (students) need to learn when learning Prolog is how to run the most general query and then make sense of what the result(s) means to help them understand the predicate.
I personally know that if I take time to write a note down rather than just reading it, or as many young people do today of just taking a picture with their phone, I remember it much more, and recall it faster and in more detail.
Maybe the problem is not to teach people, but how to better teach virtual assistants (Alexa, Siri, Google Assistant, Cortona), as that seems to be where the newer generation goes for answers. Not that I agree with that.
If we run PLDoc on that it should just see the test case as a code block and print it in the docs. PLDoc has nothing to run, it’s not executable, it’s just a printed example.
What we’d want then is something that could parse these test cases out of the doc-string,
consult the file in question, and then call the test cases. It could be quite distinct from
PLUnit and even SWI-Prolog. But parsing is my biggest blind-spot in CS…
When I worked on IBM Prolog, we had a script that took the system documentation, extracted all the examples, and ran them to verify that they all worked. I don’t recall anything special in the markup language. But this wasn’t for literate programming(*) - it was for verifying that the documentation was correct.
(*) I once had a 100-line script (written by someone at Quintus - perhaps Tim Lindholm) that generated nice LaTex from code that used special comments. But it didn’t do anything special with test cases – they were just like any other code. [Literate Programming for Prolog is a lot easier than other languages because Prolog allows predicates to be defined in almost any order (directives can be a bit of a problem).
The most promising part I see is that it is now not uncommon for examples in the documentation to be incorrect, often even simply containing syntax errors (because the IDE doesn’t check syntax inside comments). If PlDoc would know an example is supposed to be valid and complete it could evaluate it and complain.
I think there is a lot of value in checking that examples in documentation don’t have typos and actually run. Combining illustrative examples and documentation would lead to good habits in unit tests, improving development besides the quality of documentation.
But am I going to be the person to actually implement this? Uhm, I’m just a hobbyist programer, journalist by trade…
OK, so I’ve started playing around with a program that could possibly do this. First I tried to do it in SWI-Prolog, but my attempts at reading a file as plain text and parsing out code blocks in comments were just miserable. If someone can provide a DCG or something for this then I’m happy to continue it in SWI-Prolog. Again, I’m not good at parsing!
In the meantime, I’m building a prototype in Python3. My approach is this:
Read in the file
Parse out the code blocks in comments marked ```test using Regex
Write out a .plt file
Call run_tests and let SWI-Prolog do the testing
Delete .plt file by default
It’s not perfect, it’s not robust, it’s not in SWI-Prolog (so adds a dependency). But it’s enough to raise a question:
PLUnit uses options in the tests (e.g. test(foo, [nondet]) :- bar.), these should be included in the code comment block, so how do we like:
% ```test([Options]) and the program names the test with the line in the file
Or % ```test(foo, [Options]) and we just put that into the test file as the head
I really like the second one, it would mirror PLUnit so closely, we’d just have to shift things round a bit. But % ```test renders fine in PLDoc, as soon as we add a space or bracket into that line we’ll need to change PLDoc to handle it.
An alternative proposal, we don’t provide the full power of PLUnit assuming this usecase is for examples only that will succeed and unify all their test arguments in the code-block so the user can see everything clearly. In this case we can parse:
%% sort(+List, -Sorted) is nondet.
% True if Sorted can be unified, either
% by magic or science
%
% ```example
% ?- sort([4,3,1,2], Sorted),
% Sorted = [1, 2, 3, 4] ;
% false.
% ```
sort(In, Out) :- magic(In, Out) ; science(In, Out).
Grabbing a nondet option from the first line and assuming no options otherwise. Nothing to change in PLDoc and closer to the queries typically used in the docs.
I don’t know why it doesn’t show in the docs, but there is xref_comment/[3,4] which parses the comment for you.
If you have the source code like this:
%% sort(+List, -Sorted) is nondet.
% True if Sorted can be unified, either
% by magic or science
%
% ```example
% ?- sort([4,3,1,2], Sorted),
% Sorted = [1, 2, 3, 4] ;
% false.
% ```
sort(In, Out) :- magic(In, Out) ; science(In, Out).
in a file called /tmp/s.pl, you can run the following query:
1 ?- S='/tmp/s.pl', xref_source(S), xref_comment(S, Head, Summary, Comment).
S = '/tmp/s.pl',
Head = user:sort(_13402, _13404),
Summary = "True if Sorted can be unified, either by magic or science.",
Comment = "%% sort(+List, -Sorted) is nondet.\n% True if Sorted can be unified
, either\n% by magic or science\n%\n% ```example\n% ?- sort([4,3,1,2], Sorted)
,\n% Sorted = [1, 2, 3, 4] ;\n% false.\n% ```" .
Python is unfortunately a strong argument against my idea: it has supported doctest since way back but hardly anyone uses it because test driven development has to be drummed into programers from an early age.
Ideally getting working illustrative examples into documentation has to be part of the basic development cycle. And it can easily be done now by cutting and pasting relevant tests from the *.plt files, but the existing documentation shows this involves too much admin for many developers.
Unfortunately, creating a Python script to do this will be no easier than cutting and pasting, so I’m a bit of a “doomster and gloomster” (to quote Boris Johnson) about it getting used much.
