Documentation systems (PlDoc, LpDoc, etc.)

Discussion
1

Just for clarification, the Ciao documenter, LPdoc, supports both directive-based and structured comment-based documentation.

2

Thanks. I was not aware of that. It does of course raise the question on what you consider the “best” approach?

3

The third approach of course is embedding the code in the documentation, in your literate source file. I think this is a valid approach but the existing tooling does not support working on such literate sources directly. I am not even sure I know how a modern IDE for literate programming should behave and what it should look like.

4

Good question Jan! I think there are at least two issues here really:

A first one (the one I think you were referring mostly to, Jan) is whether to be directive-based or structure comment-based, i.e.,

%! #Title
%! foo(+X,-Y): Description.
etc.

versus

:- doc(section, "Title").,
:- pred foo(+X,-Y) : ..."Description".
etc.

Documentation comments have, as you mention, the advantage that the file can be loaded into any Prolog without having to define directives or operators. Also, they can sometimes be more compact and readable. On the other hand directives can be quite appropriate in some cases, for example if what they state is more complex and/or also part of the program, such as, for example, assertions. This is why Ciao Prolog’s LPdoc allows both doccomments and directives. In any case, it is also probably a matter of taste.

A separate issue is what to use for the text formatting commands, i.e., markdown (# Title, - , etc.), or markup (@section{Title}, @item … @end{itemize}, etc.).

Also probably a matter of taste: people used to LaTeX, texinfo, etc. may tend to like markup better, whereas otherwise people may prefer markdown.

It can also depend on the type of text: for me markdown is very convenient for small to medium-sized texts with relatively standard formatting. Markup can be more resilient and perhaps better suited for longer texts, specially if there is complex formatting, since it makes it easier to have lots of
commands. It can also be more resilient; in markdown it is sometimes easy to make mistakes, e.g., with indenting, that can only be seen in the output.

A solution is to be able to use both markup or markdown so that e.g., files can be documented with markdown but one can escape to markup if needed (this is also what is done in LPdoc).

So, I guess my answer is that there is not necessarily a ‘best’ approach? :slight_smile:

5

For the complex things we have of course the predicate as example, where, at least for SWI-Prolog, the first part is formal syntax and violation gives an error when documentation is generated.

%!  foo(+X,-Y)

I’ve always considered Ciao assertions as part of the program. Surely I would not consider using structured comments for them (despite the disadvantage that it makes the code less portable). Possible we should have some general extension mechanism like C #pragma or __attribute__. Of course, the price is reduced readability :frowning:

Funny. I’ve done enough markup languages (troff, scribe, LaTeX, TexInfo) but I typically prefer markdown for documentation. In particular inside source code as it is easier to read. Long or short doesn’t matter too much. Only if you want complex things like non-trivial tables it gets nasty.

6

Forgetting for a moment about the exact technology. If we consider a module which is part of the standard library. There are two separate needs:

  • For the “client”, documenting the library interface (the exported predicates): what predicates are available, how are they meant to be used, possibly with examples.
  • For the “contributor”, documenting the library code (the actual source): explaining the non-obvious decisions in the implementation.

The trend is to keep both kinds of documentation as close to the real source code as possible. It all ends up as (possibly structured) comments in the source file. Most documentation systems are aware of the two use cases, they usually present a view meant for the client, but have references to the contributor view. SWI-Prolog also keeps a lot of its client documentation in doc files that live a separate life.

My point is that there are two separate use cases for documentation and those are in fact handled differently at the moment.

Maybe it is worth having two separate approaches to documentation? Some considerations:

  • Client documentation might benefit from examples that are in fact executed (with protocol of the results) during the document generation. So, a “notebook”. There are already multiple “example” approaches in SWI-Prolog’s web site: this indicates a clear need but no perfect solution yet.
  • Contributor docs can in fact be just comments in the source file. This is already the case. It doesn’t need any special presentation as client docs do.
  • The client docs can be embedded in the source, too, but the current situation suggests that sometimes there is no obvious way to do it.
  • Cross-referencing is the single most important feature of client docs.

The last point is why I see the markup vs markdown discussion a bit beside the point. The only really important question is, how well is my documentation going to be cross-referenced once published (online, in HTML, for 99.9% of the users…)

1 Like
7

Thanks for the writeup. Some notes to explain the current situation

  • PlDoc does separate internal from client docs in the sense that it does not generate public doc for private predicates.
  • A lot of the docs is in LaTeX. That is how the docs started. Over time, the LaTeX docs for most libraries has been replaced by PlDoc. Ideally that should be completed.
  • There are several things that I still consider missing in the the current PlDoc
    • The ability to group predicate descriptions and wrap these groups in sections. As is, there is merely a header and a list of predicates in the order of the source. It is possible to document a library in an external Markdown document and include the actual predicate descriptions from the source. This is a little awkward though.
    • Example support (as you mention)
  • Cross-referencing is based on predicate references (mostly). That works quite well, until multiple libraries export the same predicate. That could be fixed using module prefixing, but as it that doesn’t work very well :frowning:

It would be great to have many more examples. I’m not sure how to get them though. The core infrastructure is quite reasonable (I think), but it does not lead to contributions (you did a few, thanks). Who would like to contribute examples and what support would help you to start doing so?

The documentation pipeline is a bit of a mess:

  • The core is LaTeX. PlDoc is first translated to LaTeX. This generates the PDF manual.
  • The LaTeX is converted to HTML using packages/ltx2htm (Prolog).
  • The website extracts the section hierarchy and predicate descriptions from the HTML, adding search, cross-referencing, examples, etc.
8

Thank you for the explanation Jan.

What I was suggesting is that you cannot realistically put all docs in source code files. For example, the high-level overview or the glossary: where do these belong?

It would be easier to just leave some docs as docs (this is already the case!). This resolves the sectioning problem –

What exactly makes it awkward?

PS: Markup vs markdown: again, this is beside the point. It is somewhat important that whichever it is, it supports:

  • nested sections
  • that can be cross-referenced
  • and populate a table of contents

I have learned to fear the moment when any documentation format prevents me from doing what I want (limited table formatting, for example…). This is why I like (markdown) processors that allow inserting raw HTML.

9

It is a bit double. It means you write the predicate docs in the source and then a separate markdown file where you use

  - [[p/1]]
  - [[q/4]]
  - ...

Next you need some Cmake logic to tell the converter to load the associated library before doing the Markdown → LaTeX conversion. You need some place to put the markdown file and you need to keep the markdown docs in sync with the source, which is easier if we are dealing with a single file. The original design had in mind to introduce

/** <section> Title

Bla bla.
*/

Possibly that should finally be implemented? I think there is already a rule that if the /** <module> comment includes predicates as above, there are not included again.

Note that as we now distribute the libraries as .qlf, more comments have no effect on the load performance.