Just some remarks. The PlDoc system allows extracting the comments from loaded code using doc_comment/4 or as found through the cross referencer using xref_comment for not-loaded code. The doc_wiki library translates the documentation to a Prolog term, from which you can extract the right subterm.
The code for the LPN (Learn Prolog Now) proxy contains a lot of useful stuff to analyse code fragments.
In looking at the LPN proxy code and thinking about one of the goals of the original question,
it is just a small intuitive leap to realize that the same concept of extracting source code of LPN to generate working SWISH consoles could be applied to the SWI-Prolog documentation. So instead of just adding example source code to the SWI-Prolog documentation which would be read as static code, instead it could have the gear icon added and then a user could click on the gear to launch a SWISH console and play with the predicate. That would be nice.
Details for those not up on this concept.
Originally there was a paper for learning Prolog called Learn Prolog Now.
That evolved into a poplar web site: LPN
Then an enhanced version was created (Notice) that scans the LPN HTML pages for Prolog source code and converts the Prolog source code into an active component with a link, shown as a gear
that converts the source code into a SWISH console.
The code that scans the HTML is in lpn-swish-proxy
Example from enhanced LPN, 5.3 Arithmetic and Lists
So this Prolog code in the original LPN site
len([],0).
len([_|T],N) :- len(T,X), N is X+1.
becomes this in the enhanced web site
and when the gear is clicked
the Prolog source code is loaded into a SWISH console ready to be run.
I worked 10 years at Google, which heavily emphasizes testing (from unit tests through system tests), and doctest is not used – but an extended form of Python unittest is heavily used. I think that a major reason for not using doctest is that often some test setup/teardown is needed, and doctest is not very good for that. (At Google, the same test framework also supports C++ and Java, and for those, doctest is even less useful.)
It would be nice to have a tool that validates documentation examples. The extraction part should be quite easy; but integrating with plunit seems non-trivial (I recall a moderate amount of effort was put into such a tool for documenting IBM Prolog).
I’m just looking at sub_string/5 in the docs, which as an example could become:
name_value(String, Name, Value) :-
sub_string(String, Before, _, After, "="), !,
sub_string(String, 0, Before, _, NameString),
atom_string(Name, NameString),
sub_string(String, _, After, 0, Value).
?- name_value("foo=bar", foo, "bar").
More challenging to parse and very useful for examples, which leads me to the conclusion that Eric is onto a better idea with adapting the LPN SWISH thing to the docs.
doc_comment/4 looks interesting. If only it had some illustrative examples of how to use it 
One of the paradoxes I’ve noticed is the worst documented part of most programing languages is how to do documentation.
This is why I’m a fan of MIT’s How to design Programs philosophy: When people learn a new programing language, instead of getting taught “Hello World” they should start with a documentation template to get clear in their head what the purpose of their predicate/function they are writing is, what it will consume, what it will produce, with some examples which initially act as stubs but in due course (possibly after many auxilary functions have been written), do the job.
That’s a really old reference (by Internet standards) 
But the idea remains the same: in Logtalk, all data relevant for documentation (and other tools) is saved at compile time (assuming the source_data flag is on) and made available using the reflection API. This includes the predicate examples if any (no parsing required). It’s thus quite easy to programmatically access the examples and construct test queries. But this is a bit tricky to automate as there might be a need to e.g. setup a query context or clean up after it (I think that was already mention elsewhere in this thread). The examples may also be included in e.g. a protocol or category documentation instead of an object documentation.
P.S. Are the IBM Prolog manuals and other documentation publicly available? My understanding is that it’s a discontinued system.
It is all not that different from SWI-Prolog’s infrastructure. Documentation is comment, so if you do nothing it is lost. If you use :- doc_collect(true). it collects the structured comment while loading the sources. That parses the mode declaration and makes these available through library(pldoc/doc_modes). It associates the remainder of the documentation as a string with the documented predicates, making this raw material available through doc_comment/4. Next, library(pldoc/doc_wiki) can be used to turn this into a structured Prolog term. In the normal documentation setup, library(pldoc/doc_html) turns this into HTML and library(pldoc/doc_latex) into a LaTeX document.
The whole things is pretty transparent. Most functionality is only documented using reference style documentation as PlDoc in the sources. So, you need a little look around in the source to find the relevant APIs.
Best way to get examples on how to use a predicate is to do something like this:
$ cd <swi-prolog source dir>
$ grep -r -A 3 -B 3 'doc_comment' .
I tried finding the manuals but they seem to have disappeared. It is a long-discontinued system. (A long sad story … amongst other things, IBM Marketing pushed some inferior products; there were a number of bad product decisions due to IBM SAA AD/Cycle and PS/2; etc., etc.)