Term_to_json/2 documentation fixes

Help!
1

MQI uses term_to_json/2 to convert query answers to JSON and I want to link the documentation. But, unless I (and based on the comments under the docs, LogicalCaptain) am really missing something, the documentation for term_to_json/2 is pretty far off.

The doc currently says this (I’ve put my own tests indented under each bullet). As you can see from the tests, the docs are not really even close:

Convert any general Prolog term into a JSON term. Prolog lists 
are treated in a special way. Also, JSON terms are not converted. Mapping:

* Variable: `{"type":"var", "name":<string>}`
   ?- term_to_json(X, JSON).
   JSON = "_".

* Atom: `{"type":"atom", "value":<string>}`
   ?- term_to_json(a, JSON).
   JSON = a.

* Integer: `{"type":"integer", "value":<integer>}`
   ?- term_to_json(1, JSON).
   JSON = 1.

* Float: `{"type":"float", "value":<float>}`
   ?- term_to_json(1.1, JSON).
   JSON = 1.1.

* List: JSON array
   ?- term_to_json([A, 1, "a"], JSON).
   JSON = ["_", 1, "a"].

* Dict: a JSON object. Values are processed recursively. (the tag is ignored)
   ?- term_to_json(tag{a:A}, JSON).
   JSON = tag{a:"_"}.

* `json([Key=Value, ...])`: a JSON object Values are processed recursively.
   # See comments at bottom on why this fails...
   ?- term_to_json(json([a=b]), JSON).
   false.
   
* compound: `{"type":"compound", "functor":<string>, "args":<array>}`
   ?- term_to_json(my_term(A, "one"), JSON).
   JSON = json{args:["_", "one"], functor:my_term}.

Based on the tests above and looking at the code, it should say this:

Convert any general Prolog term into a JSON term which can be
used with any of the json_write, json_write_dict, json_read, 
json_read_dict predicates.  It generates the JSON format used by default in
pengines and MQI.

While library(http/json_convert) allows for
a custom mapping of Prolog terms to JSON, this 
predicate gives a default mapping that is not changeable.  For example:

   ?- term_to_json(my_predicate(a, "b", 1, X, @(true)), JSON).
   JSON = json{args:[a, "b", 1, "_", true], functor:my_predicate}.


Mapping:

* Variable: JSON = "_"
* Atom: JSON = Atom
* String: JSON = String
* Integer >= -(2**31), Integer < 2**31: JSON = Integer
* Integer < -(2**31), Integer > 2**31: atom_number(JSON, Integer)
* Float: JSON = Float
* @(JSON_Term) where JSON_Term is `true`, `false` or `null`: JSON_Term
* List: List, list items are processed recursively using term_to_json/2
* Dict: Dict, Values are processed recursively using term_to_json/2,
  the tag is retained
* Compound: Compound =.. [F|Args], JSON = json{functor:F, args:Args}, Args 
   are processed recursively using term_to_json/2
* `json([Key=Value, ...])`: Fails. Term must not be an existing 
  `json(List)` term (and must not recursively contain any).

Notes:
* If Term is (or recursively contains) existing JSON terms (i.e. terms of 
  the form `json(List)`, term_to_json fails.
* Valid JSON predicates (i.e. predicates of the form `json(List)`)
  which either contain dicts that use integers as keys or contain 
  objects like streams will fail when written to text by the 
  `json_write` and `json_write_dict` predicates. Neither can 
  be represented in JSON text. term_to_json will generate these
  predicates successfully, but they will fail when writing them to text.

Given that terms of the form json/1 were supposed to work (but I believe never have), I propose NOT doing the fix below, but fixing the docs as above to describe the current behavior…

# Description of why term_to_json/2 fails when processing json(List) terms
term_to_json(Term, Bindings, JSON) :-
    findall(X,
            (   maplist(bind_var, Bindings),
                numbervars(Term, 0, _, [singletons(true)]),
                to_json(Term, X)
            ),
            # Because JSON is in a list, it forces findall to fail if there is more than one item
            # because a cut is missing below (in to_json), multiple items are returned for json/1 terms
            # and thus it fails
            [JSON]).

to_json(json(Pairs0), Term) :-
    must_be(list, Pairs0),
    # this should be added to make this work as described 
    # in the comment above:
    # !, 
    maplist(pair_value_to_json_ex, Pairs0, Pairs),
    dict_pairs(Term, json, Pairs).
2

Hmm. This is clearly a mess :frowning: The implementation is broken in several places and the documentation has no relation to the code :frowning: This is already the case for the oldest version checked into git, but then the Pengines package started life elsewhere and the early history is lost.

The question is, how to get out of this mess? Fixing the docs to meet the implementation is surely a good step.

Roughly, I guess there are two scenarios.

  • An application wants to talk to another application using JSON. The JSON is nicely defined. The predicate json_read_dict/2 and json_write_dict/2 do, in my experience, a good job. They can only be passed a subset of Prolog (no compounds, no variables, dicts with atom keys only).

  • An application wants to faithfully transfer an arbitrary Prolog term. That should do something along the lines of the docs for term_to_json/2.

As is, the implementation of term_to_json/2 is bit in the middle. It maps most Prolog terms to JSON. The translation fails for some Prolog terms (without raising an error) and creates invalid JSON for others (e.g., using an integer key). That, IMO, are errors. The resulting JSON sort of “looks natural”, but in fact it is rather ambiguous. A variable is simply mapped to a string.

What to do? Shouldn’t MQI simply use json_write_dict/2,3? Or, alternatively use an implementation that actually does a faithful by-directional mapping between Prolog and JSON? The main disadvantage to the latter is that it is rather unnatural to have {"type":"atom", "value":<string>}

Applications using Pengines with the JSON exchange format rely on the current translation, so changing this is dangerous. The best way out for Pengines is probably to fix the docs, decide (as MQI) what to do, implement that and provide it as an alternative exchange format.
Possibly MQI should also allow specifying the serialization format and provide some plugin to add new serializations? Pengines also allows for adding new formats, which is used by SWISH.

Thanks for raising this!

2 Likes