Swiplserver: problem with create_dict on python 3.10

Help!
1

I am using:

  • Ubuntu 22.04 LTS
  • python 3.10
  • the last version of swiplserver==1.0.2 (pip install swiplserver)
  • SWI-Prolog (threaded, 64 bits, version 8.4.2)

My problem now is, it is not possible anymore to create dictionaries. An example code so far:

from swiplserver import PrologMQI, PrologThread

with PrologMQI() as mqi:
    with mqi.create_thread() as prolog_thread:
        result = prolog_thread.query("dict_create(Dict, Tag, [1-""a"", 2-""b""]).")
        print(result)

the expected result from swipl:

?- dict_create(D,proof, [1-"a",2-"b"]).
D = proof{1:"a", 2:"b"}.

If I debug the code in python, the problem is, the variable data in the function _receive of file prologmqi.py. data keeps the value null and so it leads into some endless loop.

2

I verified that this doesn’t work on the latest build, but the latest build has had some fixes so that it doesn’t hang anymore, but it does return an exception (which is what a recent fix fixed).

I’m curious though, did this exact code used to work using swiplserver?

Edit: The problem boils down to the fact that this code throws (everything after dict_create is the code that MQI runs to serialize the result:

?- dict_create(D,proof, [1-a,2-b]), 
    term_to_json:term_to_json(D, Json), 
    with_output_to(string(Json_String), 
                   (current_output(Stream), json:json_write(Stream, Json))).

ERROR: Type error: `text' expected, found `1' (an integer)
ERROR: In:
ERROR:   [11] with_output_to(string(_176784),(current_output(_176794),json: ...))
ERROR:   [10] '<meta-call>'(user:user: ...) <foreign>
ERROR:    [9] toplevel_call(user:user: ...) at /Applications/SWI-Prolog.app/Contents/swipl/boot/toplevel.pl:1162

I’ll dig in tomorrow and see if I can figure out the fix.

Edit: SWI Prolog dicts support keys that are integers, JSON does not. json_write/2 (and json_write_dict/2) throw if you try to serialize a dict with an integer key to JSON and that’s what’s going on here.

Edit: I’m trying to see how pengines solves this issue (so I can copy that approach) or if it exists there too, but I’m not sure how to get a pengine set up to be serializing raw JSON as a response. I did figure out that using the pengines reply_json/1 predicate hits the same issue, so I suspect the same issue exists there:

http_json:reply_json(proof{1:a,2:b}).
Content-type: application/json; charset=UTF-8

{
  
ERROR: Type error: `text' expected, found `1' (an integer)
ERROR: In:
...
1 Like
3

OK, the problem is that the format used “on the wire” by MQI is JSON since that is the most interoperable, supported, etc. I.e. it is friendly to mostly languages.

The dict you are creating is a valid dict in SWI Prolog (and in Python for that matter) but it is not a valid JSON key in either language (or in the JSON specification). If you convert a dict with integer keys to JSON, Python converts the keys to strings:

>>> json.dumps({1:"a", 2:"b"})

'{"1": "a", "2": "b"}'

But SWI Prolog throws an exception, as you’ve seen. If you change the keys to be an atom, it works fine (although it does strip the tag, which is the behavior of term_to_json/2):

?- dict_create(D,proof, [key1-a,key2-b]), 
    term_to_json:term_to_json(D, Json), 
    with_output_to(string(Json_String), 
                   (current_output(Stream), json:json_write(Stream, Json))).
D = Json, Json = proof{key1:a, key2:b},
Json_String = "{\"key1\":\"a\", \"key2\":\"b\"}",
Stream = <stream>(0x6000029ebf00).

Options for MQI are:

  1. Leave it as is: Workarounds are many: use an atom key, wrap what you are doing in a predicate that massages the data into another form, etc. There are other things that can’t be serialized to JSON, like the query open_null_stream(X) that have to be wrapped as well so it isn’t unprecedented.

  2. Update json_write and json_write_dicts to do what Python does. Seems like this could affect a lot of code…

  3. Create a local version of json_write that does what Python does but only applies to MQI. I don’t like forking this kind of low level routine for lots of reasons…

  4. Provide a predicate that does that conversion that people can use. This predicate could also be used to convert any tags so that they get returned and to do other conversions over time if there are other JSON serialization cases like this that appear. Something like json_formatter(JSON_in, JSON-out, Options).

  5. Use the predicate from #4 by default to always (or optionally) do the conversion. This is the approach taken for attributes in MQI currently.

@jan any thoughts here? My inclination is to go with 4 or 5. #4 has the bonus of not being in the code path unless the user needs it, doesn’t add a bunch of options to MQI, etc. But I am curious if pengines has a different approach to this issue when sending back JSON (or if the predicates built in #4 could be useful for that scenario as well).

4

Hmmm. JSON cannot fully represent a Prolog dict in a natural way. We could of course invent some (verbose) JSON representation of any dict. I doubt this makes much sense for MQI. After all, MQI is there to allow some other language do delegate part of a computation to Prolog. IMO the application should design a JSON data scheme that interfaces data relevant to the application. As any JSON object can be represented naturally in Prolog, there is no problem. And yes, besides that Prolog can represent data in ways that poorly map to JSON. That should barely be an issue for an application.

In other words, I think a user of MQI (or any other binding to Prolog or even between two arbitrary languages) should design an interface that consists of a data schema and functions/methods/predicates/… that can be called and implement that. This opposed to making arbitrary calls on Prolog and feed the results thereof back to some other call on Prolog. In part this cannot work differently. For example, you cannot open a stream in Prolog through MQI and use that stream. Prolog streams are represented by blobs that can be serialized (written), but deliberately cannot be read. This allows Prolog to perform garbage collection on such objects. As a result, if you need to use some functionality through MQI that involves a Prolog stream you must define a more high level predicate in Prolog that performs all actions that create, use and close the stream and call that through MQI.

So, I would not change anything …

P.s. As for Pengines, the same applies. SWISH uses the json-html format for replies which hold the overall structure of the reply in JSON, but represents Prolog terms as HTML.

2 Likes
5

That is what I was thinking. There is already a specification for JSON schema, see: JSON Schema

The down side of a JSON schema is that

  1. The noted schema is not official AFAIK.
  2. Many people don’t use a schema when given the chance.
  3. AFAIK there is no SWI-Prolog JSON schema validator. They do exist for other programming languages (ref)
  4. Each message type would need a new JSON schema.

:+1:

I do like MQI I just don’t like changes to SWI-Prolog that are specialized.

Related topic: json_dict/2 - Helpful for learning how to use JSON with SWI-Prolog dict

Expand the notes section for some really useful information.

6

It seems that the browser converts number keys into strings and back (sic!):

/* Google Chrome 101.0.4951.67 */
> x = {5: 'abc', 7: 'def'}
{5: 'abc', 7: 'def'}
> JSON.stringify(x)
'{"5":"abc","7":"def"}'
> y=JSON.parse(JSON.stringify(x))
{5: 'abc', 7: 'def'}

But maybe this has to do that keys are also differently handled in JavaScript:

y[7]
'def'
y["7"]
'def'

Because JavaScript seems to do:

Property names are string or Symbol. Any other value,
including a number, is coerced to a string.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Property_Accessors

Edit 28.05.2022:
In as far SWI-Prolog dicts show a different behaviour than JavaScript dicts:

?- Y = _{5: 'abc', 7: 'def'}, V = Y.7.
Y = _{5:abc, 7:def},
V = def.

?- Y = _{5: 'abc', 7: 'def'}, V = Y.'7'.
ERROR: key `'7'' does not exist in _2036{5:abc,7:def}

Same problem with Python:

>>> x = {5:"abc", 7:"def"}
>>> x[7]
'def'
>>> x['7']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: '7'

Making Python not be able to convert back, unlike JavaScript:

>>> x
{5: 'abc', 7: 'def'}
>>> json.dumps(x)
'{"5": "abc", "7": "def"}'
>>> json.loads(json.dumps(x))
{'5': 'abc', '7': 'def'}
7

Yes this makes sense. I do think I need to clarify that the JSON Serialization format will not serialize this particular case to text since it is valid JSON in Prolog and could be confusing. I’ll create a pull request that adds a couple of items to the “Mapping Prolog Terms into JSON”:

  • Pointing out that dicts with keys that are integers can be converted to the Prolog JSON format, but won’t serialize to txt
  • Pointing out the (maybe obvious) point that there are objects like Streams that won’t convert at all

And then update the message format and toplevel differences section of MQI to point there.