Click on a circle image of Jan W. (The below one is just for show)
or use this link to his profile page
Profile - jan - SWI-Prolog
Then click the Message button.
Work is progressing nicely 
It is now possible to access Python from multiple Prolog threads without crashes or deadlocks. I also improved calling Prolog from Python to also return a dict on failure, but with all Prolog variable bindings set to Python None. In addition, there is a status key that is one of True, False or 'Undefined', where the latter is a Python string that indicates the result is undefined according to the (tabled) Prolog Well Founded Semantics.
1 Like
It seems that the no-GIL Python is on the official roadmap.
Hereās a summary of the changes with --disable-gil builds, in which CPython will define Py_NOGIL:
Note that [the no-GIL CPython] will initially be an experiment. As such, the [Steering Committee] is reserving its right to drop the experiment if it becomes too burdensome to support (both in CPython and in the community in general, including lack of uptake). So for all of those who have been asking for the GIL to be removed, this is very much a āput up or shut upā situation for the community to show they support this.
Source: Brett Cannon: "The #Python steering council's decision on PEP 70ā¦" - Fosstodon
If I see updates on this, Iāll post them, but no promises that Iāll see everything. If there are questions about the āno-GILā version of CPython, thereās a fairly good chance that I know the people concerned and can get more information.
My interpretation is that we shouldnāt depend on a non-GIL version of CPython, and even if it does happen, itāll be several years at least before itās fully supported.
I suspect that the non-GIL CPython will use tcmalloc or something similar, to give higher performance with multi-threaded memory allocation. (Both Thomas Wouters and Brett Cannon work/worked at Google) If so, the bundled Python interface for SWI-Prolog might need to limit itself to tcmalloc.
I donāt know if this affects the Windows version of SWI-Prolog - however, Brett Cannon is now at Microsoft (as is Guido von Russum), so I presume that theyāll have a version of CPython that works well on Windows, although possibly with a different malloc library.
For now, the GIL/non-GIL is not that important. After some struggling it seems we can safely call Python from multiple Prolog threads but yes, the whole thing is serialized. My current understanding is that Python a single interpreter that can āhopā between threads. Actually it seems you can make multiple interpreters, but do not yet understand if (how) they cooperate. Anyway, even with the single interpreter it seems possible to free the interpreter from a thread while the thread is involved in long running non-Python code or a blocking operation. Is this picture correct?
If so, would this already work:
- (Prolog) thread 1
- takes the GIL
- calls Phython
- Python releases the GIL around blocking call or long running C/C++ code
If a second Prolog thread does the same, it seems both can make progress in the C/C++ code, no? Is this picture correct? If it is, the situation is far better than I feared as you typically want Python to access stuff that is not written in Python Is there a ready-to-use Python package to test this?
As a side note, malloc is for Prolog not a big issue. Most of the work happens on the Prolog stacks and these are private anyway. Most of the shared data structures (clauses, atoms, etc.) are accessed lock-free without any write operations. Only modification to these structures uses compare-and-swap and in some cases mutexes.
The preference for tcmalloc has to do with multi threaded programs creating a lot of temporary atoms or clauses. This leads to a rather uncommon pattern: Prolog threads do all he allocation and the gc thread does all the deallocation. Although I still do not understand the details, this causes processed under the Linux default ptmalloc implementation to grow indefinitely while leak analysis says there are no leaks. tcmalloc seems to get this type of workload correct.
1 Like
What Iām referring to is this from Initialization, Finalization, and Threads ā Python 3.12.0 documentation
Py_BEGIN_ALLOW_THREADS
... Do some blocking I/O operation ...
Py_END_ALLOW_THREADS
I think this allows other threads to get the interpreter while Do some blocking I/O operation operation is in progress. No?
SWI-Prologās PL_register_atom or PL_unregister_atom only apply to foreign code that wishes to ensure an atom stays alive. Atoms on the Prolog stacks are handled by GC. That also means that PL_unregister_atom() does not delete the atom if the reference count drops to 0. An atom is deleted if the garbage collector finds no references and the reference count is 0. Reference counting really hurts concurrency 
Update
Theresa, James and I resolved several of the data representation issues and decided to support both the original XSB interface that consists of more predicates and Python functions and my proposal based on experience with JavaScript. It seems we will end up with an interface that is sufficiently compatible to allow supporting both XSB and SWI-Prolog for typical use cases. SWI-Prolog does implement some extensions such as (now optionally) translate Python strings to Prolog strings, allow for full Unicode support including 0-codes, allow for unbounded integers and provide garbage collection.
I think the current version is ready for real usage. Be aware that minor changes are still possible and it is not unlikely that there are bugs ā¦
1 Like
The Janus interface package to Python now also got a setup.py, which allows installing it as a Python package and embed Prolog into Python. This is tested to work on Linux and MacOS (using Python 3.11 from Macports):
pip install .
After that, we can start python, load janus and have some fun:
python
>>> from janus import *
>>> [a["D"] for a in Query("between(1,6,D)")]
[1, 2, 3, 4, 5, 6]
>>> prolog()
?- version.
Welcome to SWI-Prolog (threaded, 64 bits, version 9.1.12-8-g70b70a968-DIRTY)
SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software.
...
?-
The interface is fully symmetric. Starting from both systems we can make mutually recursive calls on the other and we can run the interactive toplevel of the other system. As shown above, using prolog() from Python or ?- py_shell. from Prolog.
I have not tried to get this to run on Windows. I guess it should work, probably after minor changes. Iām not familiar with how Python extensions are normally handled on Windows. Related, while it is easy to set this up from source on a specific configuration, it is less clear how to make it flexibly using a binary distribution. In theory this should all be doable as both the Prolog and Python C APIs are pretty stable and thus the dynamic library sitting in between should be fairly independent of versions. If anyone has an idea on how to deal with that, please share. There must be similar cases.
Another issue that asks for some help is dealing with output under the swipl-win console. As is, the I/O of the embedded Python system is not connected. Can we somehow rebind the Python I/O to talk to the SWI-Prolog I/O streams?
1 Like
Congratulations, finally a tight integration! No more loose coupling, right?
No once_async() from within Python when calling SWI-Prolog
through the Janus Interface? It would execute a query once in
async modus. Same question about Query_async(), does this exist?
Then, abusing the API name consult(), which in Janus SWI-Prolog
does take a file name, but for exampe in Tau Prolog session.consult()
takes a Prolog text, so one can do, using JavaScript triple quotes:
session.consult('''
fruit(apple).
fruit(pear).
fruit(banana).
''').
Which doesnāt execute 3 goals, but instead asserts 3 facts as one is used
from the [user] directive. You can also dynamically assert rules not only
facts this way. Quite handy. Triple quotes are also available in Python.
At the moment, not. Iām not confident enough with Pythons async stuff, the C api and threads. Iām happy if someone explains how this should be done. Iām already happy that multiple Prolog threads may call Python without crashing
I donāt really call that abuse. consult/1 takes a file name. Tau normally runs in a browser and there file are fairly complicated. Still, given the """...""" notation, loading Prolog code from a string could be a nice feature. We need a good name and possibly pass an identifier such that passing the same identifier acts as a reconsult.
The abuse is on Tau-Prolog side not on Janus SWI-Prolog. Supplying a Prolog text
instead a file name into session.consult(). But they have a backdoor. You
can do the following either file system access or url connection access:
session.consult("", {file: "foobar.p"})
session.consult("", {url: "http://baz.org/foobar.p"})
But I didnāt try so much, I guess the file system access could be the route
for nodeJS, which is also supported by Tau Prolog, and the url connection
access for when in the browser. Does Janus SWI-Prolog and Python
share the same working directory access and modification?
With janus, prolog and python are in one process. So yes, they share working directory and file system.
I could not get this to work with Windows 10 and a recent install of SWI-Prolog from https://www.swi-prolog.org/download/devel
Welcome to SWI-Prolog (threaded, 64 bits, version 9.1.12)
SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software.
Please run ?- license. for legal details.
For online help and background, visit https://www.swi-prolog.org
For built-in help, use ?- help(Topic). or ?- apropos(Word).
?- py_version.
ERROR: Unknown procedure: py_version/0 (DWIM could not correct goal)
?- py_shell.
ERROR: Unknown procedure: py_shell/0 (DWIM could not correct goal)
?- pwd.
% c:/users/groot/documents/prolog/
true.
I donāt know if I missing understanding the instructions, need to wait for the nightly build to update, a Windows setting is not set, etc. There are too many variables that I donāt know for this at this time so donāt plan to try all the patterns; will be more than happy to check things on my Windows system and respond with info to resolve the problem.
Iām glad to hear that you had problems because I had them too 
but anyways maybe there is some kind of software I need to install first for things to work correctly (on MacOs). Iām focused on something else at the moment and I hope that if I try more systematically in the future it will get to work
Consider noting your problems then others here who know MacOS can try it and find solutions or confirm that they are seeing the same.
Ok. I will retry later and post the error message I was getting. It was something about wheel and some kind of subprocess that didnāt seem to work. But Iāll share the precise issue. Another thing I noticed is that Jan said he was using Macports, but on Macports I didnāt find the last Prolog release 9.1.12, but only the stable release 9.0.4 and a development release which had another number like 9.1.10. But ok, later Iāll share the error message I received.
Thanks for reporting. Seems something went wrong building the Docker image including Python for building the distribution. Rebuilding it now. The nightly build works. At least, for me given Python 3.11 installed and %PATH% setup to be able to find Python and the Python DLLs.
Some last things went wrong with the source tarball that prevented building on MacOS, so I did not push a new version for Macports As is, all should work if you install from source using Python and the Prolog dependencies either from Homebrew or Macports.
Likewise something went wrong building the PPAs for Ubuntu. They did compile the Python interface, but somehow it did not end up in the packaged version. I still do not know why
For short, from source all works fine, but all distributions failed That not uncommon with new dependencies. Iāll probably release 9.1.13 shortly. 13 must be a lucky number
2 Likes
Ok. This is the error message I get today (yesterday it was different)
marco@MacBookoBachini bin % pip install .
Processing /opt/local/bin
Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error
Ć python setup.py egg_info did not run successfully.
ā exit code: 1
ā°ā> [6 lines of output]
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 34, in <module>
File "/opt/local/bin/setup.py", line 35, in <module>
raise RuntimeError("At least SWI-Prolog version 9.1.12 is required")
RuntimeError: At least SWI-Prolog version 9.1.12 is required
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
Ć Encountered error while generating package metadata.
ā°ā> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
By the way in all this failure I casually stumbled upon a thing named spaCy (which of course you already know, I imagine) which seems to deal exactly with NLP and seems very interesting. It says it is āadvancedā and I doubt whether it is too advanced for me but Iāll give it at look nonetheless