A bundled Python interface

Request For Comments
1

Theresa Swift from XSB has convinced me that a bundled Python interface has a lot of value. She presented a paper on Janus, the XSB Python interface. The main goal is to access Python transparently and hassle-free from Prolog and thus allow transparent access to the wealth of Python interfaces. Porting this work is an obvious choice. However, the foreign API differs and, SWI-Prolog provides additional datatypes, notably dicts to make the binding much more intuitive. Further more, I recently created the JavaScript interface and that gave some new insights.

If you check out the latest git source, run gity submodule update --init and build it, it should add the package swipy (GitHub) and if Python is installed with the development files, all should configure nicely. After building, you can run Python interactively under Prolog using

?- py_shell.

You can also call e.g. ?- py_call(print("Hello world\n"), _).

See examples in packages/swipy and the very short docs built in builtdir/packages/swipy/janus.html

There are a lot of issues and questions. It is, at least on some platforms, already useful as is though :slight_smile:

  • Naming and compatibility to the XSB version. Having a shared Python interface would be nice, probably next to the more natural SWI-Prolog interface.
  • (How) can we support big integers and rational numbers?
  • Code is probably buggy at places. I’m still struggling with the Py_DECREF(), etc.
  • Barely any comment in the code yet.
  • I wonder whether calling Prolog from Python is what we want. Currently it provides a module janus with some functions and a class Prolog that provides an iterator over solutions.
  • Currently it binds to the Python discovered by CMake. Probably you want to be able to select a Python.
  • Eventually there should also be a pip install swipy. Little clue what should be in there.
  • Surely much more.

So, Python programmers, please comment and, if possible, help.

3 Likes
2

At the risk of volunteering for too much work, I’ll admit to having been part of Google’s core Python team for 5 years. It’s been at least 5 years since I’ve done any foreign functions in Python, but I think I can cope. (I also know a few of the “core developers”, although we haven’t kept in touch since I left Google.)

An alternative API is SWIG, which provides a more limited functionality but is easier to use (there are also SIP and boost.Python; also Python’s ctypes and cffi). Google found some serious problems with SWIG and implemented a replacement that was less error-prone and had more functionality: CLIF (or possibly CLIF) – I know the original author and could probably ask questions.

Protobufs are an alternative way of passing data between Prolog and Python.

One thing to watch out for is that there is ongoing work to remove the GIL (Global Interpreter Lock), which is one of the major things preventing Python from taking better advantage of multiple cores. I don’t know how this work would interact with SWI-Prolog’s multi-threading.

Also, in addition to reference counting, Python has a garbage collector, mainly to deal with circular pointers. I don’t know the details, nor how that might interact with SWI-Prolog’s garbage collector.

CPython is the “reference” implementation of Python, but there are others, such as PyPy, Cython, Jython, IronPython (and I think that there are some minimalist versions for embedded systems) – these all of some amount of incompatibility but typically give better performance (e.g. PyPy has a JIT compiler).

And there’s work at Microsoft (led by Guido Van Rossum) on improving Python’s performance by a factor of 5x (IIRC); and I think I’ve heard of proposed changes to the API for foreign functions.

2 Likes
3

Thanks Peter, I think you are already doing enough :slight_smile: Advice takes less time and already help a lot. At some point there was a binding for SWIG for SWI-Prolog. I don’t know whether it still exists. I wasn’t very impressed. We also have "ffi" pack for SWI-Prolog. It has advantages and disadvantages. It surely is still a lot harder to use than a Python binding, in particular because knowledge about Python is now much wider spread than knowledge on how C APIs are to be managed. Calling e.g. sin() is a common example, but real C APIs require input and output structures that need to be allocated and discarded. The overhead of the ffi pack is significant, notably due to the memory management that most APIs require.

Note that we also have MQI (using networking) and there is pyswi, binding Prolog to Python using Python’s ffi. That is rather incomplete though.

So, I think a clean Python binding (and I think I’m fairly close to that) is a good approach. It will make connecting Prolog to just about anything a lot more accessible for many users. It is probably also a good alternative for writing an application with a gui interface.

The GIL is a nasty problem. The current prototype is single threaded only. Adding the GIL is not that hard, but lack of proper multi-threading support reduces the value of the Python interface for certain applications significantly. It being removed is good news.

Using SWI-Prolog’s blobs we can make GC of Python object work, as we also do for JPL and JavaScript (WASM). Here again we have a bit of a GIL issue as Prolog’s atom-GC runs in another thread.

Do the various different Python implementations share the same C api? Or at least, “mostly”?

P.s. The Python C API is a lot harder to use than SWI-Prolog’s one. I also noted that when writing the ros2 interface, which at the low level is basically a port of the Python ros2 binding. The reference count management complicates the code a lot compared to the term_t indirect handle approach of SWI-Prolog (copied from Quintus) that only require the C function to allocate some term_t handles and after that the C code doesn’t need to worry about GCm stack shifts, reference counts, etc. The Prolog interface is also faster and truly multi threaded :slight_smile:

4

I have experience with C/C++/Python interfacing. I’ve used the Python/C API directly as well as higher level frameworks (Boost Python/PyBind11). I was mostly calling C/C++ from Python though not the other way around.

Have you used PyBind11? I was impressed with its capabilities. I didn’t use the “embedded python” feature but it does have one:
https://pybind11.readthedocs.io/en/stable/advanced/embedding.html

-James

5

No. Simply the Python/C API. Note that we need little:

  • Convert Python data to Prolog
  • Convert Prolog data to Python
  • Implement py_call/2 that calls a Python function/method from Prolog
  • Implement opening, iterating and closing a Prolog query from Python.

The total is less than 1,000 lines of C. Its not complete, but I think it is probably 90% of the code we’ll ever need. It mostly needs more testing (notably memory management), support for threads and more flexible building and packaging. As is, it only supports building from source in an environment that has CPython with the C development stuff installed. On a Linux system it is really easy to build. Works also for MacOS with Python installed from Homebrew.

What matters is (1) portability and (2) best performance.

Where does that leave pybind compared to the C API? Surely Boost adds a lot to the build dependencies. Does it make a difference for the various Python versions @peter.ludemann mentioned?

6

The web page says it supports a range of Python versions, but I’ve only used it for one version at a time. Also, just to clarify, PyBind11 is not part of Boost. I mentioned them together because it is derived from Boost, but part of why it was developed as a separate project is because people didn’t like having to install all of Boost. It has the nice higher level features of Boost but PyBind11 is alot smaller.

See https://github.com/pybind/pybind11

I would say it’s worth a look, it might help your use case.
James

7

Thanks. No Boost is a relief :slight_smile: @peter.ludemann would probably go for this. I typically like C better than C++. We’ll see where it goes. The current implementation is basically a rewrite in C of what I did about 6 years ago to evaluate the SWI-Prolog ffi interface. That took just half a day or so, so moving to pybind if that is useful won’t be too hard either :slight_smile:

8

Is that a ros2 interface for SWI? Is that publicly available? I had a look at the documentation and packages but I couldn’t find anything.

9

Yes. See GitHub - SWI-Prolog/rclswi: ROS2 bridge for SWI-Prolog

10

It’s not clear to me that the GIL will be removed. There was an attempt to get rid of it completely (similar to how Java got rid of the locks in various collection classes), but I don’t think it was successful – I can ask if you wish. I think that the current approach is to add “subengines”, so similar to Prolog engines, but without fine-grained multi-threading.

Python also has “async/await” and “generators” or “iterators” (with the yield keyword). For example, range(0,5) doesn’t return a list but returns something with a __next__ method that returns 0, 1, 2, etc. If you want a list, you need to pass this to the list constructor:

>>> (z % 2 for z in range(0,5))
<generator object <genexpr> at 0x7b8ce2da3d30>
>>> list(z % 2 for z in range(0,5))
[0, 1, 0, 1, 0]

Python generators fit naturally with the “get next result” from a Prolog query, and the Prolog class in janus.py appears to do that. (I think it’s not quite right: the __next__() method should call close_query() at the end because there’s no guarantee about when the __del__() will be called, even though typically it’s called as soon as the reference count for the Python object becomes zero.)

Python generators also provide an obvious way to get new results from Python by backtracking, if that’s a desirable programming model. Note that a Python generator can only be iterated through once (there is a “tee” function for duplicating a generator).

I don’t think so, unless you restrict yourself to CFFI or CTypes: Writing extension modules for pypy — PyPy documentation
(PyPy is probably the second most used implementation; it tends to lag a version or two behind CPython; and other implementations might have other restrictions/extensions.)

SWIG is mainly for wrapping existing foreign code and making it available to Python, Java, etc. (and Prolog). It’s not clear to me that’s what you want – your use case is Prolog-calling-Python and Python-calling-Prolog, correct? (That’s my interpretation of A bundled Python interface - #5 by jan)

CLIF is better than SWIG, and should be preferred if you need similar functionality (even though SWIG is more commonly used; and I don’t know if CLIF can be used as-is for Prolog). I don’t know about boost.Python/PyBind11.

The main advantage of using the C++ interface is that it takes care of the tedious “check whether the PL_*() call raised an error” code (similar to Go programming with f, err := some_call(...); if err != nil { ... } sprinkled all over the place). [We might want to make a C version of SWI-cpp2-plx.h that makes it clear whether the function can raise an error, and simplifies the error handling in C code.]
C++ also has strings, and the API takes care of things like PL_STRINGS_MARK(); it also makes cleanup of failed unification easier (Go’s defer makes cleanup easier than in C).

Having said that, the C++ interface has some restrictions when used outside the context of PREDICATE(), so C might be the safest way of doing things. :slight_smile:

11

Fantastic, thanks! This may well be useful in my research. If so I will let you know and might have time to contribute.

… at last :slight_smile:

12

It’s easier to use than Boost also - cleaner and fewer layers of classes/templates to work through. It’s pretty widely used.

For generators/iterators it would probably work to add bindings for the iter, next methods. It seems like right now it doesn’t directly have support for coroutines and async programming.

I haven’t used SWIG/CLIF - it seems like one advantage they have is they generate bindings with languages other than Python whereas PyBind is C++ <=> Python only.

13

Anyway maybe I can help in some way - let me know.
James

14

swipl.next_solution() already closes the query and makes sure this is reflected in the state, so the close_query() on the same state is a no-op.

Not really. In practice, you get away most of the time using this because every function returns TRUE on success and FALSE otherwise, raising an error when appropriate.

   return ( PL_*() && PL_*() && ...)

C++ is a bigger advantage for Python as you need to de Py_CLEAR() on each object you got from Python.

Mostly by reviewing the code. I haven’t much experience with the Python C binding, not with Python programming in general. So, the entire style and what it looks like from Python may be poor.

I pushed a lot more code. I think I’m approaching a stable state. The interface is far from stable though as I hope to resync some of the interface style with XSB. So, don’t start using this except for tests unless you are prepared for some editing on later releases.

One problem is that I added support for the GIL, but it deadlocks :frowning: I have little clue what I’m doing wrong. It could be related to this Python bug Deadlock when calling PyGILState_Ensure() from a fresh C thread · Issue #96071 · python/cpython · GitHub My Python version is 3.10.6 (from Ubuntu 22.04)

Given Python’s option to yield, I wonder how hard it would be to implement non-deterministic Prolog predicates using this? Note that the way this is handled in C is that the nondet predicate is associated to a C function. This function is passed an extra parameter that tells the function what type of call it is (first, redo, cut). On the first call and from the redo calls, you may return a pointer to a state. If you do, a redo of the predicate calls the function with the redo option and the pointer to the previous state. On a cut, you cleanup given the passed state.

15

A long time (20 years?) I interfaced C++ CORBA code with SWI-Prolog and got it working, but it was a major pain and every time I needed to bind a new interface I had to add code and rebuild everything. At some point later you suggested that a nicer approach would be an IPC mechanism instead of directly linking the code. Eventually I did just that and now I use MQTT messages to communicate back and forth between SWI and Python, Java, even C++ on Arduino environment microcontrollers and I’ll say my life has been better ever since. I’m really enjoying the recent Redis support in SWI BTW.

So when I hear about a native Python interface like you’re describing I feel the pain in my bones.

16

Hi Steve, Thanks for this insight. For a long time, I shared your opinion. To some extend I still do. On the other hand, we have to reach out to other IT components and our primary tool to do so is/was C(++). It appears that C(++) programmers capable of understanding a C(++) API and designing a bridge between that and Prolog are scarce in general and particularly among Prolog programmers.

At the same time, the facilities for SWI-Prolog to cooperate with high level languages as well as the facilities to do so for high level languages like Python have improved quite a bit. Although not particularly easy, it seems to be possible to create a robust interface to Python. What also changed is that the quality of these platforms have improved a lot, so now we can create a robust joined system rather than a shaky one that was very hard to debug.

The Python binding allows for a much wider audience to create bridges between Prolog and other IT components. As Python is, like Prolog, dynamically typed, such an interface is far less work than a C(++) based one. SWI-Prolog’s binding provides Python object garbage collection over the shared process.

Unfortunately Python’s multi threading is not very good and most libraries of Python itself use Python ctypes, C or C++. So, while going through Python is a lot easier, the result is typically slower and doesn’t scale when Prolog’s multi threading is used to exploit more cores. Notably connections that rely on the exchange of a lot of data and connect to CPU intensive tasks behind Python that you want to run on multiple cores are probably better coded using SWI-Prolog’s native C(++) interface than going through Python. Still, now you can get started quickly and when Python becomes the bottleneck, create a direct C(++) based interface.

2 Likes
17

Status update

It appears the system builds nicely on Windows too. Windows Python has been added to GitHub - SWI-Prolog/docker-swipl-build-mingw: Docker to cross-compile SWI-Prolog for Windows I triggered a build and the 64 bit version of SWI-Prolog from Download daily builds for Windows seems to work fine if Python 3.11 is installed in %PATH%.

Discussion with Theresa Swift from XSB is ongoing to try and keep the XSB and SWI-Prolog version as close as possible. Be aware that Python may look different from Prolog in new releases of this package!

1 Like
18

Can you explain that please.

19

The API is not settled. That applies to nearly every aspect of it. The overall design is not likely to change, but predicate names, data representation and Python functions are likely to change. Although the design is the same as the XSB original, naming and data representation differs considerably. We will try to sync that as far as possible. Probably the result will have two faces, one restricted to standard Prolog and one using e.g., SWI-Prolog dicts and strings.

Anyone, notably with experience in Prolog, Python and language interfaces, is cordially invited to join in this discussion. Please drop me a personal note.

1 Like
20

I couldn’t find any private message link in discourse but I’d be interested.