I am trying to replace pyswip with swiplserver/PrologMQI in our project. However, I have performance issues. Swiplserver is at least 25% slower than pyswip in my use case, but often 2-4x times slower.
I have found that some things help, such as batching calls as much as possible. However, even in the simplest case of a single call, swiplserver/PrologMQI is still slower.
Is this issue known? Is there any way to improve the performance or suggested tips? I could not find another mention of it in the forum.
Thanks,
Andrew
EDIT: I have attached a Python script and a Prolog file that benchmarks the performances. My actual use case is, however, much slower.
MQI is only a few months old. (history) So you are on the bleeding edge.
As I have not used it I can not help but didn’t want you to think this was being ignored.
Sometimes the creator @ericzinda will check this site and respond.
If possible for you, try to use another, simpler messaging system for comparison – i am looking at websockets and, so far, it seems to work well for me.
That is surely to be expected. pyswip combines Prolog and Python in one process and performs direct calls between the two. MQI runs Prolog as a separate process using network communication and (de)serialization of messages passed between the two. Where pyswip has several problems wrt. stability, thread handling and installation trouble, MQI resolves most of these but at a much higher call overhead.
When using MQI, you typically want to define Prolog predicates that allow making as few as possible calls to Prolog. So, instead of all the individual steps, just create a Prolog predicate that takes all inputs as arguments and provides the results as an answer. Now you only have performance problems if this is not possible because the computation is spread over Python and Prolog at a low level of granularity and that cannot be changed for example because you need Python to fetch information from some system for which there is no Prolog interface as a high frequency.
For short, there are scenarios where pyswip is the best choice. Many tasks can be redesigned to reduce the impact of the higher call cost. That typically implies more Prolog programming though
Sorry I was on a long vacation or I would have responded. Great summary @jan! Yes, I was motivated to build the MQI interface after hitting difficult installation, a few bugs and missing features with pyswip. After struggling to update pyswip to address these myself, I wanted a more “loosely coupled” approach that would be easier to adjust to my (and others’) needs over time and not require as much work to maintain or track SWI Prolog features. I was also hoping for a design that would be simpler to integrate into other languages.
This approach does have a performance tradeoff as @jan mentions, however.
It’s not clear to me that using “network” and serialization/deserialization is necessarily a big performance bottleneck. That’s what Google uses, and they care a lot about performance(*) – and their primary way of combining programs is by putting each service element in its own “chroot jail” or container, using TCP (including local loopback or connecting over a high-speed network) and protobufs. The standard Google server uses a heavily multi-threaded HTTP-ish server on a multi-core machine with callbacks.
[The alternative, using direct linking, leads to “DLL hell” (or “.so hell”) – and I’ve experienced that at another large Internet company]
(*) The C++ compiler optimization group at Google is funded by the reduction in machines resulting from improving one program’s performance by about 1% each year.
