Standalone 64-bit Linux Binary for SWI-Prolog Program


I am trying to get a standalone executable for my Prolog program on 64-bit Ubuntu. For this reason I am using the command:
$ swipl --stand_alone=true -o program_standalone -g main -c

According to the documentation page on Creating Executables in Linux, in order for this command to produce program_standalone that works in a Prologless environment without moaning about missing shared libraries, SWI-Prolog must be statically linked which should be the default in 32-bit Linux distributions. Since my system is 64-bit, I had to link it statically.

According to this Thread, I should pass -DSWIPL_SHARED_LIB=OFF to the building process to obtain the statically-linked SWI-Prolog I so long for.

So, I have included that juicy bit of information in the building process described in the page Installation on Linux, and the steps I performed (after obtaining a fresh copy today via git) are:

mkdir build
cd build
ctest -j 4
ninja install

However, after everything is done, I get the following.

$ ldd program_standalone (0x00007ffc63bd3000)` => /usr/lib/swi-prolog/lib/x86_64-linux/ (0x00007fde7107d000) => /lib/x86_64-linux-gnu/ (0x00007fde70c8c000) => /lib/x86_64-linux-gnu/ (0x00007fde70a62000) => /usr/lib/x86_64-linux-gnu/ (0x00007fde707e1000) => /lib/x86_64-linux-gnu/ (0x00007fde705c4000) => /lib/x86_64-linux-gnu/ (0x00007fde703a5000) => /lib/x86_64-linux-gnu/ (0x00007fde701a1000) => /lib/x86_64-linux-gnu/ (0x00007fde6fe03000) => /lib/x86_64-linux-gnu/ (0x00007fde6fbfb000)
/lib64/ (0x00007fde715e0000)

If I understand things correctly – which is doubtful; I should get no dependencies after the ldd command.

Where did I go wrong?

Many thanks.

I think when saving the program, you need to set foreign=save as well. I had this working at one point, but I don’t recall exactly how now…

What is “saving the program” in this context?

I was thinking running qsave_program/2, but the same thing as running the swipl command you gave above.

You can’t get a fully static binary, there is always a dependency on and the direct dependencies of (e.g. such as

The foreign=save option saves all other dynamic libraries in the resulting file (such as from packs or other libraries loaded with use_foreign_library/[1,2]).

I really wish we had the ability to produce truly static binaries. Deploying is always a nightmare with dynamic libraries.

There is some basic support for producing a static libswipl.a but it lacks most of the functionality because it can’t load packs or any of the other dynamic libraries where most of the functionality is.

1 Like

For building a fully standalone Python program, a modification to dlopen() was proposed … however, the change was sufficiently large (allowing dlopen to load from a file+offset, ideally with unzip), that it was rejected by the Linux community. I’m not sure what the current status is for Python; but if they couldn’t do it, there’s not a lot of hope for SWI-Prolog.

Maybe the solution is to just bundle the executable with a minimal set of .so files?

That’s what I’ve been doing; the statically-built version at least ends up with basically just, plus libc and the like, then I put that up and start it with a script that sets LD_PRELOAD_PATH

Ah, I didn’t know this. Very interesting. Fixing dlopen() to work with a file+offset would have been great. Do you have link to the place where this was discussed?

Sorry, I couldn’t find any references to the dlopen() discussion. When I think about it, the proposal was to replace the “file” argument in dlopen() with a file-like object.

There was a Pycon talk, but I couldn’t find much detail (and it probably didn’t mention dlopen): is similar to what Google uses now (without any changes to dlopen and without the hermetic capabilities).

Nothing really. The option avoids depending on, so you only depend on standard libraries. You’d have to use -static to gcc (if I recall well) to get a fully statically linked core executable. That is not in the CMake setup. You may consider creating a PR for that.

The problem @peter.ludemann is referring to is about loading foreign libraries and bundling these with the stand alone executable. That can be done with --foreign=save. This adds the .so file to the state and loads it by creating a temporary file, doing dlopen() and removing it. Python wanted a solution that did not require a temporary file. That doesn’t exist on Linux.

A further issue is whether dlopen() still works on a fully statically linked executable. I think -static kills dlopen. You should be able to link dynamically, but provide static linking for the less frequently used shared objects.

For short, in the current state you can create a one-file executable that depends on the system .so files. With some work you might be able to get one that only requires access to temporary files.

Thank you everybody.

So the conclusion is that I need to collect all the libraries that the pseudo standalone binary needs and make them available in the target deployment environment.

This is a sentence from Creating Executables in Linux: “If the SWI-Prolog kernel is statically linked (default on Linux/i386) and the state does not use external packages that provide shared objects, you are done”. I think I satisfy the constraints but I am not really done; so what/who is done in this context?

In case you’re wondering why this extra restriction (no temporary files) might be desired – it has to do with deploying executables in a data center with diskless machines where you can’t depend on having the runtimes already on the machines. (As to why “diskless” … that’s (a) complicated and (b) proprietary knowledge.)

My guess is that most SWI-Prolog programs don’t need to deal with this situation; but if you do need to deploy a program to 10,000 machines, you might want to talk with me. :wink:

I know :slight_smile: So, for swi-prolog you can create a single file read-only exe if, besides the Prolog core, you only have pure Prolog code. That requires some additional hacking in the CMake setup to link the main executable truely statically. If foreign code is involved you either need multiple files, have a writable filesystem or link the required foreign libraries statically into the main executable. The latter is doable, but complicated. It is probably doable to enforce some additional conventions and modify the CMake setup such that you can provide a list of foreign extensions that must be embedded statically.

I guess that all waits until someone needs it badly enough to do it or pay for it :slight_smile:

1 Like

You say “The option avoids depending on” but the OP’s executable still depends on This is the dependency that it is really important to get rid of because different distros have different swi-prolog versions with incompatible versions of this library. My solution is currently to include my with the executable and adjust the load path to point to it – is there a better one?

Provided you used -DSWIPL_SHARED_LIB=OFF during the CMake setup, there should not be a Are you claiming this is incorrect or are you using the default build/binary?

Sorry I didn’t realize this was in the context of having built
SWI-Prolog oneself, so I guess my system’s behaviour is correct.

  I recently had a problem due to the target system (ubuntu eoan)

not including – would this have been avoided if
building my own statically linked SWI-Prolog?

Nope. As noted earlier in this thread, -DSWIPL_SHARED_LIB=OFF only gets rid of the dependency. The normal Ubuntu libraries are still dynamically linked. Changing this requires changing the CMake config files.