I am trying to get a standalone executable for my Prolog program program.pl on 64-bit Ubuntu. For this reason I am using the command: $ swipl --stand_alone=true -o program_standalone -g main -c program.pl
According to the documentation page on Creating Executables in Linux, in order for this command to produce program_standalone that works in a Prologless environment without moaning about missing shared libraries, SWI-Prolog must be statically linked which should be the default in 32-bit Linux distributions. Since my system is 64-bit, I had to link it statically.
According to this Thread, I should pass -DSWIPL_SHARED_LIB=OFF to the building process to obtain the statically-linked SWI-Prolog I so long for.
So, I have included that juicy bit of information in the building process described in the page Installation on Linux, and the steps I performed (after obtaining a fresh copy today via git) are:
You can’t get a fully static binary, there is always a dependency on libswipl.so and the direct dependencies of libswipl.so (e.g. such as libgmp.so).
The foreign=save option saves all other dynamic libraries in the resulting file (such as from packs or other libraries loaded with use_foreign_library/[1,2]).
I really wish we had the ability to produce truly static binaries. Deploying is always a nightmare with dynamic libraries.
There is some basic support for producing a static libswipl.a but it lacks most of the functionality because it can’t load packs or any of the other dynamic libraries where most of the functionality is.
For building a fully standalone Python program, a modification to dlopen() was proposed … however, the change was sufficiently large (allowing dlopen to load from a file+offset, ideally with unzip), that it was rejected by the Linux community. I’m not sure what the current status is for Python; but if they couldn’t do it, there’s not a lot of hope for SWI-Prolog.
Maybe the solution is to just bundle the executable with a minimal set of .so files?
That’s what I’ve been doing; the statically-built version at least ends up with basically just libswipl.so, plus libc and the like, then I put that up and start it with a script that sets LD_PRELOAD_PATH
Ah, I didn’t know this. Very interesting. Fixing dlopen() to work with a file+offset would have been great. Do you have link to the place where this was discussed?
Sorry, I couldn’t find any references to the dlopen() discussion. When I think about it, the proposal was to replace the “file” argument in dlopen() with a file-like object.
Nothing really. The option avoids depending on libswipl.so, so you only depend on standard libraries. You’d have to use -static to gcc (if I recall well) to get a fully statically linked core executable. That is not in the CMake setup. You may consider creating a PR for that.
The problem @peter.ludemann is referring to is about loading foreign libraries and bundling these with the stand alone executable. That can be done with --foreign=save. This adds the .so file to the state and loads it by creating a temporary file, doing dlopen() and removing it. Python wanted a solution that did not require a temporary file. That doesn’t exist on Linux.
A further issue is whether dlopen() still works on a fully statically linked executable. I think -static kills dlopen. You should be able to link dynamically, but provide static linking for the less frequently used shared objects.
For short, in the current state you can create a one-file executable that depends on the system .so files. With some work you might be able to get one that only requires access to temporary files.
So the conclusion is that I need to collect all the libraries that the pseudo standalone binary needs and make them available in the target deployment environment.
This is a sentence from Creating Executables in Linux: “If the SWI-Prolog kernel is statically linked (default on Linux/i386) and the state does not use external packages that provide shared objects, you are done”. I think I satisfy the constraints but I am not really done; so what/who is done in this context?
In case you’re wondering why this extra restriction (no temporary files) might be desired – it has to do with deploying executables in a data center with diskless machines where you can’t depend on having the runtimes already on the machines. (As to why “diskless” … that’s (a) complicated and (b) proprietary knowledge.)
My guess is that most SWI-Prolog programs don’t need to deal with this situation; but if you do need to deploy a program to 10,000 machines, you might want to talk with me.
I know So, for swi-prolog you can create a single file read-only exe if, besides the Prolog core, you only have pure Prolog code. That requires some additional hacking in the CMake setup to link the main executable truely statically. If foreign code is involved you either need multiple files, have a writable filesystem or link the required foreign libraries statically into the main executable. The latter is doable, but complicated. It is probably doable to enforce some additional conventions and modify the CMake setup such that you can provide a list of foreign extensions that must be embedded statically.
I guess that all waits until someone needs it badly enough to do it or pay for it
You say “The option avoids depending on libswipl.so” but the OP’s executable still depends on libswipl.so. This is the dependency that it is really important to get rid of because different distros have different swi-prolog versions with incompatible versions of this library. My solution is currently to include my libswipl.so with the executable and adjust the load path to point to it – is there a better one?
Provided you used -DSWIPL_SHARED_LIB=OFF during the CMake setup, there should not be a libswipl.so. Are you claiming this is incorrect or are you using the default build/binary?
Nope. As noted earlier in this thread, -DSWIPL_SHARED_LIB=OFF only gets rid of the libswipl.so dependency. The normal Ubuntu libraries are still dynamically linked. Changing this requires changing the CMake config files.
So, for swi-prolog you can create a single file read-only exe if, besides the Prolog core, you only have pure Prolog code. That requires some additional hacking in the CMake setup to link the main executable truely statically. If foreign code is involved you either need multiple files, have a writable filesystem or link the required foreign libraries statically into the main executable. The latter is doable, but complicated. It is probably doable to enforce some additional conventions and modify the CMake setup such that you can provide a list of foreign extensions that must be embedded statically.