Where is libswipl.a?

General
1

This documentation page says there should be a file at swipl/runtime/libswipl.a. When I build from source though, there doesn’t seem to be a directory called runtime and libswipl.a is nowhere to be found.
This is strange because I could swear that not too long ago (as in, somewhere in the past two years) I did do some work with this static library. Going back through swivm versions, the last build I still have a libswipl.a was 7.6.4 (though it’s not in a runtime directory).

Did libswipl.a get removed from the build since that version? Is there any way to get it back?

2

It is probably gone with the migration to CMake and nobody complained. Surely it should be possible to build it again. My CMake is getting a little rusty again :frowning: What is the use case? On x86 a static build was faster on Unix-like systems, but that doesn’t hold for X64 or any other modern CPU AFAIK. The only use case I see is using just the plain Prolog system (no extensions) to produce a single-file executable.

3

My use case is building a rust program that embeds swipl. In the rust world, static binding is the norm. Compiled binaries aren’t really supposed to have run-time dependencies. This allows binaries built on one system to run on another without too much hassle.
Unfortunately currently we can only do dynamic binding. This requires you to have swipl installed. Furthermore, libswipl.so (or the equivalent on not-linux) needs to be discoverable for the program to start. This is not always the case for every swipl install. It certainly isn’t the case on my own machine, where swivm manages all my swipl installs. To start binaries linked in this way, I need to specify LD_LIBRARY_PATH or do some post-build binary processing to hard-code the libswipl.so path.
It would be much nicer if I could just statically embed swipl.

To be honest, I don’t actually have any personal need for doing this right now. It’s more a thing that I feel should be possible as part of the rust prolog interface project (which is progressing pretty nicely by the way. I’ll probably post a preview later).

1 Like
4

Would that need to be a static library compiled as position independent, so you can link it into a dynamic library or would the preference be a really static library, i.e., one you can only link into an executable? There are very few downsides for the position independent version except for some performance degradation on old CPUs like x86.

5

Position independent is the more generic option, so I think that is better. While I doubt that anyone is going to statically embed swipl, only to then produce a dll again, you never know what people may need, so generic seems better.

6

Found Is it possible to get CMake to build both a static and shared version of the same library? - Stack Overflow. Notably the answer to use object libraries looks promising. I’ll give it a try.

1 Like
7

Ok. Created a branch static-lib at https://github.com/SWI-Prolog/swipl-devel/pull/new/static-lib. The basics are pretty simple. As usual Windows poses some challenges.

  • If we name the static library the same as the dynamic one we get two libswipl.lib files: one as import library for the dll and one as static library. One suggestion is to name the static library e.g. libswipl_static.lib. If we decide for that I guess it is best to do the same for all platforms. Opinions?

  • We use MinGW for compiling SWI-Prolog and we’ll stick with that for now as cross-compiling is a lot easier and faster and the resulting system is considerably faster. Also, while the core system can be compiled using MSVC, AFAIK nobody managed to get all dependencies together to create a complete system. The price of using MinGW that the resulting .a library seems to be unusable with MSVC. Even if the library can be converted (probably by unpacking and repacking using Microsoft tools), it requires the GCC runtime libs to be useful.

For short, it seems this is a no-brainer for POSIX systems but a big problem for Windows.

Note that The CMake config already allowed for -DSWIPL_SHARED_LIB=OFF to stop putting the core in a dynamic library. The branch adds an option -DSWIPL_STATIC_LIB=ON to always build the static library, also if the core library is dynamic.

How do you envision people using SWI-Prolog for Rust? Starting from a binary distribution? Please share your thoughts on all these questions.

3 Likes
8

Amazing, thank you! I’m building right now and I’ll try to incorporate this in my work tonight.
(In fact, while writing my reply the build finished. It looks like libswipl_static.a is being built, but it is not yet copied to the destination directory when I do ninja install. For now I’ll just copy manually.)

Regarding the library naming, I’m fine with either solution. As it is, I already need special build code to get things to build on windows, because libraries already end up in a different location (everything in the bin dir, instead of lib/<arch>/) and different environment variables are needed to locate them during build. Having to use a different library name just on windows is not an insurmountable barrier. But yes, consistency is good, so probably using the same name on all platforms is best.
Regarding MSVC, I don’t really have good opinions here. I personally do not care about MSVC as it is not part of my toolchain on windows. That said, I don’t know exactly how the rust toolchain on windows interacts with static libraries, so it may matter for me. I’ll have to test this.

Regarding rust, it may be useful to talk a bit about how the ecosystem works.

Rust development is centered around the cargo package manager. Cargo packages are entirely source based. People specify their dependencies in a Cargo.toml file, and the cargo tool downloads and compiles these dependencies at build time.

I’m developing the interface as a collection of crates, collected in the swipl-rs project. A low-level crate, swipl-fli, auto-generates bindings at compile-time from the SWI-Prolog.h header. A high-level crate, swipl, depends on this crate to implement an easy to use interface. Users of this interface would in turn depend on the swipl crate to build their own things.

So anyone that builds rust crates that depend on either the low-level swipl-fli or the high-level swipl crate will need to have SWI-Prolog installed, or their build will fail.

The primary use case for swipl-rs is implementing foreign libraries, ideally as part of SWI-Prolog’s pack system. So for that case the dynamic build is required. A secondary goal is supporting embedding of SWI-Prolog. Here either dynamic or static linking could be used, but I think static would really be preferred, as the resulting binary could then be used independently.

My plan is to support such a static build through a feature flag. Cargo crates can define a bunch of features that other crates can depend on. I intend to expose a static feature flag for people to opt into a static build.
For that to work though, I need to rely on the static library already being part of people’s installs. While I can probably document that people need a very recent version of SWI-Prolog for this library to be there, asking people to rebuild their SWI-Prolog from source won’t always be possible or convenient. That’s why I very much hope that libswipl.a (or libswipl_static.a) finds its way back into standard distributions.

9

Alright - trying to actually link against this library yields a lot of errors about not being able to find symbols from libgmp, deflate, and maybe some other stuff. Is this intended? Should it be the job of the swipl build to produce a libswipl.a that is already statically linked against these libraries, or is it the job of whoever links with libswipl_static.a?

Edit:
Having dug a little deeper, I needed to link against the following extra libraries to get something that’d run a minimal example (build a term hello(world) and print it with writeq/1 and nl/0).

  • gmp
  • curses
  • tinfo
  • z (zlib)

if I link against those (statically or dynamically) my basic example works.

Edit 2: And just to point out a huge flaw in my plan here - even when statically linking against the core, that doesn’t solve the dependency on a swipl install in a particular location. It looks like my example program won’t run unless the original install is present.

10

You may be interested in looking at this trick:

To allow dynamic libraries in statically linked code

11

Before changing anything I guess it is good to think about the pros and cons. Static linking of course avoids runtime link and version issues. The building is harder though as for a dynamic libswipl everything needed by the dynamic library is a dependency of this library. The set of dependencies of the dll/so/dylib differs between platforms and configuration options used when building SWI-Prolog. To figure this out one has three options, none of which is really up-to-date I’m afraid:

  • Use swipl-ld
  • Use swipl --dump-runtime-variables
  • Use a CMake export/import file

For short, linking to a static libswipl is considerably harder.

Next, even with a static libswipl applications may want to use libraries that have foreign components. These still need dynamic linking and on some platforms you cannot use these if the SWI-Prolog kernel is statically linked into the executable. This limitation holds for MacOS and Windows. On ELF based systems (e.g., Linux) the main executable needs to be linked using -rdynamic to make the PL_* symbols accessible.

On Windows we can ask for the absolute location of libswipl.dll, which is used to find the rest of the SWI-Prolog installation. On POSIX systems the system is in general not relocatable and the installation path is compiled into the library (typically you can make the system relocatable with some system dependent tweaks but that is another story).

As a side story a lot of my work now concerns ROS and they do everything using dynamic libraries. They choose exactly the opposite as Rust and decided to deal with them. So far “properly” IMO.

12

The more I look into this and the more you tell me, the more I’m becoming convinced that the static build is hardly worth it. The one benefit it’d give me right now is that on linux, it’d be able to build a binary that doesn’t have to locate libswipl.so at runtime, which means I don’t have to set LD_LIBRARY_PATH. There are other workarounds for this though, such as modifying the built binary to embed a load path.

The cost of the static build is that the build becomes more complicated. I’ve got this figured out for linux now (just link against those four libs that seemed to be missing, but is that all?), but since the resulting binary still depends on a swipl install being present at the hardcoded location, I’ve really not won much (though I guess this is configurable at startup).

As you describe it, it sounds like a static build is pretty crippled on most systems, with linux being an exception rather than the rule. With all that in mind, I don’t think supporting static builds has to be a priority.

What I really want is a single binary that embeds everything. qsave lets us do that if swipl is the main process, but as far as I’m aware, there’s nothing like qsave for the case where the main process is not swipl. Maybe there’s a way to generalize qsave to allow a different binary to take the role of swipl? Or maybe this is already possible?

13

Its not constant. libgmp is normally a dependency, but some people decide to build without, either for license reasons or because they want an as small as possible system. The terminal access libraries vary from system to system and linking may or may not need special linking options to support threads.

That is a hard issue, but there are solutions :slight_smile: . Normally, SWI-Prolog looks for the saved state at the back of the executable. It does so by trying to find the executable (easy on Windows, not so easy on POSIX systems but it will work most of the time), open the binary (may not be allowed on POSIX) and check for a zip archive at the end. Of course you can specify the location of the zip archive using -x. Finally, you can create the zip archive, turn it into a .o file, link it with the executable and tell the system to find its resources using PL_set_resource_db_mem()

The foreign extensions can be embedded using --foreign=save. I think this works on most systems. The result is a bit clumsy though as it writes the .so/.dll file to a temporary location and than loads the library. On POSIX system it immediately unlinks the file. That doesn’t work on Windows, so it merely tries to remove these files on shutdown (but may not get that far when crashed or killed ungracefully).

Note that the 8.x versions also allow accessing files in the zip file as normal files using res://<path>. This allows putting the entire library in the executable and load files from it.

14

@maren If your current opinion is that the dynamic route is best, I propose to merge the branch as the stuff to make a static library is there now, but by default is it not build. I guess that if it is build it should also be added to the installer (forgot that). Agree?

1 Like
15

Yes, sounds good. No sense in shipping a static library by default if it’s not going to work as people would expect on many platforms. I can leave static build as an experimental option in my code and instruct people to rebuild swipl if they really really want to.

1 Like
16

Thanks for the feedback. Pushed. Also updated the old manual section. Deleted most if it, merely leaving a summary of the argumentation from this topic.

2 Likes
17

A generic comment about static vs dynamic executables … I’ve worked at two large-scale internet providers; one used dynamic libraries (Yahoo!) and one used statically linked executables (Google). The dynamic libraries turned out to be a nightmare (ABIs, for example) – it wasn’t quite as bad as “dll hell”, but still pretty bad.(*)

Even something as simple as a standard Python interpreter turned out to be very problematic at Google, and eventually (after a lot of work), Python programs were able to generate self-contained statically linked (including the interpreter) executables(**) that could run on diskless servers (diskless turned out to be another issue because the Python scripts were packaged as zipped-archives, which presented problems with dlopen, especially after the maintainers of dlopen rejected a proposed modification that allowed reading from a file stream rather than specifying a file name).

So, if not now, I anticipate others will want statically linked executables.

(*) The situation at Yahoo! was made worse by using libraries where Google used servers. For example, at Yahoo!, to look up some data, you might call a function (deployed in a library) that did a SQL query, massaged the result and returned it; ad Google, you’d call a function (statically linked) that did an RPC call to a server (using protobufs) and waited for the result. Even if the server was on the same machine, Google would use an RPC (and with technologies such as Kubernetes, the distinction of same machine / remote machine became irrelevant).

(**) The one exception is libc, which is used everywhere on Linux systems and which can be assumed to preserve ABI compatibility when upgraded.

2 Likes
18

I concur with Peter. The biggest need for static binaries will rarely be felt by developers, but the pain is felt by those who have to deploy to many systems and architectures over which they have no control. SWI-Prolog is very close to a one-file-deployment scenario with saved states and --foreign=save, but still has dependencies on libswipl, libgmp, etc.

What I would suggest is the following:

  • enhance swi-prolog to truly provide one-file deployments (we are already very close with res:// and with saved states)
  • enhance load_foreign_files/3 and therefore use_foreign_libraries/2 so that they can run from statically linked libraries into the executable, this will allow people to produce one binary with all the packages, and libraries linked in.

This need for static binaries is one of the reasons why new languages like rust and golang are built in such a way that one can produce static binaries without much pain.

EDIT: BTW, this could open a path to build applications for Android by providing a static executable bundled with the android apk that communicates with the android app via sockets or stdin/stdout.

2 Likes
19

It is complicated stuff. There are technical as well as legal implications. For example libgmp is LGPL, which means your application must comply with the LGPL license if you link this statically. As the result is not a library you are practically bound to the GPL. Several languages are faced with this problem. There is no sensible alternative to GMP if you want fast, comprehensive and reliable big integer and rational number support (last time I checked about a couple of years back).

Second there are the platform aspects. On Windows using DLLs for deployment is fairly simple: you create a directory holding the executable and all dlls it requires except for the core Windows DLLs. There are zillions of Windows applications out there that have many DLLs. On MacOS you can get similar results using otool to prepare the application as a bundle Probably works great if you use native Mac development tools, but if you port from the POSIX world it is a nightmare. On Linux the current model is to use a “snap”, which provides somewhat comparable portability as docker images, but snaps are for applications and docker images are for servers (yip, this is only an approximation of reality).

Dynamic linking both creates problems and solves them. Yip, your executable is a single self-contained file with static linking and the fact that we have resource files. But, in Windows one can (AFAIK) not combine MinGW and MSVC static code in one process due to conflicting runtime library requirements. Based on dynamic linking we can assemble single-directory executables without a C compiler or linker. Using static linking requires a C compiler, linker and probably a uniform set of conventions for using CMake to make sure all dependencies are combined the right way.

That would be nice to have. One way is to dynamically lookup the init function and call it. Unfortunately that doesn’t work on all platforms for statically linked executables :frowning: An alternative is C++ global initializers. That too has its drawback AFAIK (or are these resolved)?

For short, it is a mess :frowning: I have the impression services are the main serious application domain for SWI-Prolog. Here, Docker is a reasonable choice. If getting SWI-Prolog installed on the target platform and the sources are public, distribution as Prolog source possibly extended with dynamic libraries often works fine.

Otherwise I think the simplest way to address this is a comprehensive description on how to handle this issue on the various platforms. As is, one need detailed knowledge of the way the dynamic linking platform finds libraries and symbols and the tools such as MacOS otool or Linux chrpath.

I’m happy to accept PRs that move in this direction as long as they do not bring too serious additional complications.

3 Likes
20

That should be a topic in the category Nice to know.