Minimal build with STATIC_EXTENSIONS

I try to build a minimal relocatable build of SWI Prolog 9.0.4 on Linux, which contains only a few needed packages. I found the STATIC_EXTENSIONS switch to create only one executable, but the linking of swipl fails with following error:

[ 48%] Linking C executable swipl
/usr/bin/ld: libswipl.a(pl-cstack.c.o): in function `print_trace.part.0':
pl-cstack.c:(.text+0xb7): undefined reference to `dladdr'

I configured the build using

../swipl-9.0.4/build$ cmake -DSTATIC_EXTENSIONS=TRUE -DUSE_TCMALLOC=OFF -DINSTALL_DOCUMENTATION=OFF -DSWIPL_SHARED_LIB=OFF -DSWIPL_STATIC_LIB=ON -DBUILD_SWIPL_LD=OFF -DSWIPL_PACKAGES_BASIC=OFF -DSWIPL_PACKAGES_ARCHIVE=OFF -DSWIPL_PACKAGES_ODBC=OFF -DSWIPL_PACKAGES_BDB=OFF -DSWIPL_PACKAGES_PCRE=OFF -DSWIPL_PACKAGES_YAML=OFF -DSWIPL_PACKAGES_JAVA=OFF -DSWIPL_PACKAGES_SSL=OFF -DSWIPL_PACKAGES_TIPC=OFF -DSWIPL_PACKAGES_QT=OFF -DSWIPL_PACKAGES_X=OFF -DSWIPL_PACKAGE_LIST="clib;sgml" ..

Have I forgotten a switch?

EDIT: I think I have found the reason. cmake/Config.cmake contains following lines:

if(NOT STATIC_EXTENSIONS)
check_library_exists(dl	dlopen	      "" HAVE_LIBDL)
endif()
[...]
if(HAVE_LIBDL)
  set(CMAKE_REQUIRED_LIBRARIES ${CMAKE_REQUIRED_LIBRARIES} dl)
endif()

So if STATIC_EXTENSIONS is TRUE, the configuration defines that libdl is not needed, which is obviously not the case. So, I need to activate the switch HAVE_LIBDL manually to get the linkage working.

1 Like

Thanks. Seems to depend whether you link to the static or dynamic glibc. I’ve added an #ifdef. Also added a bit simpler way for explicit packages, e.g.

make -DSTATIC_EXTENSIONS=ON -DINSTALL_DOCUMENTATION=OFF -DSWIPL_PACKAGE_LIST="clib;plunit" -G Ninja ..
1 Like

Thanks for fixing.

I have two small further hints relating to that:

  • The SWIPL_PACKAGE_LIST option is not documented properly in CMAKE.md and on the website. It mentions a switch SWIPL_PACKAGES=List, but it should be SWIPL_PACKAGE_LIST=List, I think.
  • If STATIC_EXTENSIONS is switched on, the installed executable will contain a RUNPATH entry to <install-path>/lib/swipl/lib/x86_64-linux, although there are no shared libraries to load on that path.

Thanks. Fixed.

Hmm. Weird. It creates static libraries and then links an executable from the main program and the static libraries. Why would cmake add a RUNPATH? As far as I can tell, the cmake scripts of Prolog do not mention RUNPATH.

I tried the cmake setup above on msys2

../swipl-9.0.4/build$ cmake -DSTATIC_EXTENSIONS=TRUE -DUSE_TCMALLOC=OFF -DINSTALL_DOCUMENTATION=OFF -DSWIPL_SHARED_LIB=OFF -DSWIPL_STATIC_LIB=ON -DBUILD_SWIPL_LD=OFF -DSWIPL_PACKAGES_BASIC=OFF -DSWIPL_PACKAGES_ARCHIVE=OFF -DSWIPL_PACKAGES_ODBC=OFF -DSWIPL_PACKAGES_BDB=OFF -DSWIPL_PACKAGES_PCRE=OFF -DSWIPL_PACKAGES_YAML=OFF -DSWIPL_PACKAGES_JAVA=OFF -DSWIPL_PACKAGES_SSL=OFF -DSWIPL_PACKAGES_TIPC=OFF -DSWIPL_PACKAGES_QT=OFF -DSWIPL_PACKAGES_X=OFF -DSWIPL_PACKAGE_LIST="clib;sgml" ..

and I got some undefined references to imp_getaddrinfo and a few more when building swipl.exe and swipl-win.exe

cmd.exe /C "cd . && C:\msys64\mingw64\bin\cc.exe -O2 -gdwarf-2 -g3 -Wl,--stack,4000000  -municode src/CMakeFiles/swipl.dir/pl-main.c.obj -o src\swipl.exe -Wl,--out-implib,src\libswipl.dll.a -Wl,--major-image-version,0,--minor-image-version,0  src/libswipl.a  -lwinmm  -lws2_32  -lpsapi  C:/msys64/mingw64/lib/libgmp.dll.a  C:/msys64/mingw64/lib/libz.dll.a  -lm  -lws2_32  -lpsapi  C:/msys64/mingw64/lib/libgmp.dll.a  C:/msys64/mingw64/lib/libz.dll.a  -lm  packages/clib/memfile.a  packages/clib/files.a  packages/clib/uri.a  packages/clib/readutil.a  packages/clib/prolog_stream.a  packages/clib/md54pl.a  packages/clib/crypt.a  packages/clib/hashstream.a  packages/clib/sha4pl.a  packages/clib/cgi.a  packages/clib/time.a  packages/clib/process.a  packages/clib/streaminfo.a  packages/clib/socket.a  -lgdi32  packages/sgml/sgml2pl.a  -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 && cd ."
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: packages/clib/socket.a(socket.c.obj): in function `pl_gethostname':
C:/msys64/home/Matthias/swipl-devel/packages/clib/socket.c:1004: undefined reference to `__imp_gethostname'

The problem can be circumvented by adding “another” -lws2_32 to the end of the command. I guess the order of the libraries matters, and a proper fix would be to add ws2_32.lib to the libraries for package/clib.

Edit: It seems possible to fix this problem by changing the order of the socket libraries under WIN32, no idea why this is relevant:

if(WIN32)
  set(SOCKET_LIBRARIES gdi32.lib ws2_32.lib)
  set(HAVE_SOCKET 1)
else()

Hmm. Weird. It creates static libraries and then links an executable from the main program and the static libraries. Why would cmake add a RUNPATH? As far as I can tell, the cmake scripts of Prolog do not mention RUNPATH.

I think it is because of the explicit setting of set(CMAKE_INSTALL_RPATH "${SWIPL_ABS_INSTALL_ARCH_LIB}") in cmake/Install.cmake. I don’t know why it is set twice in this file. Once unconditionally and a second time if the lib path is not in the “cmake platform implicit link directories”.
I also do not know, why cmake creates RUNPATH entries instead of RPATH entries. Perhaps it uses RUNPATH as soon as it is available on that platform.

I can get rid of the problem of the undefined references like this (package/cmake/PrologPackage.cmake)

old (line 188+)

    if(STATIC_EXTENSIONS)
      target_link_libraries(libswipl PRIVATE ${foreign_target} ${v_c_libs})
      set_property(GLOBAL APPEND PROPERTY static_extension_libs ${v_module})
    else()

new

    if(STATIC_EXTENSIONS)
      target_link_libraries(libswipl PRIVATE ${foreign_target})
      target_link_libraries(${foreign_target} PRIVATE ${v_c_libs})
      set_property(GLOBAL APPEND PROPERTY static_extension_libs ${v_module})
    else()

such that swipl.exe depends on libswipl.a, which depends on, say zlib4pl.a, and the last one depends on libz.a.

In the old version, libz.a is sometimes shifted before zlib4pl.a in the link list, which gives undefined references under Windows.

Still busy with the static build: In the current swipl-devel, I get an undefined reference to install_test_cpp (Linux and Windows).

cmake -DSTATIC_EXTENSIONS=TRUE -DINSTALL_DOCUMENTATION=OFF -DSWIPL_SHARED_LIB=OFF -DSWIPL_STATIC_LIB=ON -DBUILD_SWIPL_LD=OFF -DSWIPL_PACKAGES_X=OFF ..
make (or ninja)

That is the error

[ 24%] Linking CXX executable swipl
/usr/bin/ld: libswipl.a(pl-load.c.o):(.data+0x178): undefined reference to `install_test_cpp'
collect2: error: ld returned 1 exit status
make[2]: *** [src/CMakeFiles/swipl.dir/build.make:166: src/swipl] Error 1
make[1]: *** [CMakeFiles/Makefile2:1962: src/CMakeFiles/swipl.dir/all] Error 2
make: *** [Makefile:166: all] Error 2

I guess you should disable these tests. As there is no dynamic loading, the extension interface is not relevant anyway and including its tests surely is not.

Unsure what is the problem (Ubuntu, current swipl-devel, note the new switch BUILD_TESTING=OFF):

$ cmake -DSTATIC_EXTENSIONS=TRUE -DINSTALL_DOCUMENTATION=OFF -DSWIPL_SHARED_LIB=OFF -DSWIPL_STATIC_LIB=ON -DBUILD_SWIPL_LD=OFF -DSWIPL_PACKAGES_X=OFF -DBUILD_TESTING=OFF ..
$ make
...
[ 27%] Linking CXX executable swipl
/usr/bin/ld: libswipl.a(pl-load.c.o):(.data+0x178): undefined reference to `install_test_cpp'

I don’t know. It should not try to build packages/cpp anyway with -DSWIPL_PACKAGES_X=OFF. The static extensions isn’t worked out well. It is mostly there to support extensions for the WASM version. Eventually it should disable the stuff that is not compatible with it.

Is this correct? In cmake/PackageSelection.cmake, cpp is among the list of “basic” packages (“X” only has xpce).

Anyway, I can move on with

cmake -DSTATIC_EXTENSIONS=TRUE -DINSTALL_DOCUMENTATION=OFF -DSWIPL_SHARED_LIB=OFF -DSWIPL_STATIC_LIB=ON -DBUILD_SWIPL_LD=OFF -DSWIPL_PACKAGES_X=OFF -DBUILD_TESTING=OFF -DSWIPL_PACKAGE_LIST="chr;clib;clpqr;inclpr;http;mqi;ltx2htm;nlp;paxos;redis;stomp;PDT;pengines;pldoc;plunit;protobufs;RDF;semweb;sgml;sweep;table;utf8proc;zlib" ..

Sorry, missed the _X. Surely this cmake command must be simplified. Using -DSWIPL_PACKAGE_LIST=… is surely the way to go to get specific packages.

I think I found the way to fix the problem with the linker dependencies (Windows, static extensions), see PR 1146

As a temporary workaround, I just cleared the content of cmake/Install.cmake to get rid of the RUNPATH entry. But this is of course not the general solution.

I pushed f817bbb7d3989c72a2101014338443cd14728f75 which skips all the RPATH stuff if the core is not build as a shared object. I think that is fine. Remains hard to access though. There are so many build targets and configurations that it is nearly impossible to validate all of these :frowning:

I don’t know if that is correct. SWIPL_SHARED_LIB / SWIPL_STATIC_LIB only has an influence to the libswipl.*, isn’t it? So the packages are still shared libraries, which need to be found by the swipl executable using RPATH. So the correct condition would be NOT STATIC_EXTENSIONS, I think.

But CMAKE_INSTALL_RPATH_USE_LINK_PATH perhaps also need to be set anyways to get RPATH for external libraries not in the standard paths.

I think it is :slight_smile: Without libswipl.so/dylib/…, you can only use foreign extensions on ELF based systems as ELF is the only binary format that uses a process wide global dynamic symbol table to resolve the PL_* functions ((X)COFF resolves against other shared objects). So, not ELF and no shared kernel means no extensions. Using ELF, we can have extensions, but these are resolved against the process’ symbol table and we do not need to find libswipl for that.

So far I do not see a configuration where this fails (but, there are many …)

Just to confirm (Windows):

cmake -DSWIPL_SHARED_LIB=OFF -DSWIPL_STATIC_LIB=ON ..
ninja
src/swipl

?- use_module(library(archive)).
use_module(library(archive)).
ERROR: Exported procedure archive:archive_set_header_property/2 is not defined
ERROR: Exported procedure archive:archive_open_entry/2 is not defined
ERROR: Exported procedure archive:archive_next_header/2 is not defined
ERROR: Exported procedure archive:archive_close/1 is not defined

Do I understand correctly that this can’t be fixed (for the reasons stated above)? In other words, under Windows, static lib means static extensions, too.

Yes. The only binary format I know about that can resolve undefined symbols against the main executable is ELF (and only when compiled with -rdynamic on gcc). Windows and MacOS use COFF based executables. On the many Unix descendants you find several binary formats, among which ELF (e.g., Linux). COFF is more specific in where symbols need to be found, which reduces the likelihood of surprises but this makes it also less flexible.

Except when trying to create a single file executable with no or only system shared object dependencies there is these days no reason not compile the core into a shared object.