Compiling with Rtools40, UCRT

mgondan1 · September 1, 2021, 8:22am

RTools40 is a minimalistic gcc compiler suite for building R and R packages under Windows. I tried to compile current swipl with the “UCRT” flavor of Rtools, whatever it is, but in contrast to “mingw32” and “mingw64”, it has a package libarchive.

One of the complications is that the libraries bundled with Rtools are all static, no dlls. That’s an R thing. R packages are themselves dlls under windows, but these libraries should preferably link to static dependencies. There is probably a reason for this.

Download and install Rtools, see here RTools: Toolchains for building R and R packages from source on Windows
The pacman repository of Rtools does not include git, so I installed git for Windows (Git - Downloading Package) and made it accessible to the path.
in Rtools (UCRT): pacman -Syu
maybe do it twice: pacman -Syu
pacman -S mingw-w64-ucrt-x86_64-cmake mingw-w64-ucrt-x86_64-libjpeg mingw-w64-ucrt-x86_64-libyaml mingw-w64-ucrt-x86_64-pcre mingw-w64-ucrt-x86_64-libarchive
git clone https://github.com/SWI-Prolog/swipl-devel
cd swipl-devel
git submodule update --init
mkdir build
cd build

cmake -DMINGW_ROOT=/ucrt64 -DINSTALL_DOCUMENTATION=OFF \
    -DSWIPL_PACKAGES_X=OFF -DWIN32_DLLS="" \
    -DLibArchive_INCLUDE_DIRS="`pkg-config --cflags-only-I libarchive`" \
    -DLibArchive_LIBRARIES="`pkg-config --libs --static libarchive`" \
    -DCMAKE_C_FLAGS=-DPCRE_STATIC \
    -DCMAKE_INSTALL_PREFIX:PATH=/usr/local -G "MSYS Makefiles" ..

Now wait 10 min
If you run the command a second time, it returns within a few instances, and you can inspect the missing dependencies: libtcmalloc, libuuid, Berkeley DB, JNI, Java, qt5widgets. libuuid can, in principle, be fixed, but let’s move on.
make

(Removed the parts that have been fixed, see conversation below).

Last problem (we’re still at 98%), in crypto4pl.c:

C:/Users/matth/swipl-devel/packages/ssl/cryptolib.c:53:1: error: static declaration of 'EVP_MD_CTX_free' follows non-static declaration
   53 | EVP_MD_CTX_free(EVP_MD_CTX *ctx)

I am unsure if this is due to clutter on my hard disk (see the recent topic), but I fixed it like this:

(still in swipl-devel/build): cd packages/ssl
notepad config.h and add the following 3 lines in the beginning:

#define HAVE_EVP_MD_CTX_FREE 1
#define HAVE_X509_DIGEST 1
#define HAVE_X509_CRL_DIGEST 1

(back in swipl-devel/build): make
make install
Now, fasten your seatbelts.
start swipl (it works without “start”, as well, but you won’t have color)

Welcome to SWI-Prolog etc.

?- member(X, [1, 2, 3]).

X = 1 ;
X = 2 ;
X = 3.

?- check_installation.

% Checking your SWI-Prolog kit for common issues ...
%
% Version: ............. 8.3.28-42-gcfeb5156d-DIRTY
% Address bits: ........ 64
% Architecture: ........ x64-win64
% Installed at: ........ c:/rtools40/usr/local
% Cores: ............... 4
%
% Checking tcmalloc ............................ not present
Warning: See http://www.swi-prolog.org/build/issues/tcmalloc.html
% Checking gmp ................................. ok
% Loading library(archive) ..................... ok
%   Supported filters:
%   Supported formats: 7zip, ar, cab, cpio, empty, gnutar, iso9660, lha, mtree, rar, raw, tar, xar, zip
% Loading library(cgi) ......................... ok
% Loading library(crypt) ....................... ok
Warning: library(bdb) .......................... NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/bdb.html
% Loading library(double_metaphone) ............ ok
% Loading library(filesex) ..................... ok
% Loading library(http/http_stream) ............ ok
% Loading library(http/json) ................... ok
% Loading library(http/jquery) ................. ok
%   jQuery from c:/rtools40/usr/local/library/http/web/js/jquery-1.11.3.min.js
% Loading library(isub) ........................ ok
Warning: library(jpl) .......................... NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/jpl.html
% Loading library(memfile) ..................... ok
% Loading library(odbc) ........................ ok
Warning: library(pce) .......................... NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/xpce.html
% Loading library(pcre) ........................ ok
% Loading library(pdt_console) ................. ok
% Loading library(porter_stem) ................. ok
% Loading library(process) ..................... ok
% Loading library(protobufs) ................... ok
% Loading library(readutil) .................... ok
% Loading library(semweb/rdf_db) ............... ok
% Loading library(semweb/rdf_ntriples) ......... ok
% Loading library(semweb/turtle) ............... ok
% Loading library(sgml) ........................ ok
% Loading library(sha) ......................... ok
% Loading library(snowball) .................... ok
% Loading library(socket) ...................... ok
% Loading library(ssl) ......................... ok
% Loading library(crypto) ...................... ok
% Loading library(table) ....................... ok
% Loading library(time) ........................ ok
% Loading library(unicode) ..................... ok
% Loading library(uri) ......................... ok
% Loading library(uuid) ........................ ok
% Loading library(zlib) ........................ ok
% Loading library(yaml) ........................ ok
Warning: Found 4 issues.
true.

Nothing unexpected. Except that libarchive claims to support RAW and MTREE, which it doesn’t, so my pull request for packages/archive4pl.c may be wrong or not sufficient.

Let’s see if we can compile things: ?- pack_install(environ).

% Contacting server at https://www.swi-prolog.org/pack/query ... ok
Install environ@1.0 from http://www.swi-prolog.org/download/pack/environ-1.0.tgz Y/n?
% Contacting server at https://www.swi-prolog.org/pack/query ... ok
% "environ-1.0.tgz" was downloaded 52 times
Package:                environ
Title:                  Demo package with C code, fetching the program environment
Installed version:      1.0
Author:                 Jan Wielemaker <jan@swi-prolog.org>
Home page:              http://www.swi-prolog.org/howto/ForeignPack.html
Download URL:           http://www.swi-prolog.org/download/pack/environ-*.tgz
Install "environ-1.0.tgz" (1,084 bytes) Y/n?
Warning: In file included from c:/rtools40/usr/local/include/SWI-Prolog.h:50,
Warning:                  from c/environ.c:1:
Warning: c/environ.c: In function 'pl_environ':
Warning: c/environ.c:9:17: warning: '__p__environ' redeclared without dllimport attribute: previous dllimport ignored [-Wattributes]
Warning:     9 | { extern char **environ;
Warning:       |                 ^~~~~~~
% gcc  -I"c:/rtools40/usr/local/include"   -c -o c/environ.o c/environ.c
% mkdir -p lib/x64-win64
% gcc  -shared -L"c:/rtools40/usr/local/bin" -o lib/x64-win64/environ.dll c/environ.o -lswipl
% make: Nothing to be done for 'check'.
% make: Nothing to be done for 'install'.
true.

Good enough for now.

?- use_module(library(environ)).
?- environ(X).

(some stuff is shown)

Very nice.

jan · September 1, 2021, 11:23am

Thanks. Added a fix for that.

It probably works by setting LibArchive_LIBRARIES. This is the disadvantage of not using DLLs: you need to know the dependencies of the library you link to. Normally CMake’s find_package(LibArchive) should deal with that. There are many ways to make this fail though, notably on Windows . libarchive has a lot of dependencies and the dependencies depend on how the library was configured and built

I guess we could check PCRE_LIBRARIES to see whether the library is static or dynamic (that is in the end heuristic as there is no platform standard for library filename extensions) and if static add this to the target CFLAGS. This is a bit CMake hacking. I don’t have a good platform to test. Happy to merge a patch

mgondan1 · September 1, 2021, 4:56pm

Regarding the second point: You wrote below that it probably works by setting LibArchive_LIBRARIES. Unfortunately, it is overwritten by find_package(LibArchive), I checked it with message("${LibArchive_LIBRARIES}") before and after find_package in swipl-devel/packages/archive/CMakeLists.txt. Would you accept something like this?

Old:

find_package(LibArchive)

New:

if(DEFINED LibArchive_INCLUDE_DIRS OR DEFINED LibArchive_LIBRARIES)

set(LibArchive_FOUND:BOOL=ON)

else()

find_package(LibArchive)

endif()

The LibArchive_FOUND is used in the later course.

Citing O‘Keefe: This is dirty, but not too dirty.

jan · September 2, 2021, 7:13am

Pushed something along these ways, well, like below. I think be specified and as non-empty strings.

if(LibArchive_LIBRARIES AND LibArchive_INCLUDE_DIRS)
  set(LibArchive_FOUND ON)
else()
  find_package(LibArchive)
endif()

This is to overcome that the CMake FindLibArchive.cmake is rather simple. It should use pkg-config to figure out the library dependencies (and pkg-config should be installed properly for libarchive).

mgondan1 · September 2, 2021, 12:37pm

Cool, thanks. Works. I will update the description above to specify both LibArchive_LIBRARIES and LibArchive_INCLUDE_DIRS.

For what it’s worth, pkg-config works fine:

$ pkg-config --libs --static libarchive
-LC:/rtools40/ucrt64/lib -larchive -lnettle -lexpat -llzma -llz4 -lbz2 -lz -lregex -ltre -pipe -lintl /ucrt64/lib/libiconv.a -lbcrypt

jan · September 2, 2021, 1:08pm

Great. Unfortunately FindLibArchive.cmake, which is part of the CMake package doesn’t use pkg-config I had a brief look to first use pkg-config in our CMakeLists.txt. It surely can be done, but it is not that easy

mgondan1 · September 2, 2021, 1:33pm

The plural is correct, or isn’t it? LibArchive_INCLUDE_DIRS (my guess would have been LibArchive_INCLUDE_DIR). It’s working fine, I am just asking (and please disregard my question if you’re busy).

jan · September 2, 2021, 2:02pm

The plural is correct. There may be multiple libraries in multiple directories involved.

mgondan1 · September 2, 2021, 3:26pm

I tried to apply a similar patch to libPCRE, but without any success. I don’t know what I am doing wrong. I tried PCRE_INCLUDE_DIRS, PCRE_C_FLAGS, any combination.

I gave up and just added -DCMAKE_C_FLAGS=-DPCRE_STATIC to the cmake command. This is obviously a hack; on the other hand, this particular global flag does not hurt anything else in the SWIPL tree, so for now, I am fine.

Edit: Ok, now I found a FindPCRE.cmake which does not exist for libArchive, so the workaround for libArchive cannot be generalized that easily.

Topic		Replies	Views
Compiling with RTools42, UCRT (simplified) Nice to know	15	2266	February 4, 2022
Compiling with RTools40 General	2	878	February 2, 2022
Rtools42, Unit test for pcre General	20	704	April 8, 2022
Pip install swipl? Looking for Group	6	555	August 19, 2023
Building swipl on Win64, MSYS2 (cmake, make, make install) General how-to	5	513	April 20, 2020

Compiling with Rtools40, UCRT

Related topics