Compiling with RTools42, UCRT (simplified)

Nice to know
1

RTools42 is a gcc compiler suite for building R and R packages under Windows. I tried to compile current swipl with the “UCRT” flavor of Rtools, this is how it works:

  • Download RTools42 from https://www.r-project.org/nosvn/winutf8/ucrt3, the name of the file was rtools42-4911-4849.exe today.
  • The pacman repository of Rtools does not include git, so I installed git for Windows (Git - Downloading Package) and made it accessible to the path.
  • in c:\rtools42, start ucrt64
  • A shell appears, type pacman -Syu
  • Then pacman -S mingw-w64-ucrt-x86_64-cmake mingw-w64-ucrt-x86_64-libjpeg mingw-w64-ucrt-x86_64-libyaml mingw-w64-ucrt-x86_64-pcre mingw-w64-ucrt-x86_64-libarchive mingw-w64-ucrt-x86_64-db
  • git clone https://github.com/SWI-Prolog/swipl-devel
  • cd swipl-devel
  • git submodule update --init
  • mkdir build
  • cd build
  • export PATH=$PATH:/x86_64-w64-mingw32.static.posix/bin
cmake -DSWIPL_PACKAGES_X=OFF\
  -DINSTALL_DOCUMENTATION=OFF\
  -DMINGW_ROOT=/ucrt64\
  -DHAVE_ARCHIVE_READ_SUPPORT_FORMAT_CPIO=0\
  -DCMAKE_C_FLAGS=-D__USE_MINGW_ANSI_STDIO\
  -G "MSYS Makefiles"\
  -DCMAKE_INSTALL_PREFIX:PATH=/usr/local ..

Then

  • make
  • make install
  • start swipl (it works without “start”, as well, but you won’t have color)
  • check_installation.
Welcome to SWI-Prolog (threaded, 64 bits, version 8.5.3-25-g01ba6464b-DIRTY)
SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software.
Please run ?- license. for legal details.

For online help and background, visit https://www.swi-prolog.org
For built-in help, use ?- help(Topic). or ?- apropos(Word).

check_installation.
% Checking your SWI-Prolog kit for common issues ...
%
% Version: ............. 8.5.3-25-g01ba6464b-DIRTY
% Address bits: ........ 64
% Architecture: ........ x64-win64
% Installed at: ........ c:/rtools42/usr/local
% Cores: ............... 4
%
% Checking tcmalloc ............................ not present
Warning: See http://www.swi-prolog.org/build/issues/tcmalloc.html
% Checking gmp ................................. ok
% Loading library(archive) ..................... FAILEDc:/rtools42/usr/local/library/archive.pl:96: Initialization goal raised exception:
Warning: Das angegebene Modul wurde nicht gefunden.

Warning: In:
Warning:   [61] throw(error(shared_object(open,'Das angegebene Modul wurde nicht gefunden.\r\n'),context(...,_4896)))
Warning:   [59] <meta call>
Warning:   [58] with_mutex('$foreign',load_foreign_library(foreign(archive4pl),archive,default(install))) <foreign>
Warning:   [55] '$run_init_goal'(shlib:load_foreign_library(...)) at c:/rtools42/usr/local/boot/init.pl:811
Warning:   [54] catch(system:'$run_init_goal'(...),_5052,system:'$initialization_error'(_5074,...,...)) at c:/rtools42/usr/local/boot/init.pl:562
Warning:   [53] catch_with_backtrace(system:'$run_init_goal'(...),_5120,system:'$initialization_error'(_5142,...,...)) at c:/rtools42/usr/local/boot/init.pl:629
Warning:
Warning: Note: some frames are missing due to last-call optimization.
Warning: Re-run your program in debug mode (:- debug.) to get more detail.
Warning: Exported procedure archive:archive_set_header_property/2 is not defined
Warning: Exported procedure archive:archive_open_entry/2 is not defined
Warning: Exported procedure archive:archive_next_header/2 is not defined
Warning: Exported procedure archive:archive_close/1 is not defined
Warning:
Warning: See http://www.swi-prolog.org/build/issues/archive.html
% Loading library(cgi) ......................... ok
% Loading library(crypt) ....................... ok
% Loading library(bdb) ......................... FAILEDc:/rtools42/usr/local/library/bdb.pl:60: Initialization goal raised exception:
Warning: Das angegebene Modul wurde nicht gefunden.

Warning: In:
Warning:   [61] throw(error(shared_object(open,'Das angegebene Modul wurde nicht gefunden.\r\n'),context(...,_20224)))
Warning:   [59] <meta call>
Warning:   [58] with_mutex('$foreign',load_foreign_library(foreign(bdb4pl),bdb,default(install))) <foreign>
Warning:   [55] '$run_init_goal'('<garbage_collected>') at c:/rtools42/usr/local/boot/init.pl:811
Warning:   [54] catch(system:'$run_init_goal'(...),_20370,system:'$initialization_error'(_20392,...,...)) at c:/rtools42/usr/local/boot/init.pl:562
Warning:   [53] catch_with_backtrace(system:'$run_init_goal'(...),_20438,system:'$initialization_error'(_20460,...,...)) at c:/rtools42/usr/local/boot/init.pl:629
Warning:
Warning: Note: some frames are missing due to last-call optimization.
Warning: Re-run your program in debug mode (:- debug.) to get more detail.
Warning: Exported procedure bdb:bdb_del/3 is not defined
Warning: Exported procedure bdb:bdb_open/4 is not defined
Warning: Exported procedure bdb:bdb_put/3 is not defined
Warning: Exported procedure bdb:bdb_close/1 is not defined
Warning: Exported procedure bdb:bdb_transaction/1 is not defined
Warning: Exported procedure bdb:bdb_init/1 is not defined
Warning: Exported procedure bdb:bdb_init/2 is not defined
Warning: Exported procedure bdb:bdb_close_environment/1 is not defined
Warning: Exported procedure bdb:bdb_version/1 is not defined
Warning: Exported procedure bdb:bdb_transaction/2 is not defined
Warning: Exported procedure bdb:bdb_getall/3 is not defined
Warning: Exported procedure bdb:bdb_enum/3 is not defined
Warning: Exported procedure bdb:bdb_get/3 is not defined
Warning:
Warning: See http://www.swi-prolog.org/build/issues/bdb.html
% Loading library(double_metaphone) ............ ok
% Loading library(filesex) ..................... ok
% Loading library(http/http_stream) ............ ok
% Loading library(http/json) ................... ok
% Loading library(http/jquery) ................. ok
%   jQuery from c:/rtools42/usr/local/library/http/web/js/jquery-1.11.3.min.js
% Loading library(isub) ........................ ok
Warning: library(jpl) .......................... NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/jpl.html
% Loading library(memfile) ..................... ok
% Loading library(odbc) ........................ ok
Warning: library(pce) .......................... NOT FOUND
Warning: See http://www.swi-prolog.org/build/issues/xpce.html
% Loading library(pcre) ........................ ok
% Loading library(pdt_console) ................. ok
% Loading library(porter_stem) ................. ok
% Loading library(process) ..................... ok
% Loading library(protobufs) ................... ok
% Loading library(readutil) .................... ok
% Loading library(semweb/rdf_db) ............... ok
% Loading library(semweb/rdf_ntriples) ......... ok
% Loading library(semweb/turtle) ............... ok
% Loading library(sgml) ........................ ok
% Loading library(sha) ......................... ok
% Loading library(snowball) .................... ok
% Loading library(socket) ...................... ok
% Loading library(ssl) ......................... ok
% Loading library(crypto) ...................... ok
% Loading library(table) ....................... ok
% Loading library(time) ........................ ok
% Loading library(unicode) ..................... ok
% Loading library(uri) ......................... ok
% Loading library(uuid) ........................ ok
% Loading library(zlib) ........................ ok
% Loading library(yaml) ........................ ok
Warning: Found 5 issues.
true.

The “archive” and the “bdb” problems disappear if the missing dlls (things like libbz2-1.dll that is used by bdb4pl.dll) are copied into the installation folder, /usr/local/bin. I don’t understand why they are not loaded from their default place, /ucrt/bin. If someone can explain to me how to fix this, it’s much appreciated.

cd /usr/local/bin
cp `ldd archive4pl.dll |grep -o "/ucrt64/bin/.*\\.dll"` .
cp `ldd bdb4pl.dll |grep -o "/ucrt64/bin/.*\\.dll"` .

The compiler flag -D__USE_MINGW_ANSI_STDIO is needed to enable things like printf("%lld", (int64_t) x). There’s probably a way to autodetect this.

1 Like
2

Ah, there’s a little error on the way (compilation continues, though):

[ 93%] QLF compiling doc_html.qlf
ERROR: c:/rtools42/tmp/rtmpcdusns/r.install1d206bf71f68/rolog/mingw64/home/library/pldoc/doc_html.pl:98:
ERROR:    source_sink `library(http/jquery)' does not exist
Warning: c:/rtools42/tmp/rtmpcdusns/r.install1d206bf71f68/rolog/mingw64/home/library/pldoc/doc_html.pl:98:
Warning:    Goal (directive) failed: pldoc_html:use_module(library(http/jquery))
make[3]: Verzeichnis „/tmp/RtmpcDUsNs/R.INSTALL1d206bf71f68/rolog/mingw64“ wird verlassen
[ 94%] Built target qlf-doc_html

I commented out the :- use_module(library(http/jquery)) directive from packages/pldoc/doc_html.pl, it is now compliling without the error. The directive does not seem to be necessary.

3

Unfortunately, generating the documentation has more dependencies than it needs to have. This patch probably clauses other problems though. As is, either build the entire suite with documentation or the core with a limited package selection without the docs. I think only JPL, ODBC and xpce can be safely skipped while building the docs. These are commonly stripped to reduce dependencies on a lot of infrastructure. The rest is quite small.

4

Yes indeed, I’ve seen it in the sources.

5

I looked into this a bit more.

After “make”, jquery.pl is actually there, just in (build)/packages/http, not in (build)/home/library/http. Same under Linux, so this seems to be intended. Anyway, during the build, library(http/jquery) does not seem to be expanded to the folder with the packages.

After the build, but before “make install”: When I invoke swipl with (build)/src/swipl.exe, library is expanded to

file_search_path(library, P).
P = app_config(lib) ;
P = swi(library) ;
P = swi(library/clp) ;

Appendix: Under Linux, there’s more:

?- file_search_path(library, P).
P = app_config(lib) ;
P = swi(library) ;
P = swi(library/clp) ;
P = swi(packages) ;
P = ‘/home/matthias/swipl-devel/build/packages’ ; <<<<<<<<<<< this!
P = ‘/home/matthias/swipl-devel/build/packages/chr’ ;
P = pce(‘prolog/lib’).

swi(packages) is not the correct one, since swi points to

?- file_search_path(swi, P).
P = ‘/home/matthias/swipl-devel/build/home’ ;
false.

6

That is correct :slight_smile: The system installs (build)/home/swipl.rc as system init file that will make it find the normal resources that it normally finds from the install directory from the build directory. That allows running (build)/src/swipl[.exe] as a full featured Prolog system without installing it. I always do so :slight_smile: It is exactly the packages search path that causes it to properly resolve library(http/jquery).

Could it be you have the environment variable SWI_HOME_DIR set? That might upset the build. cmake prints a warning:

SWI_HOME_DIR=/ cmake .
-- Configuring SWI-Prolog-8.5.5
CMake Warning at cmake/CheckEnv.cmake:2 (message):
  -- Environment variable SWI_HOME_DIR is set to /

  This suggests that the variable is referring to a existing installation of
  SWI-Prolog.  This may conflict with building SWI-Prolog as this may cause
  the build process to use components from such (older) installed version of
  SWI-Prolog.  Make sure to remove this environment from the environment
  before building the system.
7

No, SWI_HOME_DIR is not set.

I am currently trying to understand what happens in build_home.pl

8

Surely, I get the same library path on Windows than on Linux when I run (build)/src/swipl.exe.

9

You probably know, but to make sure: I am trying to run the RTools42, UCRT build, not the “normal” swipl for Windows.

10

Getting closer.

For reasons not yet fully understood, there is no swipl.rc in my (build)/home/. But there’s one in the swipl-devel directory. If I copy it manually into (build)/home/, I get one more “solution”:

file_search_path(library, P).
P = app_config(lib) ;
P = swi(library) ;
P = swi(library/clp) ;
P = swi(packages).

Which is, unfortunately, not the one I am hunting for. However, the folders that I obtained in Linux, that is, P = ‘/home/matthias/swipl-devel/build/packages’ result from this piece of code in swipl.rc:

add_package(chr, PkgBinDir) :-
    assertz(user:file_search_path(chr, PkgBinDir)),
    assertz(user:file_search_path(chr, library(chr))),
    assertz(user:file_search_path(library, PkgBinDir)). % PkgBinDir, i.e., packages/chr
add_package(http, PkgBinDir) :-
    add_package_path(PkgBinDir),
    file_directory_name(PkgBinDir, PkgDir),
    assertz(user:file_search_path(library, PkgDir)). % Not PkgBinDir, but PkgDir, i.e. packages instead of packages/http

For now, I will try to solve the issue by copying swipl.rc manually into the build folder, and just appending a line user:file_search_path(library, PkgDir), with PkgDir hopefully accepting relative paths.

11

This file is made available by src/CMakeLists.txt:315:

symlink(${SWIPL_BOOT_ROOT}/build_home.pl ${CMAKE_BINARY_DIR}/home/swipl.rc)
symlink(${SWIPL_BOOT_ROOT}/build_home.pl ${CMAKE_BINARY_DIR}/home/swipl-win.rc)

symlink() is defined in cmake/InstallSource.cmake and calls cmake -E create_symlink. Maybe tat causes the problem. I’m cross-compiling using Linux, so symlinks work just fine. For Windows there are some comments at cmake - replacement of create_symlink in windows - Stack Overflow

12

That explains everything…

– Performing Test TIME_WITH_SYS_TIME - Success
CMake Error: failed to create symbolic link ‘./swipl.rc’: Dem Client fehlt ein erforderliches Recht.
CMake Error: failed to create symbolic link ‘./swipl-win.rc’: Dem Client fehlt ein erforderliches Recht.
– Looking for sys/socket.h

I’ve seen this error message so often that I did not read it anymore. I’ll prepare a PR, changing symlink to symlink_or_copy in CMakeLists.txt, and adding the respective function to InstallSource.cmake that checks if CMAKE_HOST_UNIX is true, then create_symlink, else copy. Hope this is ok.

1 Like
13

It’s working, yay!

Three more things :slight_smile:

  1. Warnings in pl-write.c on sprintf macros “S_%” PRId64, which seem to differ between Linux and Windows (oh no!). I can get rid of the warning by adding -DCMAKE_C_FLAGS=-D__USE_MINGW_ANSI_STDIO, which forces to use mingw’s own sprintf functions, so there is a fix for this. No further action needed.

  2. Warnings around timezone and tzname. tzname is only in src/os/pl-tai.c

C:/rtools42/tmp/RtmpEtpraf/R.INSTALL290c1fa1bbf/rolog/src/swipl-devel/src/os/pl-tai.c:154:3: warning: 'tzname' is deprecated: Only provided for source compatibility; this variable might not always be accurate when linking to UCRT. [-Wdeprecated-declarations]
  154 |   return tzname[dst];
      |   ^~~~~~

I played around a bit and I think it can be overcome in the same way as in timezone:

pl-tai.c, line 60:

#if defined(__WINDOWS__) || defined (__CYGWIN__)
#define timezone _timezone
(etc.)

Line 154 (that is the place of the warning), the line looks like this:

#ifdef _MSC_VER
  return _tzname[dst];
#else
  return tzname[dst];
#endif

I wonder if the condition can be changed from #ifdef _MSC_VER to the same as above, #if defined(__WINDOWS__) || defined (__CYGWIN__). Unfortunately, I lack the insight to understand these preprocessor stuff in enough detail.

I tried it, it made the warning disappear, so I’ll prepare a PR if you don’t mind.

  1. timezone: Later.
14

I guess this could go into `cmake/port/MinGW.cmake?

Just merged your patch. At least compiles clean for me on MacOS, Linux and MinGW. Shouldn’t change anything for MSVC. I don’t think compilation on MSVC is clean anyway (if it works at all).

15

I guess this could go into `cmake/port/MinGW.cmake?

It’s complicated, see the discussion here:

https://sourceforge.net/p/mingw-w64/mailman/mingw-w64-public/thread/65f8a89.69ec.171ca7d1e27.Coremail.lh_mouse%40126.com/

Given that the swipl code uses the correct macros, e.g., sprintf(buf, “S_%” PRId64, -n), I think the problem is elsewhere.

16

Thanks. I’m a little lost in standard or version space :frowning: Anyway, the normal build procedure using Docker under Linux works and if yours works the way you want it, I’m happy.