Compiling with MSYS2


If one is looking for an SWI-Prolog pack that is built using CMake it is rather hard.
Here are two. Note that this is in the Jan W. GitHub account and not in the SWI-Prolog GitHub account.

Currently trying to install on Windows system. This will be a learning experience.

EDIT

Here is another one not by Jan W.

libssh needs libutil, see the error:

% CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
% Please set them or make sure they are set and tested correctly in the CMake files:
% LIBUTIL
%     linked by target "sshd4pl" in directory C:/Users/matth/AppData/Local/swi-prolog/pack/libssh

And libutil does not seem to be available on MSYS2 (at least not as a MinGW variant)

$ pacman -Sl |grep libutil
msys libutil-linux 2.35.2-1 [installed]
msys libutil-linux-devel 2.35.2-1 [installed]

I would not try this further (given that you’re interested in rocksdb, and given that this is actually available on msys2)

I haven’t tried it yet, still searching the rest of the packs to see if CMake is used. However I was considering to totally bypass MSYS2 and such and setup the CMake to do a native Windows build then leverage that to work back towards a cross build. Also if the source code is not available, as you are finding then would search for it going back to a GitHub clone as needed. There are other tricks I have learned but haven’t tried them yet that would be of value for this. :slightly_smiling_face:

Ok. The problem is also seen on a linux git clone. I can fix it by adding this line to cmake/LibIndex.cmake

  add_dependencies(library_index ${target})
  endif()
endforeach()

add_dependencies(library_index_library_always plugin_yaml4pl plugin_uri plugin_unicode4pl plugin_uid plugin_table plugin_syslog plugin_streaminfo plugin_ssl4pl plugin_snowball plugin_sha4pl plugin_sched plugin_rlimit plugin_redis4pl plugin_protobufs plugin_prolog_stream plugin_process plugin_porter_stem plugin_pdt_console plugin_pcre4pl plugin_odbc4pl plugin_md54pl plugin_memfile plugin_mallocinfo plugin_isub plugin_hashstream plugin_double_metaphone plugin_crypto4pl plugin_bdb4pl plugin_archive4pl)

And something similar to the CMakeFiles.txt of packages/http and SSL.

I hope this is a bit helpful.

I think so. Pushed 19650dfd33c9524471f004d56cda28d3e0f32c70 to try and address this issue. Listing the dependencies as you do will not work if the package collection is selected differently. I now add the dependencies while configuring the packages. At least it still builds. Whether it really fixes the issue is unclear.

I still wonder whether there are tools that actually analyze the dependencies of a cmake/ninja project. After all, it should be possible by monitoring the files accessed by the various build steps.

For the build with install_documentation on, one more dependency was needed in semweb/cmakelists.txt

add_dependencies(semweb turtle ntriples rdf_db)

I’ll send a PR [Edit: done]

1 Like

I guess that string_bytes/3 may still be under construction, so please disregard if this is too early to send errors. I just ran the unit tests from within msys2 and the conversion to utf16 failed (should just return aap):

string_bytes(S, [0, 97, 0, 97, 0, 112], utf16be).
S = "aap\u0000瓩퍧".

?- string_bytes(S, [97, 0, 97, 0, 112, 0], utf16le).
S = "aap戀瑹獥".

The problem is also seen in the “daily build for Windows, 64 bit” from the swi-prolog homepage.

I tried to track it a bit down, in pl-string.c, line 247 etc.

if ( PL_get_nchars(A2, &len, &s, CVT_LIST|REP_ISO_LATIN_1|CVT_EXCEPTION) )
  { if ( len % unit_length(enc) != 0 )
      return PL_syntax_error("incomplete_multibyte_sequence", NULL);

    t.text.t    = s;
    t.length    = len/unit_length(enc);
    t.encoding  = enc;
    t.canonical = FALSE;
    t.storage   = PL_CHARS_HEAP;
    // printf("len=%ld, unitlength=%d\n", len, unit_length(enc)) ; % Linux
    printf("len=%lld, unitlength=%d\n", len, unit_length(enc)) ; % Windows

    return PL_unify_text(A1, 0, &t, PL_STRING);
  }

Len is 6, unitlength is 1 under both Linux and Windows.

1 Like

The dependency is 120 MB, and it is needed if you want to install swipl-win.exe, which is not the normal Windows IDE for Swi-Prolog, but basically a white window with black font :sunglasses:

It is new and touches on fairly new code for supporting UTF-16 in Windows. That was broken. Fixed with 4d420349def89840063131bfb03f71f954109ea9.

Edit: The problem seems to have disappeared.

one more dependency issue:

[978/1197] Build home/library/INDEX.pl
ERROR: c:/msys64/home/matthias/swipl-devel/build/home/library/crypto.pl:72:
ERROR:    c:/msys64/home/matthias/swipl-devel/build/home/library/crypto.pl:72: Initialization goal raised exception:
ERROR:    '$open_shared_object'/3: Das angegebene Modul wurde nicht gefunden.

I guess some dependency is missing in packages/ssl. I’ll check this.

Question: What is this O_XOS thing? Bear with me, I am a beginner (and will always be). I am asking because this morning swipl-devel does not compile anymore under MSYS2, complaining that a certain file C:/msys64/home/Matthias/swipl-devel/build/man/pldoc2tex doesn’t exist (though it does).

I can partially fix the problem using these steps:

  • add the file extension .exe to pldoc2tex in cmake/Documentation.cmake—this does not seem to be done automatically
  • for the function OsPath in os/pl-os.c, copy the code from the O_HPFS section to the O_XOS section (EDIT: such that the directory separator / is properly translated to a backslash again)
  • in os/pl-file.c, change statfunc to stat (EDIT: to override the O_XOS-specific definition)

I hope this makes any sense.

It is implemented in src/os/windows/uxnt.[ch] and maps the file system interaction functions to _xos_<name>(). These functions are defined in uxnt.c and perform the path translation for UTF-8 char* to UTF-16 so we can use the Windows Unicode functions to exchange non-ansi file names and get rid of most of the 260 PATH_MAX limitations. It also provides some additional emulation such as opendir().

I’m working hard on this stuff now to improve long file name and Unicode file name support for a customer. So, it is a bit of a moving target :frowning: Worse, it turns out that the Windows file system API is badly broken in many places. Right now it runs much more reliably on Wine (the Linux emulator for Windows) :slight_smile:

Could you share these patches? It looks like they patch the wrong thing, but it might give me a clue what is wrong. I’ll also verify the complete build in Linux. That now fails on some JPL DLL issue :frowning:

Hope to work around the Windows bugs in the next couple of days … There may be some regression and there surely will some regression until then.

3 Likes

This is the build log:

[956/1197] Build home/library/INDEX.pl
ERROR: c:/msys64/home/matthias/swipl-devel/build/home/library/crypto.pl:72:
ERROR:    c:/msys64/home/matthias/swipl-devel/build/home/library/crypto.pl:72: Initialization goal raised exception:
ERROR:    '$open_shared_object'/3: Das angegebene Modul wurde nicht gefunden.

ERROR: c:/msys64/home/matthias/swipl-devel/build/home/library/ssl.pl:69:
ERROR:    c:/msys64/home/matthias/swipl-devel/build/home/library/ssl.pl:69: Initialization goal raised exception:
ERROR:    '$open_shared_object'/3: Das angegebene Modul wurde nicht gefunden.

[957/1197] Generating rdf2pl.html
[958/1197] -- CHR bootstrap compilation step 2
[959/1197] Generating table.tex
[960/1197] QLF compiling doc_html.qlf
[961/1197] Generating odbc.bib
[962/1197] Generating zlib.tex
[963/1197] Generating pldoc2tex
FAILED: man/pldoc2tex C:/msys64/home/Matthias/swipl-devel/build/man/pldoc2tex
cmd.exe /C "cd /D C:\msys64\home\Matthias\swipl-devel\build\man && C:\msys64\home\Matthias\swipl-devel\build\src\swipl.exe -f none --no-packs -o C:/msys64/home/Matthias/swipl-devel/build/man/pldoc2tex --class=development --no-autoload --no-stand_alone -c C:/msys64/home/Matthias/swipl-devel/man/pldoc2tex.pl -- --nopce"
ERROR: '$mark_executable'/1: file `'C:/msys64/home/Matthias/swipl-devel/build/man/pldoc2tex'' does not exist (No such file or directory)

The problem in 957 is a dependency issue, not relevant here. I consider the error in 964. The suspicious thing is the forward slashes. The program fails in pl-files.c, MarkExecutable, at “statfunc”, line 449. Changing this to “stat” appears to fix the problem. (no need to fiddle with OsPath)

There’s a separate issue which I report in the next response.

running ninja again, after having fixed the “stat” thing.

[143/155] Generating tests/test_certs/generated
FAILED: packages/ssl/tests/test_certs/generated C:/msys64/home/Matthias/swipl-devel/build/packages/ssl/tests/test_certs/generated
cmd.exe /C "cd /D C:\msys64\home\Matthias\swipl-devel\build\packages\ssl && C:\msys64\mingw64\bin\cmake.exe -E make_directory tests && C:\msys64\home\Matthias\swipl-devel\build\src\swipl.exe -f none --no-packs C:/msys64/home/Matthias/swipl-devel/build/packages/ssl/mkcerts.pl --source=C:/msys64/home/Matthias/swipl-devel/packages/ssl/tests --dest=tests && touch tests/test_certs/generated"
ERROR: c:/msys64/home/matthias/swipl-devel/build/packages/ssl/mkcerts.pl:52: mkcerts:main openssl failed with status exit(1)
ERROR:
ERROR: Can't open \\rootCA.cnf for reading, No such file or directory
ERROR: 5688:error:020010A1:system library:fopen:reason(161):../openssl-1.1.1q/crypto/bio/bss_file.c:69:fopen('\\rootCA.cnf','r')
ERROR: 5688:error:2006D080:BIO routines:BIO_new_file:no such file:../openssl-1.1.1q/crypto/bio/bss_file.c:76:
ERROR:
[144/155] Generating cryptolib.tex

maybe a consequence of the former (?)

Pushed a fix for that. Turned the entire predicate into a dummy for Windows as it does not do anything anyway. stat() is a particularly nasty function in Windows as there are a lot of variations. The xos stuff needs to intercept to get the file names correct, but it appears Windows stat doesn’t like the "\?" paths (some random selection of the API doesn’t accept these :frowning: ). I have replaced most usage of stat() by native Win32 APIs to get the desired information. Should probably do that for all :frowning:

I wonder where that comes from. Looks more like an OpenSSL issue? The build now works fine on Linux. Already did so as Wine’s stat() implementation seems to work fine …

Indeed, openssl is called with “canonical filenames”, \?\C:\path\etc.

c7201178@PC105-C720 MINGW64 ~/swipl-devel/build
$ ninja
[11/13] Generating tests/test_certs/generated
FAILED: packages/ssl/tests/test_certs/generated C:/msys64/home/c7201178/swipl-devel/build/packages/ssl/tests/test_certs/generated
cmd.exe /C "cd /D C:\msys64\home\c7201178\swipl-devel\build\packages\ssl && C:\msys64\mingw64\bin\cmake.exe -E make_directory tests && C:\msys64\home\c7201178\swipl-devel\build\src\swipl.exe -f none --no-packs C:/msys64/home/c7201178/swipl-devel/build/packages/ssl/mkcerts.pl --source=C:/msys64/home/c7201178/swipl-devel/packages/ssl/tests --dest=tests && touch tests/test_certs/generated"

Here are a few "writelns":

process_create-C:/msys64/mingw64/bin/openssl.exe-[req,-new,-config,file(C:/msys64/home/c7201178/swipl-devel/packages/ssl/tests/rootCA.cnf),-keyout,file(test_certs/rootCA/private/cakey.pem),-out,file(test_certs/rootCA/careq.pem)]-[stderr(pipe(_8744)),stdout(null),process(_8764)]
process_create-c:/msys64/mingw64/bin/openssl.exe-[extensions([,exe,com]),access(read)]
map_arg_prim_plfile-C:/msys64/home/c7201178/swipl-devel/packages/ssl/tests/rootCA.cnf
map_arg_prim_file- \\?\C:\msys64\home\c7201178\swipl-devel\packages\ssl\tests\rootCA.cnf
map_arg_prim_plfile-test_certs/rootCA/private/cakey.pem
map_arg_prim_file- \\?\C:\msys64\home\c7201178\swipl-devel\build\packages\ssl\tests\test_certs\rootCA\private\cakey.pem
map_arg_prim_plfile-test_certs/rootCA/careq.pem
map_arg_prim_file- \\?\C:\msys64\home\c7201178\swipl-devel\build\packages\ssl\tests\test_certs\rootCA\careq.pem
process_create_av-[req,-new,-config,\\?\C:\msys64\home\c7201178\swipl-devel\packages\ssl\tests\rootCA.cnf,-keyout,\\?\C:\msys64\home\c7201178\swipl-devel\build\packages\ssl\tests\test_certs\rootCA\private\cakey.pem,-out,\\?\C:\msys64\home\c7201178\swipl-devel\build\packages\ssl\tests\test_certs\rootCA\careq.pem]

Assuming that MSYS2’s openssl should be able to deal with “canonical filenames”, the problem is openssl, see here:

c7201178@PC105-C720 MINGW64 ~/swipl-devel/build/packages/ssl
$ cmd.exe /C "openssl req -new -config \\?\C:\msys64\home\c7201178\swipl-devel\packages\ssl\tests\rootCA.cnf -keyout \\?\C:\msys64\home\c7201178\swipl-devel\build\packages\ssl\tests\test_certs\rootCA\private\cakey.pem -out \\?\C:\msys64\home\c7201178\swipl-devel\build\packages\ssl\tests\test_certs\rootCA\careq.pem"
Can't open \?\C:\msys64\home\c7201178\swipl-devel\packages\ssl\tests\rootCA.cnf for reading, Invalid argument
9008:error:0200107B:system library:fopen:Unknown error:../openssl-1.1.1q/crypto/bio/bss_file.c:69:fopen('\?\C:\msys64\home\c7201178\swipl-devel\packages\ssl\tests\rootCA.cnf','r')
9008:error:2006D002:BIO routines:BIO_new_file:system lib:../openssl-1.1.1q/crypto/bio/bss_file.c:78:

Whereas the non-canonical filename works:

c7201178@PC105-C720 MINGW64 ~/swipl-devel/build/packages/ssl
$ cmd.exe /C "openssl req -new -config C:\msys64\home\c7201178\swipl-devel\packages\ssl\tests\rootCA.cnf -keyout C:\msys64\home\c7201178\swipl-devel\build\packages\ssl\tests\test_certs\rootCA\private\cakey.pem -out C:\msys64\home\c7201178\swipl-devel\build\packages\ssl\tests\test_certs\rootCA\careq.pem"
Generating a RSA private key
......................+++++
(etc.)
1 Like

OPENSSL_WIN32_UTF8 needs to be set, see below

Matthias@DESKTOP-A2T8IFC MINGW64 ~/swipl-devel/build/packages/ssl
$ cmd.exe
Microsoft Windows [Version 10.0.19044.1889]
(c) Microsoft Corporation. Alle Rechte vorbehalten.

C:\msys64\home\Matthias\swipl-devel\build\packages\ssl>openssl req -new -config \\?\C:\msys64\home\matthias\swipl-devel\packages\ssl\tests\rootCA.cnf
openssl req -new -config \\?\C:\msys64\home\matthias\swipl-devel\packages\ssl\tests\rootCA.cnf
Can't open \\rootCA.cnf for reading, No such file or directory
8664:error:020010A1:system library:fopen:reason(161):../openssl-1.1.1q/crypto/bio/bss_file.c:69:fopen('\\rootCA.cnf','r')
8664:error:2006D080:BIO routines:BIO_new_file:no such file:../openssl-1.1.1q/crypto/bio/bss_file.c:76:

C:\msys64\home\Matthias\swipl-devel\build\packages\ssl>SET OPENSSL_WIN32_UTF8=1
SET OPENSSL_WIN32_UTF8=1

C:\msys64\home\Matthias\swipl-devel\build\packages\ssl>openssl req -new -config \\?\C:\msys64\home\matthias\swipl-devel\packages\ssl\tests\rootCA.cnf
openssl req -new -config \\?\C:\msys64\home\matthias\swipl-devel\packages\ssl\tests\rootCA.cnf
Generating a RSA private key
...............................................+++++
...........+++++

For those like me wondering about OPENSSL_WIN32_UTF8

It is a OpenSSL - environment variable

OPENSSL_WIN32_UTF8

If set, then UI_OpenSSL(3) returns UTF-8 encoded strings, rather than ones encoded in the current code page, and the openssl(1) program also transcodes the command-line parameters from the current code page to UTF-8. This environment variable is only checked on Microsoft Windows platforms. (ref)

GitHub

I have a hypothesis regarding the dependency issue: Two things,

add_swipl_target in LinIndex.cmake library_index needs to “DEPEND” on the module, e.g. library_index_library_ssl should depend on ssl.

However, it should not depend on the ninja “target”, but on the files that have been produced. It does not seem to wait until the linker is finished:

[10/13] Linking C shared module src\crypto4pl.dll
[11/13] Linking C shared module src\ssl4pl.dll
[12/13] Build home/library/INDEX.pl
ERROR: c:/msys64/home/matthias/swipl-devel/build/home/library/crypto.pl:72:
ERROR:    c:/msys64/home/matthias/swipl-devel/build/home/library/crypto.pl:72: Initialization goal raised exception:
ERROR:    '$open_shared_object'/3: Das angegebene Modul wurde nicht gefunden.

ERROR: c:/msys64/home/matthias/swipl-devel/build/home/library/ssl.pl:69:
ERROR:    c:/msys64/home/matthias/swipl-devel/build/home/library/ssl.pl:69: Initialization goal raised exception:
ERROR:    '$open_shared_object'/3: %1 ist keine zul▒ssige Win32-Anwendung.

[13/13] Build home/library_ssl/INDEX.pl

The first error says that crypto4pl.dll is not yet there, the second says that ssl4pl.dll is not readable.

(Edit: removed the link because the site wanted to install things on my phone)

For now I have added some code that causes make_library_index/1 ignore errors from loading
shared objects. This is not ideal. It seems really hard to get this right tough. The solution proposed in the article does not work as you cannot make cmake targets depend directly on files. Possibly some redesign of the dependency structure for building these objects could fix the issue, but it is beyond my cmake knowledge :frowning:

The article does hint at issues of infrequently missing objects during builds. AFAIK these issues only happens on Windows (could of course be luck). That is why the command I use for unattended builds on Windows call ninja twice :frowning: