[Mingw64/Win10] : swi-prolog can't locate foreign lib, pcre4pl

Discussion
21

I think the direction is fine. I’m not sure how many problems will follow. Each platform has its own way to find shared objects/DLLs and resolve symbols. The Windows search strategy starts at the directory where the executable lives which supports the normal way applications are packaged with a folder holding the executable(s) and DLLs. DLLs need to refer to all DLLs they use. That differs from Linux and other ELF based systems where we have a global dynamic symbol table that is used to resolve symbols an executable or .so file does not have itself. Thus, .so files do not need to refer to their dependencies as long as they know the symbols are available in the process.

A plugin DLL seems to work for WIndows because the typical dependency (libswipl.dll) is in the same dir as the executable. Note that there is win_add_dll_directory/1 to add directories to the DLL search (supported in more recent Windows releases, but the old ones have died mostly). Using that to add the plugin directory to the search may solve more problems.

22

Getting there slowly… I discovered some interesting behavior, MSYS2-specific. A bunch of relevant dlls are in /mingw64/bin, e.g. zlib1.dll. So I wanted to add this to the DLL search path.

Matthias@DESKTOP-A2T8IFC MINGW64 ~/swipl-devel/build.mingw64/src
$ ./swipl -g "win_add_dll_directory('/mingw64/bin')" -g "use_module(library(archive))" -g halt
(works)

Matthias@DESKTOP-A2T8IFC MINGW64 ~/swipl-devel/build.mingw64/src
$ ./swipl -g "win_add_dll_directory('c:/msys64/mingw64/bin')" -g "use_module(library(archive))" -g halt
(works)

The thing is that MSYS2 bash translates /mingw64 to c:/msys64/mingw64/bin. Note that “c:/msys64” is mounted on “/”.

Matthias@DESKTOP-A2T8IFC MINGW64 ~/swipl-devel/build.mingw64/src
$ mount
C:/msys64 on / type ntfs (binary,noacl,auto)
C:/msys64/usr/bin on /bin type ntfs (binary,noacl,auto)
C: on /c type ntfs (binary,noacl,posix=0,user,noumount,auto)

This does not work from within swipl, and below one can see why not:

$ ./swipl
Welcome to SWI-Prolog (threaded, 64 bits, version 8.5.12-23-g9c00eb16e-DIRTY)

?- win_add_dll_directory('/mingw64/bin'), use_module(library(archive)).
ERROR: c:/msys64/home/matthias/swipl-devel/build.mingw64/home/library/archive.pl:111:
ERROR:    c:/msys64/home/matthias/swipl-devel/build.mingw64/home/library/archive.pl:111: Initialization goal raised exception:
ERROR:    Das angegebene Modul wurde nicht gefunden.
etc.

?- absolute_file_name('/mingw64/bin', F).
F = 'c:/mingw64/bin'.

?- halt.

Technically, the response of absolute_file_name/2 is wrong, because the msys64 is missing. I guess this would be difficult to solve this from within swipl, although there might be a support for this from MSYS2, I don’t know. For a quick and partial fix, I think users wouldn’t mind a message if the directory of win_add_dll_directory(Dir) is not an absolute path, as well as a warning if it does not exist on the file system. Do you agree?

Just for the files, with the correct path, module import is working fine:

Matthias@DESKTOP-A2T8IFC MINGW64 ~/swipl-devel/build.mingw64/src
$ ./swipl
Welcome to SWI-Prolog (threaded, 64 bits, version 8.5.12-23-g9c00eb16e-DIRTY)

?- win_add_dll_directory('c:/msys64/mingw64/bin').
true.

?- use_module(library(archive)).
true.
23

Nasty. One would have assumed MSYS2 would make sure its global dll directory is searched :frowning: Another angle might be to look at prolog_to_os_filename/2 for mapping from Prolog to the OS. It is not clear what the os is though. I guess for most purposes we should translate to the MSYS2 path (which means do not translate), while for some (all?) of the Win32 APIs we need to translate to the Windows path. Isn’t a lot more going wrong wrt paths because the SWI-Prolog file access on Windows relies in part on the C runtime library (I guess that is from MSYS2 and does take care of the mounts) and in part uses Win32 API calls directly (Like AddDLLDirectory()).

I’ll hold the PR until things are clear. First of all, is there an MSYS2 runtime call to get the Windows file name and, if so, where should we use it?

It gets a bit of a mess on Windows :frowning: We have WSL, which luckily is just Linux. We have the normal MinGW build, which is 99% pure Windows and we have both MSYS2 and Cygwin, both providing a bit more Unix-like system.

24

I think from the technical perspective, there’s nothing really wrong: MSYS2 bash has its own rules for translating paths between Unix-style and Windowish, which also includes “mount points”. In other words, MSYS2 command line arguments are already translated to the canonical windows style, see the little program below:

#include <stdio.h>

int main(int argc, char** argv)
{
  if(argc > 1)
    printf("arg 1 = %s\n", argv[1]) ;

  return 0 ;
}

Invoking it with different arguments, we see that it acts quite idiosyncratic. For example, there’s no drive M: on my computer.

c7201178@PC105-C720 MINGW64 ~/path
$ ./a.exe 123
arg 1 = 123

c7201178@PC105-C720 MINGW64 ~/path
$ ./a.exe /mingw64
arg 1 = C:/msys64/mingw64

c7201178@PC105-C720 MINGW64 ~/path
$ ./a.exe /m
arg 1 = M:/

c7201178@PC105-C720 MINGW64 ~/path
$ ./a.exe /mm
arg 1 = C:/msys64/mm

I’d rather not dive into such a rabbit hole. In other words, for swipl-win, the OS is Windows, not MSYS. And there are no MSYS paths, just MSYS-bash paths (edit: which are invisible to the program). And I think this is actually intended by MSYS. In fact, since you mention Cygwin: I think Cygwin differs in that respect, trying to put Unix on top of Windows, with all the unix-stuff including paths.

The purpose of my PR is completely different, and only loosely related: It informs if a user gave a relative path as the argument of win_add_dll_directory/1, which according to the docs, requires absolute paths. A nice side effect of this is that MSYS swipl users are informed if they accidentally use MSYS-bash paths, that is, if they use something like win_add_dll_directory(’/mingw64/bin’) instead of win_add_dll_directory(‘c:/msys64/mingw64/bin’).

I hope that I could make things clearer. Sorry for being so persistent, but I mean, this is a PROLOG forum :slight_smile:

25

It actually makes sense. If I understand correctly, DLLs are not being searched for in the PATH by recent Windows versions. So, DLLs like zlib1.dll are not found even if they are in the PATH. On the other hand, DLLs are being searched for in the folder of the executable, that’s probably why zlib1.dll & friends are copied into swipl.exe’s folder.

For what it is worth, the cookie version, that is, win_add_dll_directory/2, succeeds on my system (Windows 10). If it wouldn’t, the current swipl falls back to modifying the PATH. But that’s not relevant here.

The only thing that needs to be done on MSYS is to add $MINGW_ROOT/bin to the DLL search path once in the beginning. I’ll keep experimenting with this before I suggest a PR. It won’t affect the regular Windows version.

26

I noticed one more thing in win_add_dll_directory/1,2:

(library/shlib.pl)

win_add_dll_directory(Dir) :-
    (   current_prolog_flag(windows, true)
    ->  (   catch(win_add_dll_directory(Dir, _), _, fail)
        ->  true
        ;   prolog_to_os_filename(Dir, OSDir),
            getenv('PATH', Path0),
            atomic_list_concat([Path0, OSDir], ';', Path),
            setenv('PATH', Path)
        )
    ;   domain_error(operating_system, windows)
    ).

win_add_dll_directory/2 may fail for two reasons: Either the OS does not support it (I think this mainly applies to old Windows versions), or Dir does not exist. I guess the former is the reason for the catch. If, however, the folder does not exist, it is a bit pointless to add Dir/OSDir to the PATH. In the current version, the “else”-branch is called whenever win_add_dll_directory/2 fails, and, most importantly, it succeeds in all cases.

I suggest the following:

  • informational or silent message if Dir is a relative path instead of an absolute path, for the reasons stated in the other posts.
  • warning if Dir does not exist. Edit: error instead of a warning, because I cannot figure out a use case in which this is intended.

I hope this makes sense.