Ann: SWI-Prolog 8.5.17

Dear SWI-Prolog user,

SWI-Prolog 8.5.17 is ready for download. For the normal Prolog
distribution there is not much new. Most of the work is in the WASM port
that allows running Prolog in your browser. As a result there have been
quite a few configuration and testing changes to better support single
threaded and otherwise limited builds.

For the normal Prolog version there merely two fixes. One is a
regression in the Windows version after the work on improving the
Windows file system interaction: open('NUL', ...) or in general
accessing the reserved file names in Windows was broken (reported by Paulo Moura). Second issue is
a possible memory corruption in shared tabling which used by
library(pcre).

The WASM version has been improved a lot. The interfaces should be
fairly stable now. They are documented in the main manual. One of the
reasons behind this release is to make this documentation available. The
docs should come online at
https://www.swi-prolog.org/pldoc/man?section=wasm

In a nutshell the status for the WASM version is

  • The build process is now fairly smooth, at least on Linux and MacOS

  • The #wasm_demo shell (REPL) demo shows that all basic interaction
    is possible. The demo itself is very limited, notably due to the
    use of a simple <textarea> for editing.

  • The port of @PaulBrownMagic’s application from Tau-Prolog show that
    event processing and DOM manipulation work. A partial Tau-Prolog
    compatible library(dom) illustrates how the interaction works.

  • There are high level friendly interfaces to call between
    JavaScript and Prolog. Calls can be in both directions and may
    be nested (JavaScript calling Prolog calling JavaScript, etc.)

  • We can load Prolog code from a URL, e.g., from GitHub raw content.
    This in addition to loading Prolog code from <script> elements,
    JavaScript strings or downloading to the local filesystem and loading.

    Enjoy — Jan

SWI-Prolog Changelog since version 8.5.16

  • DOC: More WASM documentation.

  • ENHANCED: notrace/1: avoid callback through C. This avoids possible
    C stack overflows and allows yielding from below notrace/1.

  • MODIFIED: WASM: Renamed js_yield/2 to await/2 and js_can_yield/2
    to is_async/2.

  • FIXED: Possibly dangling tref->trie pointer when shared tabling
    creates the same variant table in two threads and disposes one of them.
    This hopefully addresses issue#1041. It surely fixes a bug though.

  • FIXED: Possible corruption of tries. By using PL_BLOB_UNIQUE we
    good reuse an atom after freeing the trie and creating a new one at
    the same address. As we save the symbol anyway, making use of unique
    atoms is meaningless here.

  • WASM: Make ‘$confirm’/2 hookable and implement it by means of the
    browser confirm() function.

  • WASM: Make URL compilation work on slow connections and connections
    that do not provide a modification time through a HEAD request.

  • FIXED: Exception preservation in undo/1

  • WASM: Allow compiling from a URL.

  • WASM: js_yield/2: if the result is a single JavaScript string, return
    it as a Prolog string for reducing garbage.

  • FIXED: source_file/2 in mode (-,+) never reported any files.
    Breaks coverage analysis tool.

  • FIXED: on_signal/3 for SIGINT called the handler asynchronously.
    This should only happen if SIGINT is bound to the debug
    handler. Reported by Raivo Laanemets.

  • WASM: Added is_object/1,2 to test Prolog terms for being a JavaScript
    object reference (of some class).

  • WASM: Added pseudo function instanceof() to expression evaluation.

  • WASM: Ported Paul Brown’s Tau-Prolog example.

  • ADDED: library(dom), providing Tau-Prolog compatible DOM
    interaction. Partial implementation.

  • WASM: Added Prolog.load_string() and Prolog.load_scripts()

  • WASM: Typos in docs and mixup of arguments in rational numbers.

  • WASM: Re-implement sleep/1 using an abortable Promise.

  • WASM: Introduce the notion of abortable promises.

  • WASM: Wrap js_yield/2 to forward exceptions.

  • DOC: Started adding WASM documentation to main manual

  • WASM: Added js_can_yield/0.

  • WASM: Use a WeakMap rather than additional attributes to get the
    reverse object --> id mapping

  • WASM: Return all objects except for instances of “Object” by reference
    rather than as a dict.

  • WASM: Print blobs that embed JavaScript objecs as <js_Class>(Ref)
    rather than <js_object>(Ref).

  • ADDED: js_yield/2 may be passed a promise, causing Prolog to wakeup
    when the promise is resolved.

  • ENHANCED: PL_unify_bool() accepts same atom values as
    PL_unify_bool_ex() and PL_get_bool()

  • ENHANCED: js_yield: get return value as term.

  • ENHANCED: js_yield: allow passing a term rather than a string.

  • INSTALL: Windows: Copied deleting the Uninstall registry key to the
    64 bit section.

  • WASM: Much restructuring to prolog.js. Add Prolog.forEach() and
    Prolog.abort() to remain responsive when running slow queries.

  • FIXED: Backtracking after a foreign yield.

  • DOC: remove outdated restriction regarding and {} as operators

  • ENHANCED: prolog_colour:colourise_db/3: process asserted bodies

  • FIXED: prolog_colour:qualify_op/3: recognize as an operator.
    Use SSU rules.

  • CLEANUP: Moved default output routines to the core and Prolog
    class. Simpified and documented the shell.html and test.html examples.

  • WASM: Simplify starting the WASM version by moving more defaults into
    swipl-web.js.

  • FIXED: Windows: this registry key is not deleted by Uninstaller.exe
    Unistalling swipl at Windows 10 Add/Remove App window should now work

  • DOC: Add missing failure return info for more more foreign functions.

  • WASM: Use an unbound tag for dicts created from JavaScript objects.
    If the class is not “Object”, bind the Prolog tag to the JavaScript
    class name.

  • WASM: Introduce classes for Prolog objects we cannot represent
    naturally in JavaScript such as variables, compounds, etc.

  • CLEANUP: Large rewrite of WASM prolog.js to use ES6 class syntax
    for the Prolog class.

  • MODIFIED: WASM: representation of rational numbers.

  • WASM: Use -s BIGINT to exchange Prolog integers up to 64 bits.
    Update Prolog.get_integer() to return values as JavaScript number when
    possible and JavaScript bigint otherwise. Updated Prolog.toProlog()
    to convert bigint to a Prolog large integer.

  • TEST: Added test for Windows NUL files

  • FIXED: Issue#153: Windows: can no longer open reserved files such as
    nul. Reported by Paulo Moura.

Package http

  • WASM: Support more the HTML related infrastructure.

  • PACKAGE: Include http_quasiquotations.pl and the term_html.pl imported
    from the Pengines package into the minimal installation.

Package ltx2htm

  • FIXED: Memory error.

Package pengines

  • PACKAGE: Move term_html.pl to the http package. It is not specific
    for Pengines.

Package xpce

  • FIXED: PceEmacs sgml mode: use setup_call_cleanup/3 to ensure Prolog
    streams to editors are always closed.
3 Likes

Just tried builing under msys2:

  • little problem in packages/xpce (already submitted a PR). The problem did not show up earlier because I compiled with SWIPL_PACKAGES_X=OFF in the past
  • long path name support does not yet seem to work. It can also be reproduced with the normal swipl-win from the Windows Installer on swi-prolog.org, see below.

make_directory with paths of length 1, 200, and 200+200

?- make_directory(a).
true.

?- make_directory(a123456789b123456789c123456789d123456789e123456789f123456789g123456789h123456789i123456789j123456789k123456789l123456789m123456789n123456789o123456789p123456789q123456789r123456789s123456789t123456789).
true.

?- make_directory('a123456789b123456789c123456789d123456789e123456789f123456789g123456789h123456789i123456789j123456789k123456789l123456789m123456789n123456789o123456789p123456789q123456789r123456789s123456789t123456789/a123456789b123456789c123456789d123456789e123456789f123456789g123456789h123456789i123456789j123456789k123456789l123456789m123456789n123456789o123456789p123456789q123456789r123456789s123456789t123456789').
ERROR: directory `'a123456789b123456789c123456789d123456789e123456789f123456789g123456789h123456789i123456789j123456789k123456789l123456789m123456789n123456789o123456789p123456789q123456789r123456789s123456789t123456789/a123456789b123456789c123456789d123456789e123456789f123456789g123456789h123456789i123456789j123456789k123456789l123456789m123456789n123456789o123456789p123456789q123456789r123456789s123456789t123456789'' does not exist (No such file or directory)

Thanks. Merged.

I know. This seems a Windows limitation though. The low level is _wmkdir() and is correctly passed \\?\ + <the directory name> as a canonical absolute path. It works fine under Linux Wine, but indeed under Windows 11 it fails as you describe. The other issue is that most of these functions report too long names as ENOENT, meaning “existence error” :frowning: I don’t know what else I can do. If anyone knows how to improve on this, let me know.

The bash msys shell accepts the long paths in mkdir, so there should be some way to do this. I’ll try to debug this later today and report the result.

FWIW, my system is Win 10, not 11, but I guess that does not make a difference.

1 Like

Another minor thing is a uninitialized warning in packages/xpce/src/gnu/getdate.c, for your kind consideration:

Somewhere at line 850/851: if(yylen > 0) yyval = etc. The code in getdate.c seems to be parser-generated, therefore difficult to read. It seems, however, that there’s a more recent version available, which I think isn’t GPL either, e.g., here: Chrony: getdate.c | Fossies (see the comment at line 21).

There, the condition if(yylen > 0) is omitted with a comment. Maybe it can be used instead.

getdate.c says “This code is in the public domain and has no copyright” :slight_smile: I’m happy to make minor patches to it, but otherwise I intend to avoid touching this code as much as possible. None of my compilers complain about this. If your’s does, initializing at declaration with a comment seems the most simple solution.

Ah, ok, but the “new” code at the fossies archive says the same. But I see your other point
(rather not touch it)

69 /* First part of user prologue. */
70 #line 1 "getdate.y"
71
72 /*
73 ** Originally written by Steven M. Bellovin <[smb@research.att.com](mailto:smb@research.att.com)> while
74 ** at the University of North Carolina at Chapel Hill. Later tweaked by
75 ** a couple of people on Usenet. Completely overhauled by Rich $alz
76 ** <[rsalz@bbn.com](mailto:rsalz@bbn.com)> and Jim Berets <[jberets@bbn.com](mailto:jberets@bbn.com)> in August, 1990.
77 **
78 ** This code is in the public domain and has no copyright.

8.5.17 is available as official Docker image. See Docker Hub

1 Like

Actually, no. On my system, the path that is sent to _wmkdir is not absolute, and there’s no \\?\ prefix. I was able to make this to work on my Windows 10 using the, well, pseudocode below. Sorry for my bad C, but I hope it may be helpful anyway:

int
_xos_mkdir(const char *path, int mode)                 // aaaa
{ TCHAR buf[PATH_MAX];

  char buf1[PATH_MAX];
  if ( !_xos_absolute_filename(path, buf1, PATH_MAX) ) // c:/msys64/home/matthias/aaaa
    return -1;

  char buf2[PATH_MAX];
  if ( !_xos_os_filename(buf1, buf2, PATH_MAX) )       // c:\msys64\home\matthias\aaaa
    return -1;

  char buf3[PATH_MAX];
  sprintf(buf3, "\\\\?\\%s", buf2) ;                   // \\?\c:\msys64\home\matthias\aaaa

  if( !_xos_os_filenameW(buf3, buf, PATH_MAX) )        // Same in UTF16
    return -1 ;

  return _wmkdir(buf);
}

This were my use cases:

$ src/swipl
Welcome to SWI-Prolog (threaded, 64 bits, version 8.5.17-3-g7f90f6d1f-DIRTY)
SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software.
Please run ?- license. for legal details.

    CMake built from "c:/msys64/home/c7201178/swipl-devel/build"

For online help and background, visit https://www.swi-prolog.org
For built-in help, use ?- help(Topic). or ?- apropos(Word).

make_directory(aaaa).
true.

make_directory('aaaa/aaaa').
true.

make_directory(a123456789b123456789c123456789d123456789e123456789f123456789g123456789h123456789a123456789b123456789c123456789d123456789e123456789f123456789g123456789h123456789).
true.

make_directory('a123456789b123456789c123456789d123456789e123456789f123456789g123456789h123456789a123456789b123456789c123456789d123456789e123456789f123456789g123456789h123456789/a123456789b123456789c123456789d123456789e123456789f123456789g123456789h123456789a123456789b123456789c123456789d123456789e123456789f123456789g123456789h123456789').
true.

halt.

PS. If I want to delete these newly created folders from the Windows Explorer, the Windows “Wastebasket” complains that the file path is too long :slight_smile:

Oops. Thanks for catching this. The patch to deal with NUL, etc. broke this and I didn’t notice because the tests are normally executed on Linux+Wine which is handles long files much better than Windows itself :slight_smile:

Should work again. Please let me know.

Partial success! The uses cases from my previous post work fine now:

make_directory(aaaa).
make_directory('aaaa/aaaa').
make_directory(a123456789b123456789c123456789d123456789e123456789f123456789g123456789h123456789a123456789b123456789c123456789d123456789e123456789f123456789g123456789h123456789).
make_directory('a123456789b123456789c123456789d123456789e123456789f123456789g123456789h123456789a123456789b123456789c123456789d123456789e123456789f123456789g123456789h123456789/a123456789b123456789c123456789d123456789e123456789f123456789g123456789h123456789a123456789b123456789c123456789d123456789e123456789f123456789g123456789h123456789').

The unit test still fails, unclear what is going on. I tried to replace the test by something simple, such as

test(5) :-
    test_steps(safe,
        [ make_directory('aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'),
          make_directory('aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa')
        ]).

Which fails with a strange error message under both Linux and MSYS:

ERROR: c:/msys64/home/c7201178/swipl-devel/src/tests/files/test_file_names.pl:119:
        test 5 is subject to occurs check (STO):
ERROR: Finite trees (error checking): deterministic success in 0.00 seconds
ERROR: Rational trees: received error: make_directory/1: directory `aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' does not exist (File exists)
ERROR: Finite trees: received error: make_directory/1: directory `aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' does not exist (File exists)

Under Linux, it fails, as well, with a less strange error message.

I am wondering if this level of complexity* is needed for the unit test. Would you mind if I simplify them a bit?

*I mean expressions such as make_directory(/Dir/(long(100)=Dir2))

How do you run these? For me these tests run just fine. By default tests are executed under different occurs check settings. That is rather useless in this case. It seems the cleanup step is not executed correctly and thus it tries to create directories that already exist. My log on Windows 11 are below. Started by hand as I do not have ctest on that machine

PS P:\build.win64> src/swipl.exe ..\src\test.pl --no-core files
% SWI-Prolog test suite.
% To run all tests run ?- test.
%
Running scripts from files (test_file_names.pl).....[1.348sec].(test_files.pl).................[0.279sec].(test_glob.pl)..............................[0.026sec].(test_pipe.pl)...[0.085sec]. done
% Started at Thu Sep 22 15:08:04 2022
% 0.953 seconds cpu time for 1,904,281 inferences
% 7,262 atoms, 5,282 functors, 4,404 predicates, 69 modules, 185,808 VM-codes
%
%                     Limit   Allocated      In use
% Local  stack:           -       20 Kb    2,312  b
% Global stack:           -       28 Kb    6,568  b
% Trail  stack:           -       30 Kb        0  b
%        Total:    1,024 Mb       78 Kb    8,880  b
%
% 22 garbage collections gained 10,110,200 bytes in 0.016 seconds.
% 29 atom garbage collections gained 3,572 atoms in 0.000 seconds.
% 32 clause garbage collections gained 620 clauses in 0.016 seconds.
% Stack shifts: 2 local, 3 global, 6 trail in 0.000 seconds
% 2 threads, 0 finished threads used 0.000 seconds
All tests passed

Yours is working fine:

c7201178@PC105-C720 MINGW64 ~/swipl-devel/build
$ src/swipl.exe ../src/test.pl --no-core files
% SWI-Prolog test suite.
% To run all tests run ?- test.
%
Running scripts from files (test_file_names.pl).....[0.323sec].(test_files.pl).................[0.103sec].(test_glob.pl)..............................[0.015sec].(test_pipe.pl)...[0.078se
c]. done
% Started at Thu Sep 22 18:06:30 2022
% 0.344 seconds cpu time for 1,830,831 inferences
% 6,774 atoms, 4,998 functors, 4,336 predicates, 65 modules, 176,579 VM-codes
%
%                     Limit   Allocated      In use
% Local  stack:           -       20 Kb    2,040  b
% Global stack:           -       28 Kb    6,296  b
% Trail  stack:           -       30 Kb        0  b
%        Total:    1,024 Mb       78 Kb    8,336  b
%
% 23 garbage collections gained 10,058,840 bytes in 0.000 seconds.
% 27 atom garbage collections gained 3,387 atoms in 0.000 seconds.
% 30 clause garbage collections gained 599 clauses in 0.000 seconds.
% Stack shifts: 2 local, 3 global, 6 trail in 0.000 seconds
% 2 threads, 0 finished threads used 0.000 seconds
All tests passed

ctest does not:

$ ctest -R files --output-on-failure
Test project C:/msys64/home/c7201178/swipl-devel/build
    Start 15: swipl:files
1/1 Test #15: swipl:files ......................***Failed    0.62 sec
Running scripts from files ...
ERROR: c:/msys64/home/c7201178/swipl-devel/src/tests/files/test_file_names.pl:119:
        test 5: ls([],['.',..])

.
Script c:/msys64/home/c7201178/swipl-devel/src/Tests/files/test_file_names.pl failed
..................................................... done
*** 1 tests failed ***


0% tests passed, 1 tests failed out of 1

Total Test time (real) =   0.65 sec

The following tests FAILED:
         15 - swipl:files (Failed)
Errors while running CTest

Interesting :frowning: You can run ctest -V -R files to get the full command line and then figure out which difference to the above simplified command makes the difference.

Surely the ls test, using the opendir() emulation with uses FindFirstFileW(), etc. is one of the Windows primitives that doesn’t seem to like long paths very well :frowning: I have no clue why. Didn’t find some other way to enumerate the directory members that might work …

Quick question: What about using exists_directory/1 or delete_directory/1 for testing? (instead of ls). I see that this may hide other, underlying problems, of course.

The cleanup is done by delete_directory_and_contents(TestDir), which in the end uses directory_files/2. The ls is a test to see whether we can correctly get the contents of a directory. That too uses directory_files/2. directory_files/2 is implemented using opendir() which uses FindFirstFileW().

Ah, got it.

  • make_directory(…long dir name…) works
  • directory_files(long, F) works, F = [., …]
  • make_directory(long/long) works too
  • directory_files(long/long, F) unifies F with the empty list
  • make_directory(long/long/long) works again
  • directory_files(long/long, F) still unifies F with the empty list

Yes. That is also my conclusion. Now the question becomes whether it is me (still) doing something wrong or it is just Windows being broken? I think my calls to FindFirstFileW() are as good as it gets and I haven’t found an alternative. On the other hand, the Windows explorer can enter the long directories. One could consider changing the working directory and use relative paths, but SetCurrentDirectoryW() is deprecated AFAIK and definitely doesn’t like long paths. Microsoft is right that changing working directory in threaded applications is typically a bad idea.

Note that on Wine this all works fine :slight_smile:

I think I found the problem, see this patched opendir in uxnt.c.

The patch below works, although it might be a better idea not to invoke openfile with a directory that has a trailing backslash.

opendir(const char *path)
{ TCHAR buf[PATH_MAX];
  TCHAR *edir;
  DIR *dp = malloc(sizeof(*dp));

  if ( !dp )
  { errno = ENOMEM;
    return NULL;
  }

  /* -2: make sure there is space for the "\*" */
  if ( !_xos_os_filenameW(path, buf, PATH_MAX-2) )
  { free(dp);
    return NULL;
  }

  // In case there's already a backslash at the end, append only a "*"
  edir = buf+_tcslen(buf);
  if('\\' == buf[_tcslen(buf)-1])
    _tcscat(edir, _T("*")) ;
  else
    _tcscat(edir, _T("\\*"));

  if ( !(dp->data = malloc(sizeof(WIN32_FIND_DATA))) )
  { free(dp);
    errno = ENOMEM;
    return NULL;
  }
  dp->first = TRUE;
  // wchar_t *pattern = buf+_xos_win_prefix_length(buf); /* see (*) */ // do not cut the \\?\

  dp->handle = FindFirstFileExW(/* pattern */ buf, // use buf not pattern
                                FindExInfoBasic, dp->data,
                                FindExSearchNameMatch, NULL,
                                FIND_FIRST_EX_LARGE_FETCH);

  if ( dp->handle == INVALID_HANDLE_VALUE )
  { *edir = 0;

    if ( _waccess(buf, R_OK) )          /* Dir does not exist or is unreadable */
    { closedir(dp);
      return NULL;
    }
  }

  return dp;
}
1 Like

You saved my day! Seems I overlooked several things :frowning: Test now passes except that I had to prevent the test suite from trying to run the relative file access tests from deep directories for Windows. Passing "\\?\" to SetCurrentDirectoryW() gives some interesting results as that causes processes started from SWI-Prolog to get the directory including this prefix and many applications do not like that. Who knows though, maybe I missing something too :slight_smile:

1 Like