Ann: SWI-Prolog 8.3.2

Dear SWI-Prolog user,

I’m happy to announce SWI-Prolog 8.3.2. Notably users of multi-threading
are encouraged to update. Highlights:

  • Sebastian Sardina did a lot of work cleaning up JPL (the Java
    interface). Notably support Maven en reorganize and update the
    documentation. This version also supports SWI-Prolog’s dict
  • Adrian Wong contributed yet another set of documentation patches.
  • wildcard_match/2 and expand_file_name/2 provided proper support
    for ASCII only. Using Unicode patterns or targets could access
    memory outside allocated regions, causing crashes, leaking data
    or giving wrong result. Keri found this. I’ve reimplemented
    most of it to properly support full Unicode.
  • There was a bug in message queues that could cause thread_get_message/1,2
    to block indefinitely despite a matching message arrived. Fixing
    this bug results in much more predictable timing of HTTP responses
    and the thread regression tests.
  • There was a race condition in temporary module handling that could
    cause SWISH to crash.
  • Another race was possible when two threads tried to compile the same
  • The HTTP server worker scheduling has been stress tested and improved
    considerably. This notably avoids starting far too many workers under
    heavy load.

Most of this stuff should go into 8.2.1 as it addresses serious
stability issues. Some of the patches are quite large though, so I first
want to see how well they hold in the real world.

Enjoy --- Jan

P.s. The Macports update has failed due to a missing dependency. In theory that can be fixed by adding a patch file against the source. That is a bit too much work though, so unless someone wishes to become maintainer of the Macports version we can only hope we have more luck with 8.3.3.

SWI-Prolog Changelog since version 8.3.1

  • FIXED: library(prolog_codewalk): handle file/4 message terms.
    Matt Lilley.

  • FIXED: Location of jpl.jar in development tree to be able to run in
    the dev tree.

  • PORT: Windows: make condition variable abstraction compile again.

  • ADDED: Foreign API to reuse the internal lock-free hash table

  • DOC: Removed obsolete section on Boehm-GC support.

  • FIXED: Make module enumeration safe for temporary modules. This
    affects notably current_predicate/1 in SWISH. current_predicate/1
    using an unbound module now skips enumeration of temporary modules.

  • FIXED: Possible race condition when two threads try to compile the
    same file.

  • FIXED: Possible inconsistent source file reference count.

  • CLEANUP: No longer create a source file object for a derived source.
    This doesn’t seem to be used anywhere and the new garbage collecting
    approach for source files would reclaim the file immediately anyway.

  • FIXED: Make source file admin use reference counts and delete no
    longer needed source file admin entries. Discarding them is needed by
    Pengines. The hard-coded discard from ‘$destroy_module’/1 could cause
    a use-after-free issue in SWISH when using :- include(file)..
    The patch is rather involved. Probably there are easier ways to fix
    this particular issue. Usage based reclaim of source files is a step
    towards garbage collecting code in general though.

  • CLEANUP: Property balance ‘$start_consult’/2 and ‘$end_consult’/1.

  • ADDED: message_queue_property/2: property waiting(-Count) to find
    the number of threads waiting for this queue.

  • ENHANCED: thread_send_message/2: reduce time the queue mutex is held.

  • FIXED: thread_get_message/1,2 could not wakeup after a message arrived.

  • DEBUG: Improved debug channel MSG_QUEUE, notably avoid printing to
    the possibly redirected current output using pl_writeln().

  • FIXED: destroying affected tables should first collect the
    network. Edward Schwartz

  • DOC: Format * Justify paragraphs

    • Align CMake options

    • Align table columns

  • DOC: Fix typos in

  • DOC: Fix typos in manual

  • ADDED: wildcard_match/3. Required because wildcard_match/2 no longer
    listens to case sensitivity of the file system. Suggested by Keri Harris.

  • FIXED: Missing error case for a non-terminated [.. set.
    Included more test cases.

  • FIXED: wildcard_match/2: peek after string on unmatched “{”

  • FIXED: wildcard_match/2 and expand_file_name/2 used to print a
    warning and throw a domain_error on an illegal pattern. It now throws
    a syntax_error exception.

  • TEST: Added files test dir, now containing

  • FIXED: Memory leak in text canonicalization

  • FIXED: expand_file_name/2 and wildcard_match/2 - Avoid a possible
    crash using Unicode patterns. Found by Keri harris. - Fully support
    Unicode in patterns and file names. - Remove the length limit for
    compiled patterns (used to be 1024 chars).

  • MODIFIED: wildcard_match/2 used to be case insensitive if file names
    are case insensitive. Now wildcard_match/2 is always case insensitive.

  • DOC: Remove duplicate license information * Move the new license down
    to replace the old one

    • Refer directly to the license page on the website

    • Update the copyright year

Package cpp

  • ENHANCED: Add copy constructor. This prevents warnings from modern
    compilers and preserves the existing copy semantics.

Package http

  • ENHANCED: HTTP workers for new connections to keep track of the
    activity rate and adjust to the best wait policy, trading time for
    memory minimisation.

  • ENHANCED: http server backlog changed from 5 to 64. Probably doesn’t
    have much effect, but 5 seems old school.

  • FIXED: library(http/http_dyn_workers) could start far too many workers
    due to (1) waiting a negative time to check the queue was draining and
    (2) not respecting the max_workers setting.

Package jpl

  • IMPROVED: documentation, tutorials and initialization of engine

  • IMPROVED: doc pages, much better Development section

  • ADDED: Maven dependency for jpl.jar in doc

  • ADDED: Extended POM to deploy packages on GitHub

  • ADDED: Github Maven building process

  • UPDATED: Javadoc with Dict included

  • CLEANUP: Removed obsolete and unused code from org.jpl7.util

  • BUILD: Avoid warning if junit.jar is not present

  • ADDED: Maven support and dir layout - Upgraded to 7.6.1

  • MODIFIED: move all examples to src/examples/ folder - Fixed #62

  • ADDED: Dictionaries support

  • ADDED: Dictionary as a type of Term

Package pengines

  • ADDED: Make pengines redirected I/O work on created threads.

Package pldoc

  • FIXED: Manual to serve e.g., ``/pldoc/doc_for?object=*->/2’’.

Package xpce

  • ENHANCED: Better visible current tab and better positioning of
    the label.

There seems to be a file descriptor leak. I rolled back to V8.3.1 (278a4b032) and rebuilt to verify that it wasn’t a problem with my code.

I tried rebuilding with the latest version (V8.3.2-3-gd0d08885e), but that broke even worse (weird messages about “Unknown procedure”.

I’ll try rerunning with strace to try to figure out what’s going on.

So, there seem to be two problems with 8.3.2. One is very rare; the other I’m not sure of. In both cases, I don’t have a simple test case. I’ll send more details separately to @jan (but not today).

  1. (very rare) consulting a file sometimes gives a bogus “unknown procedure” message (there’s no missing procedure, and the procedure that’s claimed to be missing isn’t used). This is (I think) triggered by wrap_procedure and only shows up with --threads=yes (--threads=no works fine). I can reproduce it, but only with one particular source and goal. (I had a previous problem with compiling to a .qlf file that went away with --threads=no; but that mysteriously ceased to reproduce: Cannot allocate memory (tcmalloc) building .qlf file + re_replace crash)
  • The likely change:
    git difftool --tool=tkdiff 3314e0692fcd760d6935da4bf14f6fbdd50878e2 f91a985a3d7e3b3d37de616e9ffd72ea15b7d0a3
  1. FD leak
  • The likely change:
    git difftool --tool=tkdiff cf00d089bcef49e5a00fd410f4dda0613601e27e 528d0b03e5fca71a42f8438164d9cb2c0d80634c

For the FD leak, I’ve opened a github issue to track my progress. The leak only occurs sometimes, but so far I can’t identify the precise conditions that trigger it.

It appears that the FD leak only happens when a file is loaded (using load_files/2) that contains a directive to turn off autoloading. This triggers an infinite recursion of loading files from the library. I’m adding some instrumentation to the bootloader, to try to figure out what’s going on

A simple work-around is: don’t disable autoloading.

I think this should be fixed with 21e0a70e91d7aa9c5dd83b00f0d49e22c1ed8b60

Now I wonder whether it is a good idea to switch the autoload file from restarted saved state. The issue was due to poor interaction between explicit autoloading and implicit autoloading as performed by the saved state generation.

Confirmed that it’s fixed (and I’ve closed the issue on github).

Maybe start a new discussion on this? I’ve read through the autoload, resources, and load_files documentation, and it’s not entirely clear to me; also when I tried turning autoload off, I got various strange errors (I haven’t spent the time to look more deeply into them; I just left autoload on).