Ann: SWI-Prolog 9.3.4

Releases
1

Dear SWI-Prolog user,

SWI-Prolog 9.3.4 is ready for download. This version comes with a lot
of internal changes, which means that the stable releases (9.2.x) from
now on only backports critical bugs.

This version got the first two rounds of the planned internal reorganization. More concrete:

  • Prolog data is now 64 bits, regardless of the platform. In practice,
    this affects the Win32, WASM and small systems such as older
    Raspberry Pis. As a consequence, stack usage on these platforms
    nearly doubles, but stacks are now limited by memory rather than
    having an 128Mb limit. Efficient (tagged) integers now have
    57 bits on all platforms (was 25 on 32 bit platforms).
  • Prolog data now uses absolute addressing, rather than stack offsets.
    I expected this to have significant performance impact, as introducing
    relative addresses reduced performance (but was the chosen way to
    allow many threads to use all memory on 32 bit systems). Unfortunately
    this the impact is not measurable. Apparently modern CPUs do enough
    look ahead to get the data ready when needed in parallel. Nevertheless,
    this is a step ahead as it allows for further simplification, optimizing
    type tagging for 64 bits and new features such as direct access to terms
    in engines. Hopefully those changes will have some performance impact.

Although all tests pass on all platforms, some regression is likely. Please
report bugs. These changes made some latent bugs surface. Other highlights:

  • Fixed crash in PL_get_mpq()
  • Better memory management for bignum operations. Avoid possibly
    memory leak in format/3 on bignums. Introduce max_integer_size flag
    to limit the memory of bignums (avoiding resource errors). By default,
    bignums are limited to the stack limit.
  • Unicode fixes: QLF I/O and upcase/downcase that changes characters <= 255
    to larger code points.
  • Various portability issues
  • Sweep (GNU Emacs) issue with certain locales (@oskardrums)
  • Janus: Two Python object reference count issues. Added py_import/2,
    effectively realizing import module as alias

Also push Ubuntu PPA for Noble Numbat (24.04)

Enjoy --- Jan

SWI-Prolog Changelog since version 9.3.3

  • PORT: Include Ubuntu 24.04 (Noble Nombat) in PPA builds

  • PORT: Use -funwind-tables on ARM systems with glibc, also for clang

  • CLEANUP: Remove various unused macros and join macros that are now the
    same Also document the current state of the Prolog data representation.

  • ENHANCED: Removed accessing Prolog data using relative addresses This
    patch removes accessing Prolog data using relative addresses to the
    stack bases. This was long ago introduced to facilitating access to
    all memory on 32-bit systems using multiple threads.

    Now all prolog data is 64 bits, also on 32-bit systems and hence we can
    access all memory using a tagged pointer.

  • PORT: Deal with Android android-execinfo library.

  • FIXED: generation of swipl.pc pkg-config file Now uses LIBSWIPL_DIR to
    get the correct -L flag when the system is installed as global library.

  • FIXED: Reconsult when the loaded code sets the trace flag on
    a predicate. This was due to a conflict over the predicate flags.
    As the flags field was widened, we can avoid these problems.

  • PORT: #1265 Deal with static_assert() after label

  • FIXED: WASM: fix fid arg position in this.query call in
    __call_yieldable

  • FIXED: #1262 upcase_atom/2 and friends on ISO → wide If input is
    ISO-Latin-1 and the converted character requires wide characters
    the system trapped an assertion error. Now correctly converts to
    wide characters.

  • FIXED: Clang handling of max_integer_size Clang cannot handle
    setjmp()/longjmp in PL_next_solution(), but it can in the simple
    evalExpression() function. This does not have measurable impact on
    performance, so we enabled this as default. This fixes limiting the
    integer size for WASM and Clang builds on Linux. This problem did
    not surface on the MacOS version.

  • FIXED: Compute new stack size when we reach the limit. Suffered from
    integer overflow on 32-bit machines and got several edge cases wrong.

  • CLEANUP: Allow for gcc -Wconversion With 64 bit word it is easy
    to get conversion errors. Compiling with -Wconversion helps
    tracking these.

  • FIXED: comparison of MPZ and MPQ with 64 bit integers if long is
    4 bytes

  • FIXED: Push string when stack > 128Mb on 32-bit hardware

  • DEBUG: Added scripts/gdbinit Share gdb script with useful debugging
    commands. [no ci]

  • FIXED: Normalize LibBF numbers created from 64 bit integers on VS2022
    VS2022 does not support 128-bit integers and therefore we must use
    32-bit “limbs”

  • PORT: VS2022: void* arithmetic, wrong order for SIZEOF_WORD

  • FIXED: forAtomsInClause() handling of B_*_?C instructions These called
    PL_unregister_atom() on the atom, rather than the argument function.

  • FIXED: DEBUG(CHK_SECURE) clean of memory (wrong size)

  • FIXED: Type handling in copy_term/2 and friends.

  • ADDED: PL_atom_index() and PL_atom_from_index() Turn atoms into small
    objects that can be combined with other pointer sized objects with
    space for a tag.

  • BUILD: Respect $CFLAGS and $CXXFLAGS in a CMake build types

  • CLEANUP: Get rid two definitions for trail entries

  • CLEANUP: Prepare hash tables for the M64 data model This huge commit
    changes pl-table.c to be word → word rather than void* → void* as
    we plan to make words 64 bit, also on 32 bit systems. This patch also
    introduces 4 table types, TableWW, TableWP, TablePW and TablePP. These
    tables map between words and pointers in all 4 possible ways.

    This patch also involves small cleanups and fixes encountered while
    reading the code and use FOR_TABLE() everywhere, removing the old
    for_table() macro.

  • FIXED: prototype for get_string_text()

  • FIXED: make markAtomsOnGlobalStack() safe This could crash on all
    platforms. It is notably dangerous using the experimental O_M64
    model.

  • TEST: Disable max_integer_size test for emscripten Doesn’t handle
    longjmp() correctly in release mode. Works in debug mode, but that
    is not useful.

  • FIXED: write clauses holding > 0xffff Unicode chars in strings to
    QLF on Windows.

  • FIXED: forAtomsInClause() handling of B_*_?C instructions These called
    PL_unregister_atom() on the atom, rather than the argument function.

  • FIXED: Temp buffer handling when stack is close to top of address range

  • DEBUG: Fixed chk_secure report on invalid term due to
    PushPtr()/PopPtr() This construct to turn raw pointers into term
    references to preserve them during GC can create referenc pointers
    to the local stack. Another possible fix is to turn such references
    into direct term references. That avoids local stack references,
    but as this is only used around GC and GC itself uses local stack
    reference pointers, there is little to gain.

  • FIXED: unifiable/3 internal consistency Unifiable did not correctly
    handle stack resizing. This is detected when running using chk_secure.
    Not sure whether it can also cause real issues, but the new code is
    simpler and safe.

  • FIXED: powm/3 function: wrong type check on 3rd argument.

  • FIXED: Declaration of term stack for evaluating expressions

  • FIXED: Write bignums when inside arithmetic. Should normally not be
    needed, but debugging is much easier if we can.

  • PORT: Avoid spurious warnings from spoiling pthread_setname_np()
    test This used to use -Werror to ensure a compiler failure if the
    number of arguments do not match the prototype, but this is already
    enforced in todays compilers.

  • FIXED: use_module/2 with except and reexport/2 with except to
    print warning These predicates failed if the source did not export
    the excluded predicate. Now it prints a warning and ignores the
    non-existent predicate.

  • DEPRECATED: Sseek(), Stell()

  • FIXED: Another fileno issue.

  • ADDED: library(sandbox): allow for Dict.get(key,Default)

  • FIXED: Stream initialization for swipl-win.exe Was broken since
    recent changes keeping track of fileno. Reported by Jan Burse.

  • ADDED: PL_system_error() This patch also adds documetation for
    PL_fatal_error().

  • ADDED: Prolog flag max_integer_size This flag limits the allocations
    on behalve of bignum and rational number processing, DoS attacks
    by exhausting memory using huge numbers as well as poor interrupt
    handling due to lack of signal handling inside the bignum libraries.

  • ENHANCED: Use temprary malloc API for bignum arithmetic

  • ENHANCED: Keep small bignums on the stack

  • CLEANUP: Simplify arithmetic context This is no longer nested and can
    (thus) be simplified.

  • FIXED: format/3: Ensure balanced AR_BEGIN()/AR_END() On errors it was
    possible that AR_END() was not called. This may lead to memory leaking
    and errornous operation when linked in a process that also uses gmp.

  • ENHANCED: Improve memory reuse after writing huge integers.

  • FIXED: #1254 crash in PL_get_mpq()

Package clib

  • PORT: Avoid warning on __RCSID() in bsd-crypt.c Suppresses a warning
    on CRAN win builder

Package cpp

  • MODIFIED: PL_get_integer() no longer accepts floats

  • TEST: exceptions in blob callback

  • DEPRECATED: PlStream::tell(), PlStream::seek() Also fixes initilization
    errors. Uses PL_system_error() to signal misuse rather than assert()

  • FIXED: Possibly uninitialized variable This is a temporary patch.
    blob compare() and acquire() cannot raise a Prolog exception.
    This should print a message and abort the process.

  • FIXED: handle nil blobs in callbacks

Package http

  • MODIFIED: library(http/http_load) to verify certificates This library
    allows for loading files from http/https URLs. It used to skip
    certificate checking. Checking is now enabled and the Option argument
    may be used to disable it (using the cert_verify_hook option).

  • FIXED: Undefined predicate schedule_gc/2 in library(http/http_session)

Package pldoc

  • ENHANCED: Deal with a large selection of diacritics in the LaTeX
    backend. When converting PlDoc markdown to LaTex, we now generate
    proper LaTeX sequences for ten common diacritics.

Package semweb

  • FIXED: #112 C11 compatibility (no declaration after label)

  • FIXED: Message propagation when rdf_load_db/1 fails.

  • FIXED: Type error

Package ssl

  • PORT: Fix test for X509_get0_signature() on Windows. This is still
    problematic as -Werror -c assumes gcc/clang/… We could use
    try_compile as in cmake/CheckFloatingPointFormat.cmake of the
    main source?

Package sweep

Package swipy

  • DOC: janus.query() etc optional arguments is called inputs

  • FIXED: Memory leak when generating a Prolog dict from a Python dict

  • FIXED: Wrong Python refcount update. Reported by Jerry James as !8.
    That is is not entirely correct though.

  • FIXED: iterator janus.query(): properly handle data conversion
    exceptions

  • ADDED: py_import/2: Allows to import a module with a different name.

4 Likes
2

Thanks Jan, that is great work!
We have it tested in

$ eye --version
eye --quiet --version
EYE v10.5.1 (2024-04-24)
SWI-Prolog version 9.3.4
starting 57 [msec cputime] 42 [msec walltime]

and

$ eye https://eyereasoner.github.io/eye/reasoning/peano/peano.n3

works fine but using the local crashes

$ eye peano.n3
eye --quiet peano.n3
EYE v10.5.1 (2024-04-24)
SWI-Prolog version 9.3.4
starting 75 [msec cputime] 63 [msec walltime]

SWI-Prolog [thread 1 (main) at Thu Apr 25 13:04:20 2024]: received fatal signal 11 (segv)
C-stack trace labeled "crash":
  [0] save_backtrace() at /home/jdroo/github.com/SWI-Prolog/swipl-devel/src/os/pl-cstack.c:337 [0x7f1f6855b515]
  [1] print_c_backtrace() at /home/jdroo/github.com/SWI-Prolog/swipl-devel/src/os/pl-cstack.c:911 [0x7f1f6855b6e0]
  [2] sigCrashHandler() at /home/jdroo/github.com/SWI-Prolog/swipl-devel/src/os/pl-cstack.c:949 [0x7f1f6855b81b]
  [3] __sigaction() at ??:? [0x7f1f68233520]
  [4] do_unify___LD() at /home/jdroo/github.com/SWI-Prolog/swipl-devel/src/pl-prims.c:276 (discriminator 1) [0x7f1f684948e5]
  [5] unify_ptrs___LD() at /home/jdroo/github.com/SWI-Prolog/swipl-devel/src/pl-prims.c:507 [0x7f1f6849d6c7]
  [6] PL_next_solution___LD() at /home/jdroo/github.com/SWI-Prolog/swipl-devel/src/pl-vmi.c:729 [0x7f1f6845b0e8]
  [7] query_loop() at /home/jdroo/github.com/SWI-Prolog/swipl-devel/src/pl-pro.c:155 [0x7f1f684a2ae2]
  [8] prologToplevel() at /home/jdroo/github.com/SWI-Prolog/swipl-devel/src/pl-pro.c:583 [0x7f1f684a3461]
  [9] swipl(+0x10eb) [0x55ab3a4370eb]
  [10] __libc_init_first() at ??:? [0x7f1f6821ad90]
  [11] __libc_start_main() at ??:? [0x7f1f6821ae40]
  [12] swipl(+0x1135) [0x55ab3a437135]
Prolog stack:
  [169] labelvars/4 [PC=4 in clause 2]
  [166] labelvars/4 [PC=14 in clause 5]
  [164] labelvars/4 [PC=14 in clause 5]
  [162] labelvars/4 [PC=14 in clause 5]
  [160] labelvars/4 [PC=14 in clause 5]
  [158] labelvars/4 [PC=14 in clause 5]
  [156] labelvars/4 [PC=14 in clause 5]
  [154] labelvars/4 [PC=14 in clause 5]
  [152] labelvars/4 [PC=14 in clause 5]
  [150] labelvars/4 [PC=14 in clause 5]
    ...
    ...
  [82] labelvars/4 [PC=14 in clause 5]
  [80] labelvars/6 [PC=19 in clause 2]
  [76] istep/4 [PC=13 in clause 1]
  [75] <http://example.org/#add>/2 [PC=115 in clause 2]
  [74] <http://example.org/#add>/2 [PC=35 in clause 2]
  [73] <http://example.org/#add>/2 [PC=35 in clause 2]
  [72] <http://example.org/#add>/2 [PC=35 in clause 2]
  [71] <http://example.org/#add>/2 [PC=35 in clause 2]
  [70] <http://example.org/#add>/2 [PC=35 in clause 2]
  [69] <http://example.org/#add>/2 [PC=35 in clause 2]
  [68] <http://example.org/#add>/2 [PC=35 in clause 2]
  [67] <http://example.org/#add>/2 [PC=35 in clause 2]
  [66] <http://example.org/#add>/2 [PC=35 in clause 2]
    ...
    ...
  [30] <http://example.org/#add>/2 [PC=35 in clause 2]
  [29] <http://example.org/#multiply>/2 [PC=38 in clause 2]
Running on_halt hooks with status 139
Killing 276858 with default signal handlers
Segmentation fault
3

Thanks. That reproduces. Onto the stack fore tomorrow …

1 Like
4

Pushed c34b0d0b97dfaf71922b520b9eb737ec8a6433bd to fix this. Please do more testing!

1 Like
5

Thanks @jan for this prompt fix, am very impressed!
All our eye reasoning tests in eye/reasoning at master · eyereasoner/eye · GitHub now work fine:

----------------------
Running reasoning/test
----------------------

3outof5/                     89 msec OK
4color/                     121 msec OK
ackermann/                  670 msec OK
acp/                         90 msec OK
allen/                      138 msec OK
backward/                    79 msec OK
bcrule/                     448 msec OK
bi/                         116 msec OK
bmi/                        104 msec OK
bmt/                       1383 msec OK
ccd/                        301 msec OK
cha58/                       53 msec OK
complex/                    110 msec OK
crypto/                      58 msec OK
cs/                          98 msec OK
csv/                        143 msec OK
dcg/                        136 msec OK
derived-rule/                87 msec OK
diamond-property/           145 msec OK
disjunction-elimination/     86 msec OK
dp/                         800 msec OK
dqc/                        202 msec OK
dt/                        1366 msec OK
easter/                     166 msec OK
edt/                       2753 msec OK
entail/                      43 msec OK
euq/                         46 msec OK
fcm/                        130 msec OK
fgcm/                       242 msec OK
fibonacci/                   66 msec OK
filter/                      45 msec OK
gcc/                        115 msec OK
gedcom/                     155 msec OK
glass/                      102 msec OK
gps/                        611 msec OK
graph/                       56 msec OK
h2o/                         50 msec OK
hanoi/                       86 msec OK
image/                      313 msec OK
iq/                          50 msec OK
lee/                        495 msec OK
lingua/                   15020 msec OK
lldm/                        77 msec OK
lubm/                      5035 msec OK
map/                        346 msec OK
medic/                       44 msec OK
mi/                          44 msec OK
mmln/                       253 msec OK
n3-dev/                     282 msec OK
n3c/                        164 msec OK
n3gl/                        41 msec OK
n3patch/                     43 msec OK
n3quad/                      42 msec OK
n3s/                         43 msec OK
nbbn/                        58 msec OK
npnsurfaces/                617 msec OK
pack/                        50 msec OK
padovan/                     61 msec OK
peano/                      331 msec OK
pi/                         787 msec OK
polygon/                     90 msec OK
pptbank/                     50 msec OK
preduction/                 327 msec OK
qgen/                        79 msec OK
rdf-star-reasoning/          81 msec OK
rdf-star/                   103 msec OK
rdf12/                      309 msec OK
rdfsurfaces/               2831 msec OK
reif/                        79 msec OK
resto/                       56 msec OK
restpath/                    64 msec OK
rgb/                        133 msec OK
rif/                        335 msec OK
rpo/                         60 msec OK
rs/                         172 msec OK
seq/                         96 msec OK
skos/                        64 msec OK
snaf/                        79 msec OK
socrates-metachain-b2f/      85 msec OK
socrates-metachain-f2b/      87 msec OK
socrates-star/              124 msec OK
socrates/                    84 msec OK
swet/                       260 msec OK
tak/                        265 msec OK
temp/                        47 msec OK
tfcontext/                   55 msec OK
ttl-to-rdfa/               1139 msec OK
turing/                     138 msec OK
turtle-dev/                 273 msec OK
twf/                        157 msec OK
usm/                         49 msec OK
utf8/                        43 msec OK
witch/                       44 msec OK
zebra/                      119 msec OK

45 sec 94 OK 0 FAILED
6

The debugging tools are pretty good :slight_smile: I am quite happy that these large changes to the internals appear to be feasible. Although, I’m a bit disappointed that none of the thousands of tests spotted this one :frowning:

For all, please test. I’ll release 9.3.5 shortly as this is a rather crucial bug.

1 Like
7

I get a Wunused-variable (debian, clang)

cd swipl-devel/build
CC=clang CXX=clang++ cmake -DINSTALL_DOCUMENTATION=OFF -DSWIPL_PACKAGES_JAVA=OFF -DSWIPL_PACKAGES_X=OFF -DSWIPL_PACKAGES_ODBC=OFF -DUSE_GMP=OFF -DSWIPL_PACKAGES_PYTHON=OFF -S ..
make
...
/home/matthias/swipl-devel/src/pl-dict.c:235:3: warning: unused variable '__PL_ld' [-Wunused-variable]
  WITH_LD(arg)
  ^
/home/matthias/swipl-devel/src/pl-builtin.h:429:24: note: expanded from macro 'WITH_LD'
        for (PL_local_data_t *__PL_ld = (ld), *__loopctr = NULL; \
                              ^
/home/matthias/swipl-devel/src/pl-alloc.c:448:3: warning: unused variable '__PL_ld' [-Wunused-variable]
{ GET_LD
  ^
/home/matthias/swipl-devel/src/pl-builtin.h:352:35: note: expanded from macro 'GET_LD'
#define GET_LD    PL_local_data_t *__PL_ld = GLOBAL_LD;
8

Another problem in the Windows version, but this seems to be on the list already (test_string.pl, line 88).

?- string_lower("wh\u0178", L), string_upper(L, U).
L = "whĂż",
U = "WH\u009F".
9

Thanks. Fixed. There are probably quite a few more places where LD is no longer used, but compilers seem to be not very good finding them. Clang-17 on Fedora found another one :slight_smile:

B.t.w. why are you using clang? gcc does a significantly better job for SWI-Prolog, in particular on x64. But even on Fedora Asahi Remix on a Mac M1 (arm64) I get almost 20% speedup from gcc-13 compared to clang-17. The problem was mentioned here before: clang is poor at optimizing huge functions. Instead, we implement VM instructions using a function for each instruction. This improves clang’s optimization a lot, but the advantage is mostly lost in the resulting function call overhead.

10

Some of the R flavors use it, therefore, I tried it, as requested above :slight_smile:

11

Looks like a bug in Windows. This calls towupper() and towupper(255) (the ÿ) return 009F, which is a control character. The test works fine under Wine on Linux :slight_smile: As case conversion relies on the OS locale support, there is not much we can do if this is broken. I’ll disable the test for Windows.

You can freely mix gcc and clang. Surely for C on Linux.

12

There is a way out. While towupper() is broken, _wcsupr_s() can be used and also avoids that we cannot deal with code points > 0xffff on Windows. Pushed a fix and re-enabled the test.

1 Like