I’m happy to announce SWI-Prolog 8.3.7. This version comes with some
important fixes and improvements. Highlights:
Matt Lilley help finding a crash in the tabling test that notably
occurred on (his) MacOS, but can in theory affect any version.
Fix to expand_file_name/2 for Windows on non-ascii file names.
Made frozen/2 compatible with SICStus, reporting all delayed
goals (not only those from freeze/2) on all variables inside
a given term (rather than only on the toplevel variable).
Handle arg/3 calls in the VM if the last argument is a first
variable, i.e., known to be unbound by the compiler.
Avoid exception in markdown handling that can make parts of
the documentation unavailable.
Enjoy — Jan
SWI-Prolog Changelog since version 8.3.6
MODIFIED: Issue#665: Added read_term_with_history/2 and moved
read_history to library(backcomp). read_term_with_history/2 provides
access to all the read_term/2 options, notably the syntax changing
options. After discussion with Robert van Engelen.
MODIFIED: qsave_program/2: preserve the autoload flag in the calling
process and preserve autoloading if it is enabled and the save class is development
ENHANCED: frozen/2 to avoid copying. This not only enhances
performance, but also maintains the variable identity.
FIXED: Possible crash in tabling, notably occurring on MacOS due
to wrong assumptions wrt representing pointers in Prolog integers.
With a lot of help from Matt Lilley.
FIXED: frozen/2 could fail after recent enhancements.
FIXED: Handle failure inside ‘fast’ opcodes which D_BREAK replaces
with ‘slow’ opcodes that do not trigger FASTCOND_FAILED. To do this,
if we hit D_BREAK and there is a fast condition pending, convert it
into a real choicepoint
FIXED: Issue#657: Debug flags such as print_write_options in a
saved state were overruled by the initialization. Robert van Engelen.
DOC: Issue#658: read_history/6 wrong info for substituting the
event. Robert van Engelen.
ENHANCED: Compile simple arg/3 calls to the VM
ENHANCED: Allow listing and source level debugger to work for clauses
the contain non-atom blobs.
MODIFIED: numbervars/4 using the singletons(true) option to
number variables in LR order rather than in the order where the
second occurrence is found. This notably makes the output of
portray_clause/1-3 easier to read.
MODIFIED: frozen/2 now reports any goal that is delayed on some
variable inside the first argument rather than only goals delayed
due to freeze/2 and only checking for a direct variable.
MODIFIED: portray_clause/1,2,3: portray (a,b),c the same as a,(b,c). This is compatible with SICStus and makes sense as the
compiler also compiles both shapes to the same code.
FIXED: Issue#652: expand_file_name/2 for Windows: double UTF-8
encoding.
PPA: Removed eoan from PPA list
Package bench
PORT: Allow running on SICStus.
Package pldoc
FIXED: Markdown exception when processing a block like a code block
followed by a possibly header underline (— or ===).
Package sgml
ADDED: XML space mode strict that never changes white space.
Required for signature generation and checking. By Matt Lilley.
Regarding the arg/3 optimization, is there any chance this could be extended for the case where the third argument is a nonvar, i.e., testing vs. generating?
With 8.3.7 it appears that arg(3,T,A), A=1 is faster than arg(3,T,1).
Type "apropos word" to search for commands related to "word"...
Reading symbols from /tmp/swi-prolog-devel/src/build/src/swipl...
(No debugging symbols found in /tmp/swi-prolog-devel/src/build/src/swipl)
Attaching to program: /tmp/swi-prolog-devel/src/build/src/swipl, process 143699
[New LWP 143701]
[New LWP 145376]
[New LWP 145377]
[New LWP 145378]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
0x00007f18f2bda8f0 in __pthread_clockjoin_ex () from /usr/lib/libpthread.so.0
(gdb) bt
#0 0x00007f18f2bda8f0 in __pthread_clockjoin_ex () from /usr/lib/libpthread.so.0
#1 0x00007f18f32a7ee6 in pl_thread_join2_va () from /tmp/swi-prolog-devel/src/build/src/libswipl.so.8
#2 0x00007f18f32e9cae in PL_next_solution () from /tmp/swi-prolog-devel/src/build/src/libswipl.so.8
#3 0x00007f18f3327d8b in query_loop () from /tmp/swi-prolog-devel/src/build/src/libswipl.so.8
#4 0x00007f18f3327c2c in prologToplevel () from /tmp/swi-prolog-devel/src/build/src/libswipl.so.8
#5 0x000055fea96f6059 in main ()
(gdb)
We get there together At least that makes a lot more sense. The thread tests are full of things that may deadlock if something goes wrong. What platform, compiler, etc.? Does this repeat (notably ctest -R thread)?
Hmm. Doesn’t reproduce using gcc 10.0, glibc 2.31 (Ubuntu 20.04).
Not sure how to continue. Using ctest -V you get the command being executed. You can run this using gdb and examine the hung process and call PL_backtrace(25,0) in the appropriate thread.
I am getting the problem again, but only if I run the package manager by itsef (after removing the package manager cache). This is what I get from ctest by running in the build tree produced by the package manager (Please see the first EDIT, I got a segfault with a nice backtrace):
ctest -V -R thread
UpdateCTestConfiguration from :/home/u/.cache/yay/swi-prolog-devel/src/build/DartConfiguration.tcl
UpdateCTestConfiguration from :/home/u/.cache/yay/swi-prolog-devel/src/build/DartConfiguration.tcl
Test project /home/u/.cache/yay/swi-prolog-devel/src/build
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 26
Start 26: swipl:thread
26: Test command: /home/u/.cache/yay/swi-prolog-devel/src/build/src/swipl "-f" "none" "--no-packs" "-q" "/home/u/.cache/yay/swi-prolog-devel/src/swipl-8.3.7/src/test.pl" "--no-core" "thread"
26: Test timeout computed to be: 10000000
26: Running scripts from thread ............
26: WARNING: plunit_queue_gc:create: Did not reclaim queues in 10 runs.
26: This can be bad luck and is thus not considered a test failure.
26: WARNING: plunit_queue_gc:create: Did not reclaim queues in 10 runs.
26: This can be bad luck and is thus not considered a test failure.
26: plunit_queue_gc:create: Required 4 retrys before all queues were GCed
Perhaps this gives some more clues.
EDIT: I ran the ctest again and I got this:
ctest -V -R thread
UpdateCTestConfiguration from :/home/u/.cache/yay/swi-prolog-devel/src/build/DartConfiguration.tcl
UpdateCTestConfiguration from :/home/u/.cache/yay/swi-prolog-devel/src/build/DartConfiguration.tcl
Test project /home/u/.cache/yay/swi-prolog-devel/src/build
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 26
Start 26: swipl:thread
26: Test command: /home/u/.cache/yay/swi-prolog-devel/src/build/src/swipl "-f" "none" "--no-packs" "-q" "/home/u/.cache/yay/swi-prolog-devel/src/swipl-8.3.7/src/test.pl" "--no-core" "thread"
26: Test timeout computed to be: 10000000
26: Running scripts from thread .............
26: plunit_queue_gc:create_non_empty: Required 4 retrys before all queues were GCed
26: ..............
26: SWI-Prolog [thread 23 () at Wed Sep 9 17:53:06 2020]: received fatal signal 11 (segv)
26: C-stack trace labeled "crash":
26: [0] save_backtrace() at :? [0x7ff84fced3b6]
26: [1] print_c_backtrace() at :? [0x7ff84fced4d9]
26: [2] sigCrashHandler() at :? [0x7ff84fced5f6]
26: [3] dispatch_signal() at :? [0x7ff84fd3b7ec]
26: [4] __restore_rt() at sigaction.c:? [0x7ff84faa86a0]
26: [5] lookupBlob.cold() at pl-atom.c:? [0x7ff84fc5918b]
26: [6] lookupAtom() at :? [0x7ff84fcf5b25]
26: [7] PL_unify_text() at ??:? [0x7ff84fd25eda]
26: [8] concat.constprop.0() at pl-prims.c:? [0x7ff84fd34ee2]
26: [9] pl_atom_concat3_va() at pl-prims.c:? [0x7ff84fd34d83]
26: [10] PL_next_solution() at ??:? [0x7ff84fcf7413]
26: [11] callProlog() at :? [0x7ff84fd15ca1]
26: [12] start_thread() at pl-thread.c:? [0x7ff84fcb26ea]
26: [13] start_thread() at pthread_create.c:? [0x7ff84f5e73e9]
26: [14] __GI___clone() at :? [0x7ff84fb6b293]
26: Prolog stack:
26: [3] system:atom_concat/3 [PC=1 in supervisor]
26: [2] forall/2 [PC=11 in clause 1]
26: [0] system:$c_call_prolog/0 [PC=0 in top query clause]
26: Running on_halt hooks with status 139
26: Killing 259596 with default signal handlers
1/2 Test #26: swipl:thread .....................***Failed 3.40 sec
test 27
Start 27: swipl:thread_wait
27: Test command: /home/u/.cache/yay/swi-prolog-devel/src/build/src/swipl "-f" "none" "--no-packs" "-q" "/home/u/.cache/yay/swi-prolog-devel/src/swipl-8.3.7/src/test.pl" "--no-core" "thread_wait"
27: Test timeout computed to be: 10000000
27: Running scripts from thread_wait ......... done
27: All tests passed
2/2 Test #27: swipl:thread_wait ................ Passed 2.13 sec
The following tests passed:
swipl:thread_wait
50% tests passed, 1 tests failed out of 2
Total Test time (real) = 5.53 sec
The following tests FAILED:
26 - swipl:thread (Failed)
Errors while running CTest
EDIT2: Some more info. The problems seem to appear with PGO compilation only, with a PGO compiled binary sometimes I get these lines:
$ ctest -V -R thread
[...]
26: WARNING: plunit_queue_gc:create: Did not reclaim queues in 10 runs.
26: This can be bad luck and is thus not considered a test failure.
26: WARNING: plunit_queue_gc:create: Did not reclaim queues in 10 runs.
26: This can be bad luck and is thus not considered a test failure.
It is not reproducible all the time, but the problem seems to appear only with PGO compilation.
Hard to tell. A crash in atom_concat/3 suggests memory management issues. Are you using tcmalloc() or the system malloc()? You might want to run the test under valgrind. Yes/no PGO more looks like a timing issue or unexpected instruction rescheduling that should have been protected using a memory barrier. Notably if it is timing it may of course also depend on the hardware (notably the number of cores).
Compiled using gcc-10 with PGO on Ubuntu 20.04. Works fine.
I didn’t make any malloc configuration changes, and the binary is linked with libtcmalloc_minimal, so I presume it is using tcmalloc.
I ran the test under valgrind:
With the binary that I compiled myself (PGO/ninja), valgrind shows no leaks (valgrind --leak-check=full ctest -V -R thread )
With the binary compiled by the aur package manager, it hangs just like it happens when I run it without valgrind.
The only difference that I can tell from my own compilation and the aur package manager compilation is the ‘-fPIC’ flag, this is how the aur package manager compiles it:
We are not interested in leaks Valgrind by default does all the sanity checks on memory handling. I think you are running valgrind on ctest though. You need to run ctest -V -R thread to find the command it executes and then run valgrind on src/swipl with the right options.
valgrind src/swipl option ...
Yes The only suggestion I can think of is to build it exactly the way the package manager does (or at least as close as you get) and divide and conquer to find the option that matters. -fPIC can hardly be the issue as the real Prolog system is in libswipl.so and that is compiled with -fPIC anyway. This only causes the main executable swipl to be compiled with -fPIC, but I can hardly imagine that matters.
Here is the valgrind report (took a long time to finish):
$ valgrind -s /home/u/.cache/yay/swi-prolog-devel/src/build/src/swipl -f none --no-packs -q /home/u/.cache/yay/swi-prolog-devel/src/swipl-8.3.7/src/test.pl --no-core thread
==391775== Memcheck, a memory error detector
==391775== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==391775== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==391775== Command: /home/u/.cache/yay/swi-prolog-devel/src/build/src/swipl -f none --no-packs -q /home/u/.cache/yay/swi-prolog-devel/src/swipl-8.3.7/src/test.pl --no-core thread
==391775==
Running scripts from thread ................................................................
Warning: [Thread 23] thread_send_message/2: thread `<thread>(19,0x16939810)' does not exist
Warning: [Thread 23] thread_send_message/2: thread `<thread>(24,0x149ce610)' does not exist
Warning: [Thread 23] thread_send_message/2: thread `<thread>(24,0x15b4f260)' does not exist
Warning: [Thread 23] thread_send_message/2: thread `<thread>(19,0x149de990)' does not exist
............................... done
All tests passed
==391775==
==391775== HEAP SUMMARY:
==391775== in use at exit: 222,406,333 bytes in 46,648 blocks
==391775== total heap usage: 65,023,084 allocs, 64,976,436 frees, 4,856,095,363 bytes allocated
==391775==
==391775== LEAK SUMMARY:
==391775== definitely lost: 2,312 bytes in 137 blocks
==391775== indirectly lost: 0 bytes in 0 blocks
==391775== possibly lost: 201,026,965 bytes in 304 blocks
==391775== still reachable: 21,377,056 bytes in 46,207 blocks
==391775== of which reachable via heuristic:
==391775== newarray : 40,368 bytes in 1,141 blocks
==391775== suppressed: 0 bytes in 0 blocks
==391775== Rerun with --leak-check=full to see details of leaked memory
==391775==
==391775== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
At least that makes a lot more sense. The thread tests are full of things that may deadlock if something goes wrong. What platform, compiler, etc.? Does this repeat (notably 
Valgrind by default does all the sanity checks on memory handling. I think you are running valgrind on ctest though. You need to run 
The only suggestion I can think of is to build it exactly the way the package manager does (or at least as close as you get) and divide and conquer to find the option that matters.