Cannot allocate memory (tcmalloc) building .qlf file + re_replace crash

Help!
1

I got a weird error from the latest version of swipl, so I tried rebuilding a version that I knew worked. But I got this error when building. This was done with a clean directory on a freshly rebooted machine.

FAILED: packages/ssl/tests/test_certs/generated 
cd /home/peter/src/swipl-devel/build/packages/ssl && /usr/bin/cmake -E make_directory tests && /home/peter/src/swipl-devel/build/src/swipl -f none --no-packs /home/peter/src/swipl-devel/build/packages/ssl/mkcerts.pl --source=/home/peter/src/swipl-devel/packages/ssl/tests --dest=tests && touch tests/test_certs/generated
tcmalloc: large alloc 1122051616153600 bytes == (nil) @ 
[FATAL ERROR: at Sun May  3 16:51:19 2020
	Could not allocate memory: Cannot allocate memory]
Aborted (core dumped)
[2146/2382] QLF compiling doc_html.qlf
ninja: build stopped: subcommand failed.

In case it’s related, here’s the crash I was trying to reproduce. The crash appears to have happened in re_replace in this code (when I removed the call to re_replace, the code worked):

print_term_cleaned(Term, Options, TermStr) :-
    % print_term leaves trailing whitespace, so remove it
    with_output_to(
            string(TermStr0),
            (current_output(TermStream),
             print_term(Term, [output(TermStream)|Options]))),
    re_replace(" *\n"/g, "\n", TermStr0, TermStr).

SWI-Prolog [thread 1 (main) at Sun May  3 16:47:44 2020]: received fatal signal 11 (segv)
C-stack trace labeled "crash":
  [0] save_backtrace() at /home/peter/src/swipl-devel/build/../src/os/pl-cstack.c:332 [0x7fd3ac627d48]
  [1] print_c_backtrace() at /home/peter/src/swipl-devel/build/../src/os/pl-cstack.c:867 [0x7fd3ac627efe]
  [2] sigCrashHandler() at /home/peter/src/swipl-devel/build/../src/os/pl-cstack.c:892 [0x7fd3ac627ff1]
  [3] dispatch_signal() at /home/peter/src/swipl-devel/build/../src/pl-setup.c:551 [0x7fd3ac5b692a]
  [4] killpg() at ??:? [0x7fd3ac15cf20]
  [5] pcre_dfa_exec() at ??:? [0x7fd3aa0aef2d]
  [6] pcre_exec() at ??:? [0x7fd3aa0bd0a1]
  [7] /tmp/swipl_3143_1(+0x3076) [0x7fd3aa307076]
  [8] PL_next_solution() at /home/peter/src/swipl-devel/build/../src/pl-vmi.c:3879 [0x7fd3ac551d9f]
  [9] query_loop() at /home/peter/src/swipl-devel/build/../src/pl-pro.c:144 [0x7fd3ac59bda4]
  [10] prologToplevel() at /home/peter/src/swipl-devel/build/../src/pl-pro.c:485 [0x7fd3ac59c64b]
  [11] PL_initialise() at /home/peter/src/swipl-devel/build/../src/pl-init.c:1107 [0x7fd3ac5ddd03]
  [12] /tmp/pykythe_test/pykythe.qlf(+0x756) [0x55bd94db7756]
  [13] __libc_start_main() at /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:344 [0x7fd3ac13fb97]
  [14] /tmp/pykythe_test/pykythe.qlf(+0x7aa) [0x55bd94db77aa]
Prolog stack:
  [30] pcre:re_foldl_/6 [PC=1 in supervisor]
  [28] pcre:re_replace/4 [PC=55 in clause 1]
  [26] pykythe_utils:dump_term_impl/3 [PC=29 in clause 1]
  [25] must_once:must_once/1 [PC=6 in clause 1]
  [22] pykythe_utils:do_if/2 [PC=10 in clause 1]
  [20] pykythe:process_nodes_impl/7 [PC=20 in clause 1]
  [18] pykythe:process_module_from_src_impl/5 [PC=94 in clause 1]
  [15] must_once:must_once/1 [PC=6 in clause 1]
  [13] apply:maplist_/2 [PC=8 in clause 2]
  [10] system:catch/3 [PC=2 in clause 1]
  [9] catch_with_backtrace/3 [PC=6 in clause 1]
  [8] pykythe:pykythe_main/0 [PC=33 in clause 1]
  [7] system:catch/3 [PC=2 in clause 1]
  [6] catch_with_backtrace/3 [PC=6 in clause 1]
  [5] $toplevel:run_init_goal/2 [PC=17 in clause 1]
  [2] system:catch/3 [PC=2 in clause 1]
  [0] system:$c_call_prolog/0 [PC=0 in top query clause]
Running on_halt hooks with status 139
2

Hmmm … the strange allocation bug doesn’t seem to consistently reproducible. The re_replace bug is. (I don’t know at which version of swipl re_replace was working because I haven’t run that particular debug code for a while.)

3

Bit odd. The build works fine with AddressSanitizer, which should pick up most allocation issues. The re_replace should be unrelated as pcre isn’t used in the build process. Can you produce the term on which it fails?

edit Pushed a possible fix for pcre package as this seems related to an older issue. Don’t know whether this fixes the issue as I cannot easily reproduce yours.

4

I noticed that the weird memory allocation is for ‘0x3fc8000002000’ bytes. I’ve never debugged x86 at the machine level, so don’t know how to interpret it further, but it looks suspicious.

To reproduce the re_replace bug, consult the attached file and run foo(Z).re_replace_bug.pl (8.5 KB)

5

Thanks. The re_replace bug doesn’t happen in the current HEAD and reproduces after going back to before the patch, so this seems fixed.

I cannot comment on the ssl issue. Seems a bug, but doesn’t reproduce on any of the quite a few machines used to build and test at the moment. It also doesn’t trigger under AddressSanitizer or valgrind. Could be anything, including some broken dependency.

The usual way out is to run the failing build step under gdb/lldb, get a stack trace and possibly the values of some related variables.

6

My best guess is that the bug is a dereference error … it isn’t an “ssl” error; it just shows up in the ssl test … the actual error is in creating a .qlf; and I got the same error (with the same weird number) in a separate creation of a .qlf, which also mysteriously went away. Wild guess: a missing mutex?

(I can’t rebuild swipl right now: source_sink library(doc_latex) does not exist; and I have zero knowledge of cmake and ninja)

7

It doesn’t all seem to be that dramatic. I doesn’t crash in QLF creation, but while creating the SSL test files. That is at least what I read from this:

To debug you re-run this command while putting gdb/lldb in front of /home/peter/src/swipl-devel/build/src/swipl and try to get a proper stack trace.

Should be fixed again. Made some CMake script changes that indeed causes the build to fail as I discovered trying to build the git on another machine than the dev machine :frowning:

8

I should not speculate on the causes of bugs before coffee. :clown_face:

I confirm that this particular instance of the bug has gone away. Note that other large strings did not trigger the bug; it took one specific string to trigger it. Therefore, I suspect that the bug hasn’t gone away; it’s just a rare Heisenbug and will appear again, maybe tomorrow, maybe a century from now.

9

It seems unlikely that the two bugs are related. The pcre crash is due to the way PL_get_nchars() and friends buffer temporary strings. This changed, causing wrong usage of this to (now) crash on long strings by reading unmapped memory. Before this caused (possibly) wrong output, so we are making progress IMO :slight_smile: Keri and I are looking into this.

Generating the SSL test data is done by a Prolog program (used to be shell, but that isn’t portable :slight_smile: ). That code doesn’t use pcre. It might be subject to the temporary string issues, but that too is unlikely as that should almost always lead to a SIGSEGV exception.

10

With the current git version, starting with a fresh build directory on macOS, I still get:

[3/156] Checking (git) version
Updated GIT version
[156/156] Generating home/doc/manindex.db
[11/12] Install the project...
-- Install configuration: "RelWithDebInfo"
CMake Error at src/cmake_install.cmake:36 (file):
  file INSTALL cannot find
  "/Users/pmoura/Documents/Prolog/swipl-devel/src/home/library/INDEX.pl": No
  such file or directory.
Call Stack (most recent call first):
  cmake_install.cmake:60 (include)


FAILED: CMakeFiles/install.util 
cd /Users/pmoura/Documents/Prolog/swipl-devel/build && /opt/local/bin/cmake -P cmake_install.cmake
ninja: build stopped: subcommand failed.

Any additional info I can provide to help diagnose and fix this build issue?

11

All seems find again. Sorry for the build troubles.

12

With the latest PPA (SWI-Prolog version 8.3.1 for x86_64-linux), I got this error:

[FATAL ERROR: at Mon Jun  8 14:46:15 2020
	Could not allocate memory: Cannot allocate memory]

I could not reproduce the error using after rebuilding with V8.3.1-2-g68f8757ae).

@jan - is this worth further investigation? If so, what do you suggest I do?

The problem seems to be related to initialization … I was running my test files by using the “-l” command line option (and some “-g” goals); when I removed the :- initialization directive and also the “-l” from the command line, the PPA version worked.

13

Possibly. Can you run under gdb and set a breakpoint on fatalError(). If that traps the backtrace may give a hint. All sounds pretty weird. The only thing I am aware of that if you link to a shared object that also links to tcmalloc things break badly. That causes issues when trying to load the prepackaged RocksDB, at least on Debian based systems.

14

This probably isn’t very helpful … maybe I need some additional arguments to gdb? (e.g., I didn’t seem to get a breakpoint on fatalError and the backtrace doesn’t have symbolic information - it appears that the swipl in the PPA is stripped. :frowning: I did git checkout V8.3.1 and rebuilt, but couldn’t reproduce the problem with that.

$ file /usr/bin/swipl /usr/lib/swi-prolog/bin/x86_64-linux/swipl ../swipl-devel/build/src/swipl
/usr/bin/swipl:                             symbolic link to ../lib/swi-prolog/bin/x86_64-linux/swipl
/usr/lib/swi-prolog/bin/x86_64-linux/swipl: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=3bc583d4364339939ccac607b72ad0404e2e2075, stripped
../swipl-devel/build/src/swipl:             ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=291d3264cafdca1e7ced8a02a58195929b69aa42, with debug_info, not stripped

$ gdb  --args /usr/bin/swipl --no-tty -g 'plunit:load_test_files([])' -g plunit:pykythe_run_tests -t halt -l pykythe/pykythe.pl -- --pythonpath='/tmp/pykythe_test/SUBST/home/peter/src:/tmp/pykythe_test/SUBST/BUILTINS:/home/peter/src/pykythe/typeshed/stdlib/3.7:/home/peter/src/pykythe/typeshed/stdlib/3:/home/peter/src/pykythe/typeshed/stdlib/2and3:/usr/lib/python3.7' test_data/dummy_dir/dummy_file.py
GNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/swipl...(no debugging symbols found)...done.
(gdb) b fatalError
Function "fatalError" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (fatalError) pending.
(gdb) run
Starting program: /usr/bin/swipl --no-tty -g plunit:load_test_files\(\[\]\) -g plunit:pykythe_run_tests -t halt -l pykythe/pykythe.pl -- --pythonpath=/tmp/pykythe_test/SUBST/home/peter/src:/tmp/pykythe_test/SUBST/BUILTINS:/home/peter/src/pykythe/typeshed/stdlib/3.7:/home/peter/src/pykythe/typeshed/stdlib/3:/home/peter/src/pykythe/typeshed/stdlib/2and3:/usr/lib/python3.7 test_data/dummy_dir/dummy_file.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff61eb700 (LWP 8493)]
[FATAL ERROR: at Tue Jun  9 11:17:18 2020
	Could not allocate memory: Cannot allocate memory]
[Thread 0x7ffff61eb700 (LWP 8493) exited]

Thread 1 "swipl" received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007ffff769b801 in __GI_abort () at abort.c:79
#2  0x00007ffff7a827e3 in ?? () from /usr/lib/swi-prolog/lib/x86_64-linux/libswipl.so.8
#3  0x00007ffff7b175a4 in ?? () from /usr/lib/swi-prolog/lib/x86_64-linux/libswipl.so.8
#4  0x00007ffff7b17649 in ?? () from /usr/lib/swi-prolog/lib/x86_64-linux/libswipl.so.8
#5  0x00007ffff7a7452a in ?? () from /usr/lib/swi-prolog/lib/x86_64-linux/libswipl.so.8
#6  0x00007ffff7aa7763 in ?? () from /usr/lib/swi-prolog/lib/x86_64-linux/libswipl.so.8
#7  0x00007ffff7aa70d8 in ?? () from /usr/lib/swi-prolog/lib/x86_64-linux/libswipl.so.8
#8  0x00007ffff7aaaf09 in ?? () from /usr/lib/swi-prolog/lib/x86_64-linux/libswipl.so.8
#9  0x00007ffff7ab0e3f in ?? () from /usr/lib/swi-prolog/lib/x86_64-linux/libswipl.so.8
#10 0x00007ffff7ab1a6f in ?? () from /usr/lib/swi-prolog/lib/x86_64-linux/libswipl.so.8
#11 0x00007ffff7ab259d in ?? () from /usr/lib/swi-prolog/lib/x86_64-linux/libswipl.so.8
#12 0x00007ffff7ab2f86 in ?? () from /usr/lib/swi-prolog/lib/x86_64-linux/libswipl.so.8
#13 0x00007ffff7b3b832 in ?? () from /usr/lib/swi-prolog/lib/x86_64-linux/libswipl.so.8
#14 0x00007ffff7b3bcb7 in ?? () from /usr/lib/swi-prolog/lib/x86_64-linux/libswipl.so.8
#15 0x00007ffff7a89afa in PL_next_solution () from /usr/lib/swi-prolog/lib/x86_64-linux/libswipl.so.8
#16 0x00007ffff7ad63f4 in ?? () from /usr/lib/swi-prolog/lib/x86_64-linux/libswipl.so.8
#17 0x00007ffff7ad6c9b in ?? () from /usr/lib/swi-prolog/lib/x86_64-linux/libswipl.so.8
#18 0x00007ffff7b182c3 in PL_initialise () from /usr/lib/swi-prolog/lib/x86_64-linux/libswipl.so.8
#19 0x00005555555547b6 in ?? ()
#20 0x00007ffff767cb97 in __libc_start_main (main=0x555555554790, argc=13, argv=0x7fffffffe0b8, 
    init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe0a8)
    at ../csu/libc-start.c:310
#21 0x000055555555480a in ?? ()
(gdb)
15

No, this doesn’t help much :frowning: Is your own build also linked against tcmalloc? Does it use any foreign libraries?

16

Yes, I use tcmalloc; there’s this line from cmake -G Ninja ..:

-- Found LibTCMalloc: /usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so

It’s part of package libgoogle-perftools-dev, which has these .so files:

/usr/lib/x86_64-linux-gnu/libprofiler.so
/usr/lib/x86_64-linux-gnu/libtcmalloc.so
/usr/lib/x86_64-linux-gnu/libtcmalloc_and_profiler.so
/usr/lib/x86_64-linux-gnu/libtcmalloc_debug.so
/usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so
/usr/lib/x86_64-linux-gnu/libtcmalloc_minimal_debug.so

But it appears that you aren’t using tcmalloc:

$ ldd /usr/bin/swipl  # Installed from PPA
	linux-vdso.so.1 (0x00007ffcad7fa000)
	libswipl.so.8 => /usr/lib/swi-prolog/lib/x86_64-linux/libswipl.so.8 (0x00007fba93f5c000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fba93b6b000)
	libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007fba93941000)
	libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007fba936c0000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fba934a3000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fba93284000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fba93080000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fba92ce2000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fba92ada000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fba944e7000)

$ ldd ~/src/swipl-devel/build/src/swipl
	linux-vdso.so.1 (0x00007fffa7f0f000)
	libtcmalloc_minimal.so.4 => /usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4 (0x00007f2a371b8000)
	libswipl.so.8 => /home/peter/src/swipl-devel/build/src/libswipl.so.8 (0x00007f2a36e2b000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2a36a3a000)
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f2a366b1000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f2a36313000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f2a360fb000)
	libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f2a35ed1000)
	libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f2a35c50000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f2a35a33000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f2a35814000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f2a35610000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f2a35408000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f2a37605000)

The relevant packages on my system (5.3.0-51-generic #44~18.04.2-Ubuntu SMP) are:

$ dpkg -s libgoogle-perftools4 libtcmalloc-minimal4
Package: libgoogle-perftools4
Status: install ok installed
Priority: optional
Section: libs
Installed-Size: 992
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Architecture: amd64
Source: google-perftools
Version: 2.5-2.2ubuntu3
Depends: libtcmalloc-minimal4 (= 2.5-2.2ubuntu3), libc6 (>= 2.27), libstdc++6 (>= 5.2), libunwind8
Conflicts: libgoogle-perftools0
Description: libraries for CPU and heap analysis, plus an efficient thread-caching malloc
 The gperftools, previously called google-perftools, package contains some
 utilities to improve and analyze the performance of C++ programs. This includes
 the full features: an optimized thread-caching malloc() and cpu and heap
 profiling utilities.
Homepage: https://github.com/gperftools/gperftools
Original-Maintainer: Daigo Moriwaki <daigo@debian.org>

Package: libtcmalloc-minimal4
Status: install ok installed
Priority: optional
Section: libs
Installed-Size: 371
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Architecture: amd64
Source: google-perftools
Version: 2.5-2.2ubuntu3
Depends: libc6 (>= 2.14), libgcc1 (>= 1:3.0), libstdc++6 (>= 5.2)
Description: efficient thread-caching malloc
 The gperftools, previously called google-perftools, package contains some
 utilities to improve and analyze the performance of C++ programs. This is a
 part of that package, and includes an optimized thread-caching malloc.
Homepage: https://github.com/gperftools/gperftools
Original-Maintainer: Daigo Moriwaki <daigo@debian.org>

$ dpkg -s swi-prolog swi-prolog-nox swi-prolog-x
Package: swi-prolog
Status: install ok installed
Priority: optional
Section: interpreters
Installed-Size: 69
Maintainer: Jan Wielemaker <jan@swi-prolog.org>
Architecture: amd64
Version: 8.3.1-1-g49260b9f7-bionicppa2
Depends: swi-prolog-nox (= 8.3.1-1-g49260b9f7-bionicppa2), swi-prolog-x (= 8.3.1-1-g49260b9f7-bionicppa2)
Suggests: swi-prolog-doc, prolog-el
Description: ISO/Edinburgh-style Prolog interpreter
 SWI-Prolog is a fast and powerful ISO/Edinburgh-style Prolog compiler with a
 rich set of built-in predicates. It offers a fast, robust and small
 environment which enables substantial applications to be developed with it.
 .
 SWI-Prolog additionally offers:
 .
  * A powerful module system
  * Garbage collection
  * Unicode character set handling
  * Unbounted integer and rational number arithmetic
  * Multithreading support
  * A powerful C/C++ interface
  * GNU Readline interface
Homepage: http://www.swi-prolog.org

Package: swi-prolog-nox
Status: install ok installed
Priority: optional
Section: interpreters
Installed-Size: 19612
Maintainer: Jan Wielemaker <jan@swi-prolog.org>
Architecture: amd64
Source: swi-prolog
Version: 8.3.1-1-g49260b9f7-bionicppa2
Depends: libarchive13 (>= 3.1.2), libc6 (>= 2.23), libedit2 (>= 3.1-20140620), libgmp10, libossp-uuid16, libpcre3, libreadline7 (>= 6.0), libssl1.1 (>= 1.1.1), libtinfo5 (>= 6), libyaml-0-2, zlib1g (>= 1:1.2.2), libgmp-dev, libedit-dev, libreadline-dev, libncursesw5-dev, libjs-jquery
Suggests: swi-prolog-doc, prolog-el
Breaks: swi-prolog (<< 5.8.2-1)
Description: ISO/Edinburgh-style Prolog interpreter (without X support)
 SWI-Prolog is a fast and powerful ISO/Edinburgh-style Prolog compiler with a
 rich set of built-in predicates. It offers a fast, robust and small
 environment which enables substantial applications to be developed with it.
 .
 SWI-Prolog additionally offers:
 .
  * A powerful module system
  * Garbage collection
  * Unicode character set handling
  * Unbounted integer and rational number arithmetic
  * Multithreading support
  * A powerful C/C++ interface
  * GNU Readline interface
 .
 This package contains a working SWI-Prolog installation with GUI components.
Homepage: http://www.swi-prolog.org

Package: swi-prolog-x
Status: install ok installed
Priority: optional
Section: interpreters
Installed-Size: 11338
Maintainer: Jan Wielemaker <jan@swi-prolog.org>
Architecture: amd64
Source: swi-prolog
Version: 8.3.1-1-g49260b9f7-bionicppa2
Depends: libc6 (>= 2.17), libfontconfig1 (>= 2.12), libjpeg8 (>= 8c), libx11-6, libxft2 (>> 2.1.1), libxpm4, libxt6, swi-prolog-nox (= 8.3.1-1-g49260b9f7-bionicppa2)
Description: User interface library for SWI-Prolog (with X support)
 SWI-Prolog is a fast and powerful ISO/Edinburgh-style Prolog compiler with a
 rich set of built-in predicates. It offers a fast, robust and small
 environment which enables substantial applications to be developed with it.
 .
 SWI-Prolog additionally offers:
 .
  * A powerful module system
  * Garbage collection
  * Unicode character set handling
  * Unbounted integer and rational number arithmetic
  * Multithreading support
  * A powerful C/C++ interface
  * GNU Readline interface
 .
 XPCE is an object-oriented symbolic programming environment for user
 interfaces. Although XPCE was designed to be language-independent, it has
 gained popularity most with Prolog.
Homepage: http://www.swi-prolog.org

$ ls -l /usr/lib/swi-prolog/lib/x86_64-linux/libswipl*
lrwxrwxrwx 1 root root      13 Jun  8 08:30 /usr/lib/swi-prolog/lib/x86_64-linux/libswipl.so -> libswipl.so.8
lrwxrwxrwx 1 root root      17 Jun  8 08:30 /usr/lib/swi-prolog/lib/x86_64-linux/libswipl.so.8 -> libswipl.so.8.3.1
-rw-r--r-- 1 root root 1587448 Jun  8 08:30 /usr/lib/swi-prolog/lib/x86_64-linux/libswipl.so.8.3.1
17

Shouldn’t tcmalloc only affect performance?

Anyway, I tried flipping the --threads option … the unit tests pass with --threads=no but I get the “Cannot allocate memory” error with threads=yes. (With /usr/bin/swipl; the --threads option has no effect on the executable I built.)

18

Interesting. I’ll check the PPA specs

The main reason to switch to tcmalloc was that some workloads make glibc’s ptmalloc (?) waste a lot of memory. Memory allocators vary in how much they round up objects, whether and when they join adjacent free objects, how they reuse objects, etc. There is typically also not an all out winner for allocation. So, yes, you can run out of memory with one allocator and not with another. The question is whether you really ran out of memory or not. You could use strace to figure out whether this is indeed caused by a failing system call (typically sbrk() or mmap()).

???

janw (swipl-devel; master *) 12_> swipl --no-threads
Welcome to SWI-Prolog (64 bits, version 8.3.1-12-gc15e3370d-DIRTY)
SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software.
Please run ?- license. for legal details.

    CMake built from "/home/janw/src/swipl-devel/linux"

For online help and background, visit https://www.swi-prolog.org
For built-in help, use ?- help(Topic). or ?- apropos(Word).

59 ?- thread_create(writeln(x), Id).
ERROR: No permission to create thread `user:writeln(x)' (threading disabled)
ERROR: In:
ERROR:   [11] thread_create(user:writeln(x),_26862,[])
ERROR:   [10] '$syspreds':thread_create(user:writeln(x),_26906) at /home/janw/src/swipl-devel/linux/home/boot/syspred.pl:1459
ERROR:    [9] <user>

At least we are making some progress :slight_smile:

19

Another data point … I tried upgrading on my WSL subsystem (which is Ubuntu 16.04, I think; repackaged by Microsoft). It worked fine – and also doesn’t use tcmalloc, according to ldd:

$ ldd /usr/bin/swipl
        linux-vdso.so.1 (0x00007fffebb96000)
        libswipl.so.8 => /usr/lib/swi-prolog/lib/x86_64-linux/libswipl.so.8 (0x00007febc04f0000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007febc00f0000)
        libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007febbfec0000)
        libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007febbfc30000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007febbfa10000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007febbf7f0000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007febbf5d0000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007febbf230000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007febbf020000)
        /lib64/ld-linux-x86-64.so.2 (0x00007febc0c00000)

I haven’t tried building and running on WSL; if you’d like me to, I can try it. (Upgrading to the newer version that uses Ubuntu 18.04 would be more work, however.)

Probably not relevant: I’m using library(plunit) with the .plt convention (my test line uses “-l” to load the main code without initialization – you can see my command line in the unhelpful traceback I gave yesterday).

Also, I presume that library(plunit) doesn’t use threads, which is why I don’t get a “No permission to create thread” error message. :wink:

BTW, it might make sense to ship both a stripped and an unstripped executable. Is there any harm in using an unstripped executable? - As far as I know it doesn’t use any more RAM, just a bit more disk space.

I’ll try strace on Ubuntu 18.04 later today.

20

FYI WSL 2 should be publicly available. I have been using it for two months now but signed up for the slow ring to get it. Singing up for the fast or slow ring should not be required anymore.