Swi-prolog build: various questions

[maren]

I’m currently rewriting much of the nixpkgs packaging for SWI-Prolog. Nix lets you specify all kinds of options, such as withGui, or withPython, to conditionally build a package with more features. To get the most out of this I’ve been studying the build to see what options exist.

For the most part it is straightforward, as a lot of features are auto included when a library is available at build time (and most such features can additionally be turned off explicitly). However, there are some options here that are a bit unclear to me. Hopefully someone can explain them.

gmp or libBF

If I understand correctly, swi-prolog will include gmp if

• gmp is available in the build environment
• USE_GMP=ON (which is the default)

Otherwise, swi-prolog will build with libBF.
Is this understanding correct? And how stable is this assumption going forward?

readline or libedit

Both readline and libedit have their own included prolog libraries. It follows that for a full library set, we have to depend on both. Interestingly, unlike many other libraries, these two don’t seem to have an option to turn off their package.

Separate from those libraries though, swi-prolog itself also uses either readline or libedit for its top-level.

I assume that by default readline will be used by swi-prolog for its top-level, and if it is unavailable but libedit is there, libedit will be used instead. Is that true?

qt

There’s a lot of qt stuff in swi-prolog. I was personally unaware that there was any qt support. Is this there for the XPCE implementation on non-X? Is it possible to use qt to build the graphical parts of swi-prolog instead of the ‘normal’ X way on linux?

engines

There’s an ENGINES build flag, but it is very unclear to me what it does. By default it is off, but nevertheless, swipl still builds with all the usual engine predicates.
What does this flag actually do?

protobufs

It looks like the protobufs library requires some things to be available at build time, but the build does not fail when these are missing, and I still seem to be getting a protobufs library. @peter.ludemann maybe you can tell me more about how this hooks into the build? I’m not sure how the protobufs library is supposed to work and how to test if it does what it should.

That’s it, for now

[edit] just one more…

xpce dependencies, xinerama?

It is a bit unclear what you really need to build xpce. There’s some common sets of dependencies floating around online, but I find some of them dubious.

A lot recommend including libxinerama. This is a library to support multi-display output on X. Is SWI-Prolog actually directly using this in any way? I seem to be able to compile without just fine, but I don’t know if there’s runtime consequences.

For what it’s worth, the minimum set of extra packages I need to get xpce to build using nixpkgs is:

• libXt - X toolkit
• libXext - common X extensions
• libXpm - X pixmap format
• libXft - X freetype fonts
• libjpeg - jpeg format support

This pulls in other dependencies of course, notably libX11.

[more edit]
It turns out that libxinerama is an optional dependency of xpce. So it builds without, but it still does something (what?) when present.

[peter.ludemann]

@maren

It looks like the protobufs library requires some things to be available at build time, but the build does not fail when these are missing, and I still seem to be getting a protobufs library.

There are two components to protobufs – an executable that allows transmitting and receiving protobufs and a compiler plugin (for Google’s protoc). The transmitting/receiving component is stand-alone.(*)

The Cmake flag TEST_PROTOBUFS_PROTOC (default OFF) controls whether the compiler plugin is tested … the plugin requires having protoc installed and it’s not reasonable to require that in general; it also requires python3 and its plugin plus C++'s plugin, for interoperability testing. There are some additional complications if the bootstrap ever needs to be run again (the protobuf compiler outputs protobufs, so the plugin needs the Prolog descriptors of those protobufs), but that’s highly unlikely to be needed unless some major change is made to the implementation.

I presume that @jan’s release process has at least one build with
-DTEST_PROTOBUFS_PROTOC=ON

I doubt that any of the Windows builds test the protobuf compiler plugin; my understanding is that protoc isn’t well supported on Windows but it’s trivial to use the output of protoc on Windows (after running protoc on a Linux system).

(*) There are two flavours of predicates for transmitting/receiving protobufs: one requires the output of the protoc compiler and the other (lower-level) doesn’t. The original protobufs implementation only provided the lower-level message handling, which primarily concerns serialization and deserialization; I added support for the higher-level message handling and the protoc plugin. In the world of protobuf implementations, this is a bit unusual; most only have support for the higher-level message handling and the plugin.

[maren]

Hi Peter, thanks for your prompt response.
So if I understand you correctly (and reading packages/protobufs/interop/README.md), for packaging SWI-Prolog I should not have to worry about protobuf dependencies at compile time. This is only a concern when running an extended set of tests, which is turned off by default.

Out of curiousity I added an option to my build options to include them anyway. These tests unfortunately fail for me.

The following tests FAILED:
         60 - protobufs:bootstrap (Failed)
         61 - protobufs:interop (Failed)
         62 - protobufs:demo (Failed)
Errors while running CTest
FAILED: CMakeFiles/test.util 
cd /build/source/build && /nix/store/mk7637n6h4d437gl5cplxvd3k2l66awd-cmake-3.29.2/bin/ctest --force-new-ctest-process
ninja: build stopped: subcommand failed.
error: builder for '/nix/store/0xqpg0nvb23qhdmgbxf277gjcmv0rkwh-swi-prolog-9.3.8-69360aa.drv' failed with exit code 1;
       last 10 log lines:
       > Total Test time (real) =   4.43 sec
       >
       > The following tests FAILED:
       >     60 - protobufs:bootstrap (Failed)
       >      61 - protobufs:interop (Failed)
       >        62 - protobufs:demo (Failed)
       > Errors while running CTest
       > FAILED: CMakeFiles/test.util

Not sure why. Do these tests pass for you?

[peter.ludemann]

Yes, the test suite works for me for protobufs. You might need to install all the dependencies and build/test in a fresh build directory.

ctest output

Compilation started at Sun Jul 21 20:43:37

$ LANG=en_US.UTF-8 /usr/bin/ctest -j1 -V -R protobufs

UpdateCTestConfiguration  from :/home/peter/src/swipl-devel/build/DartConfiguration.tcl
UpdateCTestConfiguration  from :/home/peter/src/swipl-devel/build/DartConfiguration.tcl
Test project /home/peter/src/swipl-devel/build
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 59
    Start 59: protobufs:protobufs

59: Test command: /home/peter/src/swipl-devel/build/src/swipl "-p" "foreign=" "-f" "none" "--no-packs" "--on-error=status" "-s" "/home/peter/src/swipl-devel/packages/protobufs/test_protobufs.pl" "-g" "test_protobufs" "-t" "halt"
59: Working Directory: /home/peter/src/swipl-devel/build/packages/protobufs
59: Test timeout computed to be: 10000000
59: % Start unit: protobuf_message
59: % [1/54] protobuf_message:original .............................. passed (0.033 sec)
59: % [2/54] protobuf_message:Test1a..th canned Golden Template ..... passed (0.000 sec)
59: % [3/54] protobuf_message:Test2 ..oogle's Golden Wirestream ..... passed (0.000 sec)
59: % [4/54] protobuf_message:Test3,..to parsed Golden Template ..... passed (0.001 sec)
59: % [5/54] protobuf_message:Test4,..olden Wirestream to Codes ..... passed (0.000 sec)
59: % [6/54] protobuf_message:Test5,..to parsed Golden Template ..... passed (0.001 sec)
59: % End unit protobuf_message: passed (0.038 sec CPU)
59: % Start unit: protobuf_message_2
59: % [7/54] protobuf_message_2:my_enum_msg ......................... passed (0.000 sec)
59: % [8/54] protobuf_message_2:embedded_key_value .................. passed (0.000 sec)
59: % End unit protobuf_message_2: passed (0.001 sec CPU)
59: % Start unit: some_message_example
59: % [9/54] some_message_example:some_message_wire ................. passed (0.000 sec)
59: % End unit some_message_example: passed (0.000 sec CPU)
59: % Start unit: repeated_fields
59: % [10/54] repeated_fields:packed1 ............................... passed (0.005 sec)
59: % [11/54] repeated_fields:not_packed_repeated ................... passed (0.000 sec)
59: % [12/54] repeated_fields:not_packed_repeated2 .................. passed (0.000 sec)
59: % [13/54] repeated_fields:not_packed_repeated2_gmp .............. passed (0.000 sec)
59: % [14/54] repeated_fields:packed_repeated ....................... passed (0.000 sec)
59: % [15/54] repeated_fields:packed_repeated2 ...................... passed (0.000 sec)
59: % [16/54] repeated_fields:packed_and_unpacked_repeated .......... passed (0.000 sec)
59: % [17/54] repeated_fields:packed_and_unpacked_repeated_gmp ...... passed (0.001 sec)
59: % [18/54] repeated_fields:repeated_key_value .................... passed (0.000 sec)
59: % End unit repeated_fields: passed (0.008 sec CPU)
59: % Start unit: protobuf_segment_convert
59: % [19/54] protobuf_segment_convert:protobuf_message ............. passed (0.000 sec)
59: % [20/54] protobuf_segment_convert:protobuf_message2 ............ passed (0.000 sec)
59: % [21/54] protobuf_segment_convert:message_string1 .............. passed (0.000 sec)
59: % [22/54] protobuf_segment_convert:message_string2 .............. passed (0.000 sec)
59: % [23/54] protobuf_segment_convert:message_string3 .............. passed (0.000 sec)
59: % [24/54] protobuf_segment_convert:message_length_delimited1 .... passed (0.000 sec)
59: % [25/54] protobuf_segment_convert:message_length_delimited2 .... passed (0.000 sec)
59: % [26/54] protobuf_segment_convert:string_length_delimited1 ..... passed (0.000 sec)
59: % [27/54] protobuf_segment_convert:string_length_delimited2 ..... passed (0.000 sec)
59: % End unit protobuf_segment_convert: passed (0.003 sec CPU)
59: % Start unit: codes
59: % [28/54] codes:uint32_codes .................................... passed (0.000 sec)
59: % [29/54] codes:uint32_codes_e1 ................................. passed (0.000 sec)
59: % [30/54] codes:uint32_codes_e2 ................................. passed (0.000 sec)
59: % [31/54] codes:uint32_codes_e3 ................................. passed (0.000 sec)
59: % [32/54] codes:uint32_codes_e4 ................................. passed (0.000 sec)
59: % [33/54] codes:uint64_codes .................................... passed (0.000 sec)
59: % [34/54] codes:uint32_int32 .................................... passed (0.000 sec)
59: % [35/54] codes:uint32_int32_e1 ................................. passed (0.000 sec)
59: % [36/54] codes:uint32_int32_e2 ................................. passed (0.000 sec)
59: % [37/54] codes:uint32_int32_e3 ................................. passed (0.000 sec)
59: % [38/54] codes:int64_float64 ................................... passed (0.003 sec)
59: % [39/54] codes:int32_float32 ................................... passed (0.000 sec)
59: % End unit codes: passed (0.006 sec CPU)
59: % Start unit: zigzag
59: % [40/54] zigzag:zigzag ......................................... passed (0.000 sec)
59: % [41/54] zigzag:zigzag_e1 ...................................... passed (0.000 sec)
59: % [42/54] zigzag:zigzag_e2 ...................................... passed (0.000 sec)
59: % [43/54] zigzag:zigzag_e3 ...................................... passed (0.000 sec)
59: % [44/54] zigzag:zigzag_e4 ...................................... passed (0.000 sec)
59: % [45/54] zigzag:zigzag_e4 ...................................... passed (0.000 sec)
59: % End unit zigzag: passed (0.001 sec CPU)
59: % Start unit: coerce
59: % [46/54] coerce:coerce ......................................... passed (0.000 sec)
59: % [47/54] coerce:coerce_e1 ...................................... passed (0.000 sec)
59: % [48/54] coerce:coerce_e2 ...................................... passed (0.000 sec)
59: % [49/54] coerce:coerce_e3 ...................................... passed (0.000 sec)
59: % [50/54] coerce:coerce_e4 ...................................... passed (0.000 sec)
59: % [51/54] coerce:coerce_e5 ...................................... passed (0.000 sec)
59: % [52/54] coerce:coerce_e6 ...................................... passed (0.000 sec)
59: % [53/54] coerce:coerce_e7 ...................................... passed (0.000 sec)
59: % [54/54] coerce:coerce_e8 ...................................... passed (0.000 sec)
59: % End unit coerce: passed (0.002 sec CPU)
59: % All 54 tests passed in 0.067 seconds (0.067 cpu)
1/4 Test #59: protobufs:protobufs ..............   Passed    0.22 sec
test 60
    Start 60: protobufs:bootstrap

60: Test command: /usr/bin/make "SWIPL=/home/peter/src/swipl-devel/build/src/swipl" "-C" "/home/peter/src/swipl-devel/packages/protobufs/bootstrap" "clean" "test"
60: Working Directory: /home/peter/src/swipl-devel/build/packages/protobufs
60: Test timeout computed to be: 10000000
60: make: Entering directory '/home/peter/src/swipl-devel/packages/protobufs/bootstrap'
60: rm -f -r foo *.tmp *.o *.pb.cc *.pb.h *_pb2.py *_pb.pl *_pb0.pl doc/ ../TAGS \
60: 	gen_pb/google/protobuf/*.proto.parse \
60: 	gen_pb/google/protobuf/*.proto.segment \
60: 	gen_pb/google/protobuf/*.proto.wire \
60: 	gen_pb/google/protobuf/*.proto.wiredump \
60: 	gen_pb/google/protobuf/*.proto.wirerawdump \
60: 	gen_pb/google/protobuf/compiler/*_pb2.py \
60: 	gen_pb/google/protobuf/compiler/*.proto.parse \
60: 	gen_pb/google/protobuf/compiler/*.proto.segment \
60: 	gen_pb/google/protobuf/compiler/*.proto.wire \
60: 	gen_pb/google/protobuf/compiler/*.proto.wiredump \
60: 	gen_pb/google/protobuf/compiler/*.proto.wirerawdump \
60: 	gen_pb/google/protobuf/compiler/*_pb2.py \
60: 	__pycache__
60: git clean -ndxf  # Should find nothing.
60: /usr/bin/protoc -I/usr/include --include_imports --descriptor_set_out=gen_pb/google/protobuf/descriptor.proto.wire \
60: 	google/protobuf/descriptor.proto
60: /home/peter/src/swipl-devel/build/src/swipl -g "test_segment_messages('gen_pb/google/protobuf/descriptor.proto.wire')" -g halt tests.pl
60: Warning: Auto-loading assertion/1 from library(debug) into /home/peter/src/swipl-devel/packages/protobufs/bootstrap/tests.pl is deprecated due to term- or goal-expansion
60: test_segment_messages succeeded with 24 segment(s) in the message.
60: /home/peter/src/swipl-devel/build/src/swipl descriptor_proto.pl <gen_pb/google/protobuf/descriptor.proto.wire >gen_pb/google/protobuf/descriptor.proto.segment
60: 
60: Warning: Auto-loading re_replace/4 from library(pcre) into /home/peter/src/swipl-devel/packages/protobufs/bootstrap/descriptor_proto.pl is deprecated due to term- or goal-expansion
60: make: Leaving directory '/home/peter/src/swipl-devel/packages/protobufs/bootstrap'
2/4 Test #60: protobufs:bootstrap ..............   Passed    0.47 sec
test 61
    Start 61: protobufs:interop

61: Test command: /usr/bin/make "SWIPL=/home/peter/src/swipl-devel/build/src/swipl" "-C" "/home/peter/src/swipl-devel/packages/protobufs/interop" "clean" "test" "run_addressbook"
61: Working Directory: /home/peter/src/swipl-devel/build/packages/protobufs
61: Test timeout computed to be: 10000000
61: make: Entering directory '/home/peter/src/swipl-devel/packages/protobufs/interop'
61: rm -f -r *_pb2.py *.wire *.o *_pb.pl *.pb.h *.pb.cc test_write test_read __pycache__ \
61: 	google/protobuf/*_pb.pl google/protobuf/*_pb2.py addressbook.wire
61: git clean -ndxf  # Should find nothing.
61: /usr/bin/protoc -Igoogle/protobuf -I. -I../demo -I../interop -I/usr/include --python_out=. --cpp_out=. test.proto
61: g++ -O0   -c -o test.pb.o test.pb.cc
61: g++ -O0   -c -o test_write.o test_write.cc
61: g++ -o test_write test.pb.o test_write.o -O0  -lprotobuf 
61: rm -f scalars1a_from_cc.wire scalars1b_from_cc.wire
61: ./test_write
61: /usr/bin/protoc -Igoogle/protobuf -I. -I../demo -I../interop -I/usr/include --python_out=. --cpp_out=. test2.proto
61: rm -f scalars1a_from_python.wire scalars1b_from_python.wire \
61: 	oneof1_from_python.wire map1_from_python.wire \
61: 	repeated1a_from_python.wire \
61: 	packed1a_from_python.wire
61: /usr/bin/python3 test_write.py
61: PATH=/home/peter/src/swipl-devel/packages/protobufs/bootstrap::$PATH /usr/bin/protoc -I. -I../demo -I../interop -I/usr/include --swipl_out='OUT_PARAMETER':. google/protobuf/unittest.proto
61: PATH=/home/peter/src/swipl-devel/packages/protobufs/bootstrap::$PATH /usr/bin/protoc -I. -I../demo -I../interop -I/usr/include --swipl_out='OUT_PARAMETER':. google/protobuf/unittest_import.proto
61: PATH=/home/peter/src/swipl-devel/packages/protobufs/bootstrap::$PATH /usr/bin/protoc -I. -I../demo -I../interop -I/usr/include --swipl_out='OUT_PARAMETER':. google/protobuf/unittest_import_public.proto
61: PATH=/home/peter/src/swipl-devel/packages/protobufs/bootstrap::$PATH /usr/bin/protoc -I. -I../demo -I../interop -I/usr/include --swipl_out='OUT_PARAMETER':. test.proto
61: PATH=/home/peter/src/swipl-devel/packages/protobufs/bootstrap::$PATH /usr/bin/protoc -I. -I../demo -I../interop -I/usr/include --swipl_out='OUT_PARAMETER':. test2.proto
61: PATH=/home/peter/src/swipl-devel/packages/protobufs/bootstrap::$PATH /usr/bin/protoc -I. -I../demo -I../interop -I/usr/include --swipl_out='OUT_PARAMETER':. test2b.proto
61: LANG=C /home/peter/src/swipl-devel/build/src/swipl test_interop.pl
61: % Start unit: scalar
61: % [1/13] scalar:scalars1a_template .............................. passed (0.040 sec)
61: % [2/13] scalar:scalars1b_template .............................. passed (0.001 sec)
61: % [3/13] scalar:scalars1a_parse ................................. passed (0.007 sec)
61: % [4/13] scalar:scalars1b_parse ................................. passed (0.001 sec)
61: % [5/13] scalar:scalars1c_parse ................................. passed (0.001 sec)
61: % [6/13] scalar:string_atom ..................................... passed (0.000 sec)
61: % End unit scalar: passed (0.056 sec CPU)
61: % Start unit: repeated
61: % [7/13] repeated:repeated1a_template ........................... passed (0.002 sec)
61: % [8/13] repeated:repeated1a_parse .............................. passed (0.003 sec)
61: % [9/13] repeated:packed1a_template ............................. passed (0.001 sec)
61: % [10/13] repeated:packed1a_parse ............................... passed (0.003 sec)
61: % End unit repeated: passed (0.010 sec CPU)
61: % Start unit: golden
61: % [11/13] golden:golden_2_5_0_parse ............................. passed (0.004 sec)
61: % End unit golden: passed (0.005 sec CPU)
61: % Start unit: oneof
61: % [12/13] oneof:oneof ........................................... passed (0.000 sec)
61: % End unit oneof: passed (0.001 sec CPU)
61: % Start unit: map
61: % [13/13] map:map ............................................... passed (0.000 sec)
61: % End unit map: passed (0.001 sec CPU)
61: % All 13 tests passed in 0.082 seconds (0.082 cpu)
61: diff scalars1a_from_prolog_template.wire scalars1a_from_cc.wire
61: diff scalars1a_from_prolog_template.wire scalars1a_from_python.wire
61: diff scalars1a_from_python.wire scalars1a_from_cc.wire
61: diff scalars1b_from_prolog_template.wire scalars1b_from_cc.wire
61: diff scalars1b_from_prolog_template.wire scalars1b_from_python.wire
61: diff scalars1b_from_python.wire scalars1b_from_cc.wire
61: diff repeated1a_from_prolog_template.wire repeated1a_from_python.wire
61: diff packed1a_from_prolog_template.wire packed1a_from_python.wire
61: g++ -O0   -c -o test_read.o test_read.cc
61: g++ -o test_read test.pb.o test_read.o -O0  -lprotobuf 
61: ./test_read
61: /usr/bin/python3 test_read.py
61: .....
61: ----------------------------------------------------------------------
61: Ran 5 tests in 0.001s
61: 
61: OK
61: PATH=/home/peter/src/swipl-devel/packages/protobufs/bootstrap::$PATH /usr/bin/protoc -I. -I../demo -I../interop -I/usr/include --swipl_out='OUT_PARAMETER':. addressbook.proto
61: PATH=/home/peter/src/swipl-devel/packages/protobufs/bootstrap::$PATH /usr/bin/protoc -I. -I../demo -I../interop -I/usr/include --swipl_out='OUT_PARAMETER':. addressbook2.proto
61: PATH=/home/peter/src/swipl-devel/packages/protobufs/bootstrap::$PATH /usr/bin/protoc -I. -I../demo -I../interop -I/usr/include --swipl_out='OUT_PARAMETER':. \
61: 	google/protobuf/timestamp.proto
61: rm -f addressbook.wire
61: /home/peter/src/swipl-devel/build/src/swipl -g test_write -g test_write -g test_read -t halt addressbook.pl
61: Warning: Auto-loading assertion/1 from library(debug) into /home/peter/src/swipl-devel/packages/protobufs/interop/addressbook.pl is deprecated due to term- or goal-expansion
61: /usr/bin/protoc --decode=tutorial.AddressBook addressbook.proto <addressbook.wire
61: people {
61:   name: "John Doe"
61:   id: 1234
61:   email: "jdoe@example.com"
61:   phones {
61:     number: "555-4321"
61:     type: HOME
61:   }
61:   timestamps {
61:     last_updated {
61:       seconds: 1721619826
61:       nanos: 957531929
61:     }
61:   }
61: }
61: people {
61:   name: "Satan"
61:   id: 666
61:   email: "satan@fb.com"
61:   phones {
61:     number: "555-1212"
61:     type: WORK
61:   }
61:   phones {
61:     number: "555-1234"
61:     type: HOME
61:   }
61:   timestamps {
61:     last_updated {
61:       seconds: 1721619826
61:       nanos: 963577509
61:     }
61:   }
61: }
61: people {
61:   name: "Crowley"
61:   id: 999
61:   timestamps {
61:     last_updated {
61:       seconds: 1721619826
61:       nanos: 964312077
61:     }
61:   }
61: }
61: make: Leaving directory '/home/peter/src/swipl-devel/packages/protobufs/interop'
3/4 Test #61: protobufs:interop ................   Passed    8.58 sec
test 62
    Start 62: protobufs:demo

62: Test command: /usr/bin/make "SWIPL=/home/peter/src/swipl-devel/build/src/swipl" "-C" "/home/peter/src/swipl-devel/packages/protobufs/demo" "clean" "test"
62: Working Directory: /home/peter/src/swipl-devel/build/packages/protobufs
62: Test timeout computed to be: 10000000
62: make: Entering directory '/home/peter/src/swipl-devel/packages/protobufs/demo'
62: rm -f -r foo *.tmp *.o *.pb.cc *.pb.h *_pb2.py
62: git clean -ndxf  # Should find nothing.
62: /usr/bin/protoc -I. -I../demo -I../interop -I/usr/include --cpp_out=. pb_vector.proto
62: g++ -O0   -c -o pb_vector.pb.o pb_vector.pb.cc
62: g++ -O0   -c -o foo.o foo.cpp
62: foo.cpp: In function ‘int main(int, char**)’:
62: foo.cpp:303:22: warning: ISO C++17 does not allow ‘register’ storage class specifier [-Wregister]
62:   303 |         register int i;
62:       |                      ^
62: g++ -o foo pb_vector.pb.o foo.o -O0  -lprotobuf 
62: /home/peter/src/swipl-devel/build/src/swipl -s vector_demo.pl -g make_tmp99 --
62: ./foo <tmp99.tmp
62: -2.2212
62: -7.6675
62: 3.14159
62: 0
62: 1.77e-09
62: 2.54e+222
62: /usr/bin/protoc --decode_raw <tmp99.tmp
62: 2: 0xc001c504816f0069
62: 2: 0xc01eab851eb851ec
62: 2: 0x400921fb54442d18
62: 2: 0x0000000000000000
62: 2: 0x3e1e688ad5c9dd17
62: 2: 0x6e1c1b6e8e1bbe1e
62: /usr/bin/protoc --decode=Vector pb_vector.proto <tmp99.tmp
62: double_values: -2.2212
62: double_values: -7.6675
62: double_values: 3.1415926535897931
62: double_values: 0
62: double_values: 1.77e-09
62: double_values: 2.54e+222
62: /home/peter/src/swipl-devel/build/src/swipl -g test_basic_usage -g halt vector_demo.pl
62: [ 'X'=666,
62:   'X2'=666,
62:   'Y'=123,
62:   'Y2'=123,
62:   'Command2'=command,
62:   'Op2'=add,
62:   'Extra2' = A,
62:   'Proto'=protobuf([atom(1,command),atom(2,add),integer(3,666),integer(4,123)]),
62:   'WireCodes'=[10,7,99,111,109,109,97,110,100,18,3,97,100,100,24,180,10,32,246,1],
62:   'Segments-raw'=[string(1,"command"),string(2,"add"),varint(3,1332),varint(4,246)],
62:   'CommandCode'="command",
62:   'OpCode'="add",
62:   'Xseg'=666,
62:   'Yseg'=123
62: ], % with constraints
62:     freeze(A,vector_demo:must_be(atom,A))
62: /home/peter/src/swipl-devel/build/src/swipl -g test_send_command -g halt vector_demo.pl
62: [ varint(1,1),
62:   message(2,
62: 	  [ fixed64(2,4607182418800017408),
62: 	    fixed64(2,4626885667169763328),
62: 	    fixed64(2,4613937818241073152),
62: 	    fixed64(2,4616189618054758400)
62: 	  ])
62: ]
62: /home/peter/src/swipl-devel/build/src/swipl -g test_send_precompiled_command -g halt vector_demo.pl
62: [ string(1,"command"),
62:   varint(2,1),
62:   message(3,
62: 	  [ fixed64(2,4607182418800017408),
62: 	    fixed64(2,4626885667169763328),
62: 	    fixed64(2,4613937818241073152),
62: 	    fixed64(2,4616189618054758400)
62: 	  ])
62: ]
62: make: Leaving directory '/home/peter/src/swipl-devel/packages/protobufs/demo'
4/4 Test #62: protobufs:demo ...................   Passed    4.01 sec

The following tests passed:
	protobufs:protobufs
	protobufs:bootstrap
	protobufs:interop
	protobufs:demo

100% tests passed, 0 tests failed out of 4

Total Test time (real) =  13.29 sec

Compilation finished at Sun Jul 21 20:43:50, duration 13.3 s

[jan]

@maren

gmp or libBF

Mostly correct. I think you can also set -DUSE_LIBBF=OFF, which will cause the system to drop big integer and rational number support. I don’t know whether that still works. Stability of these options is always a bit questionable. For now, I think it is pretty stable. That would change if we could leverage the advantages of LibBF and make it favourable as preferred implementation. I don’t see that happen anytime soon.

@maren

readline or libedit

By default both are built if the dependencies are found. By default it loads libedit. You can set a Prolog flag (readline?) in init.pl to set the default to readline. The libedit version is a bit less powerful editor, but it can be programmed in Prolog and got some nice features from there, it avoids GPL dependency and it can deal with multiple consoles (see libssh pack to allow for ssh login, each with line editing when using libedit).

@maren

qt

Qt creates swipl-win, a gui application by @CapelliC. That is used as the “app” for MacOS and the Linux snap/flatpack packages. It has no relation to xpce.

@maren

engines

The engines option is experimental and provides engines when there are no threads. This opens the route to support async predicates for the WASM version. That part is not implemented though.

@maren

xpce dependencies, xinerama

Long ago. AFAIK, the xinerama extension allows xpce to figure out the multi-display setup. That is needed to ensure new windows and popup menus appear on the current display. Without, xpce will think there is one display that is the union of all displays and thus it may place windows in the “void”.

Otherwise I think you need fontconfig as well to get proper Unicode support in font rendering.

[maren]

Thanks for your answers Jan!

@jan

By default it loads libedit. You can set a Prolog flag (readline?) in init.pl to set the default to readline.

Given that SWI-Prolog defaults to libedit if it is there, is there a point to including readline still? Is there a reason to have it enabled, besides its function in the toplevel (which is taken over by libedit)?

[jan]

@maren

Given that SWI-Prolog defaults to libedit if it is there, is there a point to including readline still? Is there a reason to have it enabled, besides its function in the toplevel (which is taken over by libedit)?

Some people like readline better The editor is much more powerful. The disadvantages are the GPL licence and the ugly API using a lot of global variables. Instead, libedit allows for creating an object and setting properties for it. But yes, only a few percent of the users would probably notice anything if we dropped libreadline.

[maren]

Some more questions

parms.h embedding compiler paths

cmake generates a file called parms.h, among other things embedding a path to the c and c++ compiler. This is then used to set the prolog flags c_cc and c_cxx`, and I presume from there on used in things like the pack build system to figure out a preferred compiler.

I imagine on many systems, there is no actual guarantee that this compiler is still there at runtime. For example when installing with debian, I don’t think you’re forced to install the compiler along with it.
Nix however has checks after building a package to see if anywhere in that package, a path to any of the dependencies is mentioned. Since that is the case here, it considers the gcc compiler a dependency.
That is fine in a system where you use swi-prolog in your dev environment, installing packs etc. It is less fun when you’re trying to embed a small swi-prolog in a container or something.

I’ve been experimenting with just patching parms.h after config to not have a full path, but just say gcc and g++, which seems compatible with its further use in build/tools.pl. This drops the dependency from my Nix closure.
Besides forgetting what compiler was used for the swipl compile, are there any bad consequences to doing this?

tcmalloc compile options

tcmalloc is another interesting case on Nix. It comes with the gperftools package, which besides tcmalloc itself contains, as the name suggests, a bunch of perf tools, making up a pretty hefty package, in which tcmalloc itself is minuscule.
Am I right in assuming that we really only need tcmalloc here, and not any of the other gperftool things? As in, if I just created a secondary package that just contained tcmalloc.so, this would work fine for swi-prolog, right?
There might also be an option to statically compile tcmalloc when the right build flag is enabled. Would that be viable?

[jan]

@maren

I’ve been experimenting with just patching parms.h after config to not have a full path, but just say gcc and g++

I think that is fine. Can you file a PR to make CMake emit the basename instead of the full path?

@maren

As in, if I just created a secondary package that just contained tcmalloc.so, this would work fine for swi-prolog, right?

The debian build depends on libgoogle-perftools-dev, which contains the library, but not the tools. Yes, all we need is tcmalloc.so. I thought we also need some headers, but a quick checks says this is not the case (but possibly I overlooked something, please test that the tcmalloc extension predicates still exist).

I think you need tcmalloc for TerminusDB

[maren]

@jan

I think that is fine. Can you file a PR to make CMake emit the basename instead of the full path?

As a build option? Or do you think this should be the behavior regardless?
I don’t think that fully embedding the exact path to compilers is necessarily bad, not even if this incurs a runtime dependency in nix, since swipl actually needs to compile at runtime with some packs. Isn’t it just legitimately a dependency?

@jan

I think you need tcmalloc for TerminusDB

Because this works out to better performance? Or because it is required cause otherwise things break?
I think I saw you mention that it works well with the HTTP server.

[jan]

@maren

As a build option? Or do you think this should be the behavior regardless?

Good question. I was thinking unconditionally. Note that compiler is the default for pack_install/2, not for compiling the bundled extensions. You have a point. In the unlikely case it does matter though. you can set c_cc and c_cxx in your init.pl to an absolute path.

@maren

Because this works out to better performance? Or because it is required cause otherwise things break?
I think I saw you mention that it works well with the HTTP server.

Isn’t TerminusDB using the HTTP server? The default ptmalloc from glibc seems to suffer from failure to reuse memory under some workloads. I spent a lot of time on it, but never really understood why this is the case. I think it is related to the fact that when using volatile atoms and dynamic predicates, allocation is done by all threads, but only the gc thread releases memory. From what I read from the ptmalloc design this should be ok though. Anyway, if you do not suffer from this, there is no problem. Performance wise it may or may not make a difference. I think tcmalloc is typically better in avoiding false sharing (handing two fragments to two threads that share a cache line).

[maren]

@jan

Isn’t TerminusDB using the HTTP server

It sure is, but I can’t say I ever extensively measured the impact different allocators make

tcmalloc is the swi-prolog default so the default packaging of swi-prolog should have it, and I see no reason to really divert from that.

[peter.ludemann]

@jan

PR to make CMake emit the basename instead of the full path

I have mixed feelings about this … but if it breaks your “nix”, I suppose it’s OK. There isn’t a way you can emit both? (I’d use them in a Makefile with defaults: use the full path if it exists, then the basename if it resolves, else $CXX)
IIRC (at least in the distant past), the C preprocessor was required on Unix systems, so typically there’s at least a C compiler in /usr/bin.

For a few packages, I use $SWIPL_CXX … GNU Make, of course, defines $CXX and other variables, and these seem to not have a full path, so if $SWIPL_CXX needs to be just g++-12 (for example), that should be fine. I assume CMake does something different (so far, I haven’t dealt with paths that use Cmake)

@maren

tcmalloc is another interesting case on Nix

On my (Debian) system, swipl uses /lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4, which doesn’t seem to be in any package. So, perhaps libgoogle-perftools-dev isn’t needed?

(tcmalloc is designed to improve performance with heavily mutli-threaded code, but it doesn’t play well with gdb IIRC)

[maren]

@peter.ludemann

but if it breaks your “nix”, I suppose it’s OK.

There are multiple solutions to problems. If this breaks something for you or anyone, this could just be done as a build flag instead. or a post-configure patch purely in the Nix packaging.

@peter.ludemann

IIRC (at least in the distant past), the C preprocessor was required on Unix systems, so typically there’s at least a C compiler in /usr/bin.

parms.h contributes to build_tools:preferred_c_compiler/1, but if it doesn’t yield a viable compiler, it would already fall back to either gcc or clang (or the c++ equivalents), as in just those binary names, not a full path to them. Notably, for Apple, these parameters were never full paths.

Note that outside of the case of building your own SWI-Prolog, there would be absolutely no guarantee that the compiler whose path gets baked into SWI-Prolog would be available wherever you install. Except on Nix, because it considers such hardcoded paths to signify runtime dependencies.

The parms are used to fill the prolog_flag c_cxx and c_cc. These can be freely set to anything at runtime, so if there is in fact a default that should be used on a system, perhaps it can be detected on install and written to a configuration file.

@peter.ludemann

There isn’t a way you can emit both? (I’d use them in a Makefile with defaults: use the full path if it exists, then the basename if it resolves, else $CXX)

Look in library/build/tools.pl and see if the logic for finding a compiler there matches what you want. I think it roughly does already.

@peter.ludemann

On my (Debian) system, swipl uses /lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4, which doesn’t seem to be in any package. So, perhaps libgoogle-perftools-dev isn’t needed?

Your libtcmalloc_minimal.so.4 did not appear out of thin air. You can do apt-file search to find out what package dropped it there.
(Edit: I just checked this on Debian Sid, and according to apt-rdepends libtcmalloc-minimal4t64 is a dependency of swi-prolog-core)
This is a Nix packaging issue, and I might see if I can get something done about it.
There’s no need to include libtcmalloc.so and friends in the same package as all the performance tools that ship in the same source package, and debian doesn’t do it like that. Neither should Nix, but I guess nobody bothered properly splitting them.

PR: generate parms.h with c_cc and c_cxx options set to binary name only by matko · Pull Request #1301 · SWI-Prolog/swipl-devel · GitHub

[peter.ludemann]

I take your point about g++ (or g++-12) vs /usr/lib/g++ (or /usr/lib/g++-12). It probably doesn’t even matter if a pack is built with the same compiler as swipl (e.g., clang vs gcc), although there’s always the possibility of some incompatibility.

As for libtcmalloc, dpkg -S doesn’t know where it is but:

$ ls -l /lib/x86_64-linux-gnu/libtcmalloc_minimal.*
-rw-r--r-- 1 root root 329674 Jun  1  2022 /lib/x86_64-linux-gnu/libtcmalloc_minimal.a
lrwxrwxrwx 1 root root     29 Jun  1  2022 /lib/x86_64-linux-gnu/libtcmalloc_minimal.so -> libtcmalloc_minimal.so.4.5.10
lrwxrwxrwx 1 root root     29 Jun  1  2022 /lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4 -> libtcmalloc_minimal.so.4.5.10
-rw-r--r-- 1 root root 168624 Jun  1  2022 /lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4.5.10
$ for i in /lib/x86_64-linux-gnu/libtcmalloc_minimal.*; do apt-file search $i; done
libgoogle-perftools-dev: /usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.a
libgoogle-perftools-dev: /usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so
libtcmalloc-minimal4: /usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4
libtcmalloc-minimal4: /usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4.5.10
libtcmalloc-minimal4: /usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4
libtcmalloc-minimal4: /usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4.5.10
libtcmalloc-minimal4: /usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4.5.10

$ apt-cache rdepends libtcmalloc-minimal4
libtcmalloc-minimal4
Reverse Depends:
  libgoogle-perftools-dev
  yrmcds
  swi-prolog-core
  suricata
  scram-gui
  scram
  xmpsolve
  mpsolve
  mandelbrot-solver
  libmps3
  libjxl-tools
  libjxl-devtools
  gpsshogi
  libgoogle-perftools4

[maren]

weird that rdepends doesn’t list it.

# apt depends swi-prolog-core     
swi-prolog-core
  Depends: libc6 (>= 2.34)
  Depends: libgmp10 (>= 2:6.3.0+dfsg)
  Depends: libtcmalloc-minimal4t64 (>= 2.15)
... omitting the rest...

[edit] or maybe I should read more carefully swi-prolog-core is in your listing.

[jan]

As far as the C compiler for packs is concerned, it really should not matter. Except for Windows, C compilers typically share the same ABI. As our C++ interface is header-only, it doesn’t matter either. In fact, if C++ is used to connect to a C++ library, we should use the same C++ compiler for interface as for the C++ library, not the same as used to compile Prolog.

On most platforms the C/C++ compilers are at stable locations on PATH. Again, mostly Windows is the exception. Extensions on Windows can be build using MinGW or MSVC, but the compiler flags differ a lot. If you want the leave the choice to the user you better use a CMake specification.

For short, I’m not entirely sure we should set c_cc and c_cxx at all. For example, the Windows binary now says

 1 ?- current_prolog_flag(c_cc, X).
 X = '/usr/bin/x86_64-w64-mingw32-gcc'.

Which is highly dubious. It is the cross compiler used for building SWI-Prolog for Windows. It should probably be set to either gcc.exe (MinGW) or cl.exe (MSVC). Also for the MacOS (stable) release on Intel, I compile using gcc-mp (the real gcc, not the clang-named-gcc) due to the much better performance, but users probably wish to use Xcode clang as that is the default C compiler.

What shall we do? I think that for a local build, specifying the compiler used as c_cc and c_cxx makes sense. Possibly also as absolute path. For a release build I think this makes no sense and we should

• Set a reasonable default for the platform. That means cc and c++ for most platforms and probably the default MinGW for Windows. Note that for using the MSVC commandline tools, you need to run some script that sets a lot of environment variables anyway. We should react on that.
• Listen to the CC and CXX environment variables in pack_install/2. I think we should do that always, also for local builds.

That indeed asks for an option. What about -DSWIPL_CC=cc -DSWIPL_CXX=c++ ?

[maren]

@jan

What about -DSWIPL_CC=cc -DSWIPL_CXX=c++

I’m happy with that. And their absence would mean the options would simply not be set?

More philosophically, do we expect the ‘normal’ SWI-Prolog install to have access to a compiler? Certainly it expects that for installing packs with native components, but not each pack has those, and not each pack that does have those will build cleanly anyway due to other missing dependencies.

As I said, I can make sure in the Nix packaging that a compiler is pulled in as a runtime dependency, in which case it’d make sense to use those options. Or leave them out to drop the runtime dependency.
But what should be the default?

[jan]

@maren

I’m happy with that. And their absence would mean the options would simply not be set?

Their absense would mean they are set as they are, to the absolute path of the compilers used for SWI-Prolog itself. This would imply that packagers need to make a choice for the default C/C++ compiler for the platform.

@maren

More philosophically, do we expect the ‘normal’ SWI-Prolog install to have access to a compiler? Certainly

I don’t think so. Python does also not install the dependencies for using pip for foreign packages, no?

@maren

But what should be the default?

Leave them out (I think)

[jan]

I will implement that. Better suggestions are still welcome