I’m happy to announce SWI-Prolog 8.1.31. This version completes the
changes I considered necessary for 8.2. Highlights:
Fixed temporary string handling with Keri Harris. See below
(can impact foreign extensions).
Moved library(tables) from XSB dialect to the main library.
Changed autoload and library indexing. This notably allows
the dependency generation to find dependencies from non-autoload
directories.
PceEmacs: added C-cC-o to add a module header and C-cC-e to add
an export to the module header.
Make the non_terminal property available on all predicates
defined using -->
Fixed quite a few dependencies still relying on autoloading.
I think all should build fine again. Notably the changes to maintaining
the INDEX.pl files causes quite a few build/install issues.
Enjoy --- Jan
SWI-Prolog Changelog since version 8.1.30
FIXED: swipl -c file.pl without relying on autoloading.
PORT: MacOS (clang 11): disable support for gcc threaded bytecode
extension (VMCODE_IS_ADDRESS) as this slows down SWI-Prolog by
about 30% instead of improving the performance with about 3% (gcc-9).
MODIFIED: Move the tables library from library(dialect/xsb/tables)
to the main library as the primary interface to reason about created
tables.
MODIFIED: Memory management for temporary strings used internally
and through the foreign language interface functions such as
PL_get_nchars(). Such strings used to be managed (by default) in a ring of 16 strings, reusing old buffers. Recent versions used memory
mapping for long strings and unmapped these. This exposed a problem
that input strings could be incorrectly reused early. Together with
Keri Harris we designed and implemented a new stack based management
for such strings. Temporary strings are now reclaimed on exit (or
failure) of a foreign predicate that created them. In addition,
there is this C block ({…}) that may be used to scope strings:
PL_STRINGS_MARK();
<code>
PL_STRINGS_RELEASE();
A new flag string_stack_tripwire can be set to get a disagnostic
message
printed if the string stack reaches some depth limit. This may hint at
using the above macros to reduce resource usage.
In addition a number of PL_* functions that used to step the string
ring
now use the above macros to reduce resource usage.
INSTALL: Fixed installation of INDEX.pl files
TEST: Fixed for non_terminal property.
ENHANCED: set non_terminal property on any predicate that has at
least one clause written as Head --> Body.
ADDED: library(prolog_deps) to also look at non-autoload
libraries. Allows for e.g. get http_open/3 into the imports using
PceEmacs C-cC-d.
ADDED: INDEX.pl files to all directories that are useful for automatic
dependency generation.
MODIFIED: Emptied and deprecated library_directory/1. The definitions
have been moved to file_search_path/2. Modified the definition
of the autoload alias to explicitly specify - app_config(lib)).
pce(prolog/lib)). - swi(library)).
This prepares for indexing all Prolog library directories. This will
be used
to simplify managing what is autoloaded and for the IDE to add
dependecies
automatically.
ENHANCED: Try to reduce the probability of getting into a deadlock
while reporting a crash.
This is pretty dubious. I think it works around some old omission but it may well mess things up now.
Fixed these two. Bit odd that I’ve never seen them, also using Ubuntu 18.04 on some of the deployment machines. Anyway, both warnings are harmless.
This is rather odd. Building the package docs depends on the prolog_home target which builds the home directory in the binary dir. Just running ninja prolog_home on a freshly configured system does create a home with these files. I’ve build the system with a quite different number of cores (changes timing), both using ninja and make and ranging from fully sequential to 32 cores. Didn’t find any issues.
Also tried another common trick: remove one of the dependencies of a target and build the target. If the dependency isn’t properly declared this will fail. But this works fine:
I get the same error with the swi_hooks.pl on a freshly installed ubuntu 19.04
I was able to build after compiling without documentation, installing and then compiling again with documentation. Might be some sort of circular dependency?
Same problem in a clean clone and with packages installed as in the documentation. Log attached. (It’s a plain text file that I named “.pl” to make Discourse happy)
Please don’t. It is a waste of space. Nobody is going to read these and as we have a dependency problem and concurrent building, a log helps little. At best prove it isn’t your fault, but with confirmations from others it was clear this was not some weird local issue
Concurrent building is nice but hard. CMake does a lot of the hard work, but especially if the build process has to go through various stages that do not use the standard CMake tooks like the C compiler and linker, you need to generate a precise description of all dependencies. If you miss one it may still build fine because you are lucky with what the arbitrary ordering selected and/or resulting from concurrent execution.
I searched for tools to find these issues several times. I think it is possible: use tools such as strace to figure out the files opened in a build step to deduce the inputs and outputs of that step and check this matches the model built in CMake (which consists of graph of targets with associated dependencies, outputs and commands). If such a tool exists, please shout!
Anyway, the steps here
allowed me to find the missing dependency. I think this one is fixed.
I don’t have a tool for you but I can tell you how things are done with Bazel, which might give you an idea.
Bazel requires completely specifying all inputs and outputs, for what it calls “hermetic” builds. It uses these to build a dependency graph, which allows it to compute the maximum parallelism for the build. Steps are removed if the inputs haven’t changed and all the outputs exist and haven’t changed - hashes are used for this, not filestamps. [Bazel can also be used to specify the unit tests; their results are also cached and only rerun as needed]
Each build step starts in a clean environment. All input files are symlinked into this environment; the step is run (which can involve multiple serial steps); only the specified outputs are linked back to the build outputs and the environment is destroyed (which deletes any non-specified output files). If any dependency is missing, the problem is completely obvious (and reproducible).
So, perhaps cmake/ninja have an option for running each step in a clean environment and deleting all non-specified output files at each step? (Another option is switching to Bazel – but I suspect that’s more work than it’s worth.)
PS: I deleted the log that I posted.
PPS: Everything builds fine for me now also.
I guess that could indeed fix this issue with CMake/configure and make/ninja. CMake basically provides a script language that creates a graph of targets that is subsequently used to write a series of Makefiles or a Ninja specification file (and some more). Ninja has nice tools to visualize dependencies. Surely great for small projects, but for understanding SWI-Prolog’s build process it is useless. I see no real reason why CMake/ninja couldn’t do something. Its a big and active community, so who knows …
Anyway, I spent about 3 months converting from configure/make to CMake. That was certainly worth the effort as maintaining the project has become a lot smoother. Reading a couple of posts, a switch to Bazel is not obviously a move in the right direction. Well, older versions of CMake were also a mess and while modern CMake is fairly elegant it still carries most of the old stuff with it. Where have I seen that before
The only infrastructure move that was fairly smooth and I haven’t regretted for a second is from CVS to GIT
I am not aware of any tools for analyzing Bazel build file dependencies.(*) If it took you 3 months to convert to CMake, it wouldn’t surprise me that it would take another 3 months to convert to Bazel. If there’s a way to wipe the environment for each step in the build, that should go a long way towards figuring out missing dependencies.
(*) At Google, build files were originally simplified Python that was used to generate gigantic Makefiles – I remember seeing Makefiles with over 800,000 lines. ISTR that these were used with an early version of one of the parallel make tools; I don’t recall people having problems with missing dependencies but maybe I wasn’t paying attention. Bazel adopted these build files and was a huge useability and performance improvement (particularly because of its caching; but also because it allowed a continuous build/test environment).
You could do like compilers do to bootstrap: a little bootstrap prolog with enough in it to handle the buld management (probably just tree management and shell execution?) then use the bootstrap prolog to build the real one.
EDIT: the real problem (advantage of cmake) might be all the platform portability macros, C quirks, etc that cmake provides (e.g. android, etc). I doubt you would want to get into the business of all the portability quirks.
Anyway, Google didn’t have me to advise them when they chose to write their build files with Python (although the Python does little more than spit out more complex configuration files that are then processed by a system written in Java).
BTW, Google also had a package deployment system that used a pure functional language modeled on Modula-2 – people had so much trouble understanding how inheritance worked with it that a tool was written to interactively figure out actual values and explain how they were derived.
Indeed. Using Prolog for dependency management and describing configuration problems as rules and constraints is trivial. CMake is a pretty ugly script language, but in recent versions the modelling of building software is quite ok and there is an awful lot of knowledge hidden in its zillions of scripts dealing with compilers, OSes, build tools, (dynamic) libraries, packaging tools, test tools, etc. After all, that is the real core of the problem. In my experience the ugly script language is a pain, but one gets used to it. Getting the dependencies all explicit such that parallel building works reliably is hard once the project is not a simple C/C++ project and requires several stages. Bazel’s solution to this is an interesting thought, although I think there could be simpler solutions.
If there’s a way to wipe the environment for each step in the build, that should go a long way towards figuring out missing dependencies.
Unfortunately that doesn’t help for building Prolog …