We have qcompile/1 (and swipl qlf compile), which takes a prolog file and turns it into qlf file for quick loading.
Looking at the SWI-Prolog build output, as well as what is present in the docker, save for 3 exceptions, none of the prolog files that ship with SWI-Prolog are actually compiled though. They’re all still plain .pl files, so presumably loading them will take longer than it has to.
Similarly, installing a pack will not qcompile automatically. This might be more reasonable, since a user’s pack dir might have to support several different swipl versions. But for the libraries that ship with the SWI-Prolog install, we know exactly what version they’re going to be used with.
Is there a reason why these libraries are not precompiled? I imagine the build could easily do that as an extra step. Are there potential incompatibilities when doing so?
As a bonus question, what actually are the compatibility constraints around qlf files and different versions of SWI-Prolog? The documentation makes it clear that this format is not intended to be backwards compatible. Does it change often? And is swipl capable of detecting incompatible qlf files and handle them gracefully?
It is surely doable. I don’t know how useful it is. The price are a lot more files and the inability to switch easily between optimized and development mode. Also development on the system gets a little more complicated, but that is easily resolved using a CMake option. I don’t know whether it is worth it. The only way to find out would be to try it and see whether it makes a significant difference in the start/load time of real applications. I have my doubt …
They record the byte codes. So, incompatible changes to the VM break compatibility. Then there are (infrequent) changes to deal with issues found in the format or to improve the format. Recent versions notably deal with VM code differences between platforms, so we can qlf compile on one system and use the file on another system as long as it is the same Prolog version.
Typically not, but there are times that changes are fairly frequent.
Normally, yes. The format has two version ids. One maintained by hand to annotate changes to the format. This also allows the loader to dynamically adjust and be able to read the older version. The second is a hash computed from the VM signature. The VM signature is defined to be the set of VM instructions and their argument (types). If this does not match it refuses to load the state/qlf. This is the most frequent reason for breaking, not always correctly. I.e., adding an instruction allows using old .qlf files (but not new qlf files on older Prolog versions). As is, they are simply considered incompatible.
You mean, why would I want to qcompile?
I kinda just assumed that the qcompiled version of a source file would automatically be ‘better’, provided it produces something compatible, Which didn’t line up with it almost being unused within SWI-Prolog itself.
If you’re not expecting a significant difference in load time for the majority of the included libraries, I have to wonder what the use for qlf files is. Does it only start being meaningful for certain sizes of input files? Like big fact databases?
In the good old days the time to find and open the library file often had more impact than loading it. The main impact for using .qlf was that they can contain multiple files, so the need for file system interaction is reduced. That notably applies for library(pce) (the GUI). These days file-system interaction is typically a lot faster and I do not know whether or not that is still the case.
I did a little test that suggests this may actually have significant impact. Tested on current development version running on Fedora 40 with AMD3950X, 128Gb mem and a fast ssd.
Loading library(lists)
From source: 6ms, from .qlf < 1ms.
Running time swipl qlf --help
From source: 57ms, with library in .qlf 34ms
I simply compiled the library by running this in the library dir
swipl qlf compile [a-z]*.pl
So, it looks like something worthwhile considering Probably should add a flag/option to force loading the source?
Very interesting to see that it does matter! In that case I would very much like to be able to compile my swi-prolog with everything as qlf. Is it really as simple as finding every .pl file and compiling it? I feel like there’s probably some caveats.
When do you want to force load from source rather than the qlf?
There’s a trick that does compilation of .pl files on first use …
If you create an empty foo.qlf, loading foo will trigger compilation of foo.pl (with a message “recompiling QLF file (incompatible with current Prolog version)”). But this trick wouldn’t work if SWI-Prolog is installed in a read-only directory. (I also wonder whether this is a bug or a feature? I don’t think it’s documented)
Some numbers for a 482K line data file (596,060 clauses):
compile time: 20.91 sec
load time .pl file: 15.78 sec
load time .qlf file: 1.01 sec
size: .pl: 128MB, .qlf: 64MB (.pl.bz2: 7.1MB, .qlf.bz2: 7.5MB)
FWIW, if you build emacs, it compiles all the .el files to .elc, which seems to take about as much time as compiling the .c source. There’s an option to further compile .elc to native code, but I turned that off because it doesn’t seem to make much different on my machine and significantly increases the build time (I build emacs from head whenever I restart my machine).
That’s interesting. So it considers it an incompatible qlf and recompiles.
There is also the prolog_flag qcompile, which can be set to auto to compile any file loaded by swi-prolog if the dir is writable.
A 15 time speedup for a large file is pretty amazing.
.qlf files are automatically maintained if they are writeable. That is, they are recompiled if they are incompatible or are of date wrt. the .pl file. This unfortunately is not perfect. Loading a file may depend on expansion hooks, flag settings, etc. If these hooks are not loaded from the file itself or the flags are not set in the file, you may end up with something different then what you expected. Also, it is possible to derive a single .qlf files from multiple source files. For example:
I see two ways forward. One would be to extend the CMake scripts to compile the library. The other would be to design a Prolog format that provides enough information to drive the compilation process entirely in Prolog, so we can use swipl qlf ... to compile and/or update the .qlf files in a directory (tree). The advantage of the latter approach is that applications can use the same. Roughly, the file should specify
Which files to compile
Create the environment for compiling (hooks, flags)
If we could figure out some build instruction that worked for packs as well that would be great.
Regarding a qlf consisting of multiple files, if I understand the docs correctly there’s two ways that can happen:
We compile using swipl qlf compile --include. All imported modules that aren’t standard libraries are automatically included (not sure how this works for the standard libraries themselves)
The compiled file imports further files with explicit qlf-building instructions
Are there other reasons why this happens?
I suspect compiling all .pl files into their individual .qlf files with no extra settings works well enough for the majority of code. It would not work if some of these files were actually qlf-including some of their own inputs automatically, or qlf-compiling an input in a special environment.
So this is just a matter of making sure that (standard) library developers can specify if they are writing code that fits one of these conditions.
Does that sound right?
They are loaded from the system. The idea emerged from creating the WASM version. Saved states are a good way to package applications, but typically they are not platform independent due to conditional compilation on features of the running Prolog system. A qlf files holding only the application files is often a good way to create a single file that loads quickly and can be loaded on different platforms. Unlike for running state, you do need a full Prolog installation to run such a file.
I think not
I think it is close. It should be enough to specify files you do not want to compile into .qlf, either because they are support file for other files or some other reason. It is rare, but it is possible to create files that cannot be converted info .qlf, for example because the (generated) clauses contain “blobs” that cannot be serialized.
We could figure out files that include other files. The blob scenario can only be trapped while compiling.
Yeah, qlf really sounds amazing for shipping around bits of executable prolog when you can reasonably trust the environments you use them with to have the required dependencies. That’s not just about the standard library either, but goes for any set of common dependencies.
How is it detected that an import would be coming from a standard library? Is it just a check for which directory the source files are in? And how does that work in the build environment, before installing? Would they already be counted as standard libraries?
So just a file listing all prolog files to be excluded from automatic compilation, either cause it is does not work out for that particular file, or because compilation is already done through some other means. Easy
The pack definition could probably be expanded to allow for specifying excluded files. maybe even a pack-wide compilation preference, like turning it off altogether for a particular pack.
Maybe the bundled libraries could have a pack-like file too, but just for specifying those exclusions?
I think it simply checks that the path is a subdirectory of the SWI-Prolog home. Check the source if you want to be 100% sure.
The build process creates the home directory structure in home below the build directory, normally as a set of symlinks to the sources.
Otherwise, I have to think a little about the options. Another issue is handling include/1. You probably do not want to compile files that are supposed to be included. Another option could be to allow for a directive, structured comment or alternative extension that explains the status of the file.
You probably don’t, but if they ended up accidentally being built anyway, nothing breaks, right? It might not be so bad to require people add such exceptions to a manual exclusion list.
You probably always can. But include is the right tool if you actually just want to verbatim include a file with no surprises. use_module/2 implies you’re using a module, which the included file doesn’t have to be. load_files/2 is far more generic, but its genericness also means it does extra stuff, depending on the environment. For example, the automatic qcompile. include/1 is simple and predictable.
It’s also in ISO Prolog so it’s not going anywhere, just in case someone wants to load the code they wrote in the 90s
AFAIK, it was introduced by ISO Old Prolog systems had consult/1 and reconsult/1. Later ensure_loaded/1 was added (Quintus?), etc. Anyway, include is mostly useful to share declarations (e.g., multifile, dynamic, op, …), or load non-modular code. Note that you cannot use consult/… to load the same non-modular file into more than one place. IMO, modern code should almost always use modules. They come at very close to zero cost and provide good modularity and sharing.
Probably should add a flag/option to force loading the source?