Changing autoload

Request For Comments
1

This relates to Autoload and qsave (swipl -c) don't go together. I think I made a lot of progress addressing this issue. Here is the status and remaining problems:

Autoload handling

Prototype implementation (on git swipl-devel, master)

  • Added :- autoload(File) and :- autoload(File, Imports). Syntax
    and semantics are a subset of :- use_module, except no loading
    happens (see below for exceptions). The autoload/2 variant defines
    the specified predicates such that a local definition is prevented.
    Both directives add some data to the module that defines autoloading
    depending on the autoload flag. This flag has these values:

    • false
      Predicates are never auto-loaded. If predicates
      have been imported before using autoload/1,2, load the
      referenced files immediately using use_module/1,2. Subsequent autoload/1,2 is mapped directly to use_module/1,2 Note that most of the development utilities such as listing/1
      now have to be explictly imported before they can be used
      at the toplevel.

    • explicit
      Do not autoload from autoload libraries, but do
      use lazy loading for predicates imported using autoload/1,2. This
      is semantically the same as false, but preserves lazy loading.

    • user
      As false, but to autoload library predicates
      into the global user module. This makes the development
      tools and library implicitly available to the toplevel,
      but not to modules.

    • user_or_explicit
      Combines explicit with user,
      providing lazy loading of predicates imported using
      autoload/1,2 and implicit access to the whole library for
      the toplevel.

    • true
      Provide full autoloading everywhere. This is
      the default.

Autoloading has a preference: (1) autoload/2, (2) autoload/1 and (3) autoload from the library.

Status and results

Added autoload/2 declarations for all files in library that come from
the core system. This implies that all dependencies for these core
libraries are now explicit. Using my default personal init.pl which
enables <N> ?- history and XSB compatible loading of .P files, startup
time resource usage is reduced significantly:

Before (Intel® Core™ i5-6260U, M2 SSD, Fedora 30)

?- statistics.
% Started at Tue Feb 11 09:56:26 2020
% 0.206 seconds cpu time for 726,531 inferences
% 6,515 atoms, 4,580 functors, 3,615 predicates, 74 modules, 136,107 VM-codes

After

?- statistics.
% Started at Tue Feb 11 10:01:17 2020
% 0.086 seconds cpu time for 247,031 inferences
% 6,001 atoms, 4,154 functors, 2,925 predicates, 49 modules, 102,780 VM-codes

Modes

The default mode is true which, if all libraries have been updated to
use autoload/2 where possible, will remain fully compatible with the
current system while providing two benefits: (1) reduced load time and
memory usage and (2) a guarantee that all relevant code is loaded if
autoload is switched to false.

The mode false is probably useful for creating stand-alone programs
reliably.

The mode user_or_explicit is intended for developing programs that
eventually want to go for false for creating an executable. While
developing it does the loading lazily which reduces startup time and
in addition it allows you to run whatever you like in user space
without declarations. This includes tools (listing, gxref, gtrace,
edit, profile, statistics, …) and running stuff like this to test
stuff inside your modules without importing time/1, forall/2, etc.

?- time(forall(between(1, 1 000 000, _), mym:mygo)).

The mode user, as false, materializes all autoload/1,2 and, as
user_or_explicit ensures the toplevel remains friendly.

Portability and compatibility

Several people here have argued for controlled import using use_module/2
(or now autoload/2). The advantage is that you can easily see what comes
from where without using tools. Another advantage is that it is
completely obvious what needs to be loaded for creating an executable
and thus there is no need to rely on the code analysis to include all
dependencies. As we know, code analysis in Prolog is by definition
potentially incomplete. I have seen actual production code using atom_concat(X,Y,Name), call(Name, ...) :slight_smile:

There is also a price to pay. In the past predicates have often been
moved from built-in to the library or the other way around. Migrating to
the library reduces the size of the core while migration to a built-in
can use a low-level C implementation to gain performance. Migration
between libraries also happened, either for compatibility with other
Prolog systems or just to clean up the organization. Using autoload is
true, this doesn’t affect any code. If use_module/2 or autoload/2 is
used and autoloading is disabled, this does break applications.

Note that the fear that running code breaks as a result of using the
autoloader is not justified. Programs first use the predicates they
define themselves, unless these conflict with the ISO core. Anything
else comes from built-in or the autoloader, where the system development
should guarantee a name/arity available as built-in or autoloading is
(1) not ambiguous, (2) does not disappear and (3) does not change
semantics. The latter (changing semantics) happens occasionally for
compatibility or due to new insights.

A similar argument applies to porting e.g., SICStus code. Typically we
can simply remove most of the use_module/1,2 calls and let the auto
loader do its magic. With explicit loading you will probably end up with
a lot of :- if(...) or significant extension of the dialect
emulation libraries. For this reason SICStus and SWI long time ago
agreed on a :- require([PredicateIndicators])., meaning “Get these
things from somewhere” that was dynamically mapped to use_module/2 in
SICStus and ignored in SWI-Prolog as the autoloader would take care.
SWI-Prolog’s require will be changed to depend on the autoload flag.

Problems

The main remaining problem has to do with the role of the user module.
SWI-Prolog provides a module inheritance system where module X
inherits from user which inherits from system where all the
built-ins live (by default, the hierarchy can be changed). The
inheritance mechanism does:

  • Make predicates available. These can be overruled locally, except
    for those marked iso.
  • Share operators. Operator lookup follows the module inheritance
    relations. This allows defining operators in user for compatibility
    with many systems that have a single global operator space.
  • Inherit the unknown flag.
  • Create a pipeline for term_expansion/2, which is first applied in the
    local module, then in user and finally in system.

The module inheritance system for predicates interacts with unknown and
autoloading. This leads to weird behaviour if these are changed from their
defaults, such as

:- module(m, [m/1]).

m(X) :- member(X, `aap`).

Now, without autoloading, calling m/1 raises an exception. Now call
or import member/2 in user and suddenly m/1 runs fine. This is of
course unacceptable. I see some ways out:

  • Block inheritance of predicates from user using some flag. This
    probably has the least implications. It feels bad to change the
    general inheritance relation for a specific module though.

  • Move user to be cousins of the other modules rather than the
    parent. The problem is that this breaks the desirable relations
    for operators, unknown and term expansion

  • Use a new module for the default interactive toplevel. It is a
    bit unclear to me how much would be affected by this. If this
    new module inherits from user and non-module files are by
    default still loaded into user, possibly not that much? For
    the toplevel to load non-module files into user itself
    is quite a kluge though. Loading code into the new toplevel
    has some advantages, such that it starts out empty rather than
    containing the various hooks that the system inherrited from
    other Prolog implementations and its past.

Any opinions?

Plan

Regardless on how the above is resolved, there are plans for two
next steps:

  • Resolve term/goal expansion. That could impact the above as it
    may resolve one of the issues wrt. the user module.
  • Use autoload consistenty throughout all libraries and extensions
    such that the system becomes fully functional regardless of the
    autoload mode.
  • Complete and make the tool I now have to create the autoload
    directives available, probably both as command line tool and
    integrated into PceEmacs.
2 Likes
2

Good progress :slight_smile: I assume that there’s still some work to be done before we can start SWI-Prolog with auto-loading disabled? With 8.1.21-119-gb1013a613-DIRTY and an init.pl file containing only the directives:

:- set_prolog_flag(autoload, false).
:- set_prolog_flag(verbose_autoload, true).

I get:

$ swipl
% Disabled autoloading (loaded 0 files)
ERROR: /Users/pmoura/lib/swipl/library/time.pl:122:
ERROR:    catch/3: Unknown procedure: time:use_foreign_library/1
Warning: /Users/pmoura/lib/swipl/library/time.pl:122:
Warning:    Goal (directive) failed: time:use_foreign_library(foreign(time))
ERROR: /Users/pmoura/lib/swipl/library/ansi_term.pl:43:
ERROR:    Exported procedure time:install_alarm/1 is not defined
ERROR: /Users/pmoura/lib/swipl/library/ansi_term.pl:43:
ERROR:    Exported procedure time:alarm/4 is not defined
ERROR: /Users/pmoura/lib/swipl/library/ansi_term.pl:43:
ERROR:    Exported procedure time:install_alarm/2 is not defined
ERROR: /Users/pmoura/lib/swipl/library/ansi_term.pl:43:
ERROR:    Exported procedure time:alarm_at/4 is not defined
ERROR: /Users/pmoura/lib/swipl/library/ansi_term.pl:43:
ERROR:    Exported procedure time:uninstall_alarm/1 is not defined
ERROR: /Users/pmoura/lib/swipl/library/ansi_term.pl:43:
ERROR:    Exported procedure time:remove_alarm/1 is not defined
ERROR: /Users/pmoura/lib/swipl/library/ansi_term.pl:43:
ERROR:    Exported procedure time:alarm_at/3 is not defined
ERROR: /Users/pmoura/lib/swipl/library/ansi_term.pl:43:
ERROR:    Exported procedure time:alarm/3 is not defined
Welcome to SWI-Prolog (threaded, 64 bits, version 8.1.21-119-gb1013a613-DIRTY)
3

Yes. Notably all files from all packages should get complete coverage using use_module/1,2 and/or autoload/1,2 (planning on the latter). I wonder about use_foreign_library/1. It may also make sense to turn it into a built-in?

Also, the core library(ansi_term) should not depend on the package library(time) and gracefully degrade is this library is not present.

1 Like
4

Don’t expect much to work, but with the latest patches it does start with your init.pl. There is not yet that much point in testing and reporting issues beyond the basic libraries as I have a script that rewrites a file, although that occasionally requires some additional tweaking.

5

A post was split to a new topic: Term and goal expansion

6

Likely related with library(time):

!     call_with_timeout_2_01: failure 
!       test goal throws the wrong error:
!         expected timeout((repeat,fail))
!         but got  error(existence_error(procedure,time:alarm/4),context(system:'$c_call_prolog'/0,A))
!       in file /Users/pmoura/Documents/Logtalk/logtalk3/library/timeout/tests.lgt between lines 39-40
7

Forgot to push the latest stuff. Using 8.1.21-120-g4172cfe36, basic startup with autoloading disabled in init.pl works for me.

8

I can now start swipl (and swilgt) with auto-loading disabled without any issues :+1: Running the Logtalk tests, there are some errors, however (I know that this is work in progress; no criticism here). One of them is with JPL, more exactly, the jpl.pl file missing the directive:

:- use_module(library(shlib)).

You mentioned that before while talking about load_foreign_library/1 and use_foreign_library/2 predicates. This is an issue likely shared with other packages.

Two of the errors I get are:

ERROR: /Users/pmoura/lib/swipl/library/jpl.pl:4318:
ERROR:    catch/3: Unknown procedure: jpl:load_foreign_library/1

and:

ERROR: /Users/pmoura/lib/swipl/library/filesex.pl:80:
ERROR:    catch/3: Unknown procedure: files_ex:use_foreign_library/2
Warning: /Users/pmoura/lib/swipl/library/filesex.pl:80:
Warning:    Goal (directive) failed: files_ex:use_foreign_library(foreign(files),install_files)
1 Like
9

Thanks for testing. It seems more is working than I expected :slight_smile: I’m still a bit in doubt how to deal with the packages and specifically with use_foreign_library/1,2 After all, this is typically used as a directive, it is widely used and this seems a bit of an overkill to ask from the user:

:- autoload(library(shlib), [use_foreign_library/1]).
:- use_foreign_library(foreign(mylib)).

On the other hand, having loading foreign libraries handled by a library makes the core system smaller and is more flexible …

10

Simply adding the directive:

:- use_module(library(shlib)).

to jpl.pl and filesex.pl solve those two errors. Given the nature of the corresponding packages and how they are used, is it worth to delay loading library(shlib)?

11

This directive just further emphasizes the idea of a single predicate namespace for an application (no matter its size) where the only purpose of modules is to avoid clashes between private predicates. In my not so humble opinion, this is a step in the wrong direction.

12

Delay loading indeed makes little sense here as you need it immediately anyway. That is a bit beside the point though. Some users (like you :slight_smile: ) prefer use_module/2 and my example gets even a bit more verbose.

I’ve decided to make sure these directives are always present and these directives explicitly pull in the library. This seems a reasonable compromise to me.

13

No. It provides a single namespace for all built-ins and the core library. That is a controlled vocabulary with no ambiguities that is not huge. Using require/1 and no general autoloading (true) you define exactly what you expect from this vocabulary and what you are not going to define yourself.

Using require/1 and all but generic autoloading, @peter.ludemann’s example getting log/2 from library(quintus) by accident won’t happen. It also ensures log/2 gets into an executable, no matter in what weird way it is called.

At the same time, many Prolog systems provide between(+Low, +High, ?Number), but you can find it built-in or in some platform dependent library. Using require you just do this:

:- require([between/3]).

If more Prolog systems would provide this your porting gets a lot easier. A simple test suite could verify this between/3 indeed does what you expect it to do. As is, the Prolog community has much more agreement on predicate names than how the predicate can be made accessible.

1 Like
14

I’m now looking to errors when using CLP(Q), CLP®, and CLP(QR). I can submit a PR to fix those errors but there’s an issue there that’s puzzling me:

ERROR: /Users/pmoura/lib/swipl/library/clp/clpr.pl:57:
ERROR:    catch/3: Unknown procedure: clpr:expects_dialect/1
Warning: /Users/pmoura/lib/swipl/library/clp/clpr.pl:57:
Warning:    Goal (directive) failed: clpr:expects_dialect(swi)

There are two files where the expects_dialect(swi) directive is found: clpq.pl and clpr.pl. Skimming over the code, is not evident for me why this directive is required for code running on SWI-Prolog. Commenting the directives fix the errors in my limited testing (the tests are for coinduction examples that just so happen to use the CLP libraries).

15 
git blame clpq.pl
...
690b7120 (Vitor Santos Costa 2009-03-10 18:03:40 +0000  53) :- expects_dialect(swi).
690b7120 (Vitor Santos Costa 2009-03-10 18:03:40 +0000  54)

I guess we should in principle leave these in. That requires loading library(dialect). We could also go for a something similar as use_foreign_library/1,2: ignore the directive at the core level if it asks for swi and demand load the dialect support otherwise. Might be an overkill, but on the other hand part of the dialect support is already present in the compiler.

There is not that much point in PRs for this. I just run use-autoload file.pl and do a quick check on its output which is in most cases correct.

16

There are also missing use_module/1-2 (or autoload/1-2) directives in three files (cplr/itf_r, clpq/itf_q.pl, and clpqr/itf.pl) that result in errors when trying to use those libraries with auto-loading turned off:

:- use_module(library(apply), [maplist/2]).
:- use_module(library(apply_macros)).
17

Added these. I had a closer look at making these libraries nicer. I fear that requires some redesign to the organization though. As is, there is a lot of cross-module calling and
re-exporting.

Pushed a lot of work to make the packages compliant. This should now hold for

archive, clib, clpqr, jpl, protobufs, sgml, ssl, yaml, zlib

These packages should now work regardless of the autoload flag setting. There is still a long way to go :frowning:

1 Like