Use_module not importing all predicates

Help!
1

I’m working on a relatively large prolog project. The amount of files has grown so much that I’d like to reorder the code a bit. So i thought of the following plan:

  • group all existing source files into separate directories.
  • have a single source file for each directory which re-exports all predicates in that directory through reexport/1.
  • modify all use_module directives so that they use the high-level reexporting module, rather than the smaller submodules inside the directories.

Unfortunately, after having done this refactor, things stop working, and I get lots of ‘Undefined procedure’ errors for predicates that should actually be resolvable. Funnily, reconsulting files containing predicates that error actually makes the problem go away.

I have no idea what is going wrong. I tried creating a minimal example where I create some toy modules that import and reexport, but can’t seem to recreate the problem.

Does this problem sound familiar to anyone? Does anyone have any idea what would cause a module to not find predicates in its dependencies on first consult, but then find them anyway on second consult?

This happens on 8.0.3.

1 Like
2

To find what is going wrong I’d use the inspection predicates such as module_property/2, predicate_property/2 and source_file_property/2. I see no reason why this could not work. But, reexport/1,2 was added as part of the YAP/SWI portability issue to recreate the yap library from the SWI-Prolog one or the other way around.

Adding a load module that imports libraries from some sub application is not uncommon, but after that you typically only have to export a limited number of predicates that are of interest to the remainder of the system. The rest are internal interfaces.

If you just want to have access to a large collection of predicates without much typing I doubt this is the way to go. If these predicates follow a naming schema that is sufficiently unique to avoid conflicts I’d consider using use_module/1 to the user module to make all predicates available or turn these libraries into autoload libraries.

I’ve seen big projects, but never really experienced the need for using reexport the way you want to do. Typically each module still needs a quite limited number of predicates from other modules. Many larger applications also use some publish/subscribe way for communication between fairly independent subcomponents. See library(broadcast).

1 Like
3

The goal is very much to just export a minimal api from each module eventually. The reexport stuff is temporary, and I hoped it’d let me get away with restructuring the files first, before determining what exactly the outer api is that I’ll have to export. Unfortunately, I then ran into this issue.

4

@jan If you’re curious, you can reproduce it by cloning terminus-server, checkout to the #hub branch and run “./start.pl test”. You need the terminus_store pack though.

5

Just a thought after a quick look at the code: have you tried doing

:- reexport('foo/bar').

instead of

:- reexport(foo/bar).

?

6

That makes sense. You can figure out the dependencies by loading the still working branch and run ?- gxref. (requires xpce installed).

7

Might give it a try, but I’m traveling and only have a small laptop that doesn’t make development easy :(.

8

The bug seems to be in the processing of use_module/1 directives. The module init contains the directive:

:- use_module(core(triple)).

In turn, the module triple definition is:

:- module(triple, [
              update/5 % we're explicitely exporting this so it can be excluded elsehwere
          ]).
:- reexport([triple/base_type,
             triple/casting,
             triple/database_utils,
             triple/iana,
             triple/literals,
             triple/temp_graph,
             triple/terminus_bootstrap,
             triple/triplestore,
             triple/turtle_utils,
             triple/upgrade_db
            ]).

After loading the application, we get:

?- module(init).
true.

init:  ?- predicate_property(update(_,_,_,_,_), P).
P = interpreted ;
P = visible ;
P = static ;
P = imported_from(triplestore) ;
P = file('/home/pmoura/terminus-server/core/triple/triplestore.pl') ;
P = line_count(264) ;
P = number_of_clauses(1) ;
P = number_of_rules(1) ;
P = last_modified_generation(23984) ;
P = defined.

init:  ?- predicate_property(terminus_instance_name(_), P).
false.

But the terminus_instance_name/1 predicate is exported from module terminus_bootstrap, whose predicates are reexported by module triple:

:- module(terminus_bootstrap,[
              terminus_instance_name/1,
              terminus_schema_name/1,
              terminus_inference_name/1,
              ...

I.e. the bug seems to be that the processing of use_module/1 directives is looking into the second argument of the module/2 directive for the imported module but ignoring the predicates exported via reexport/1 directives in the same module. We arrive to the same conclusion if we use gxref. I can also reproduce the gxref results by using Logtalk’s diagrams tool to generate a xref diagram for the module init (the tool uses library(prolog_xref) when running on SWI-Prolog).

Calls to module_property/2, however, don’t seem to be affected by this bug:

?- module_property(triple, exports(L)).
L = [terminus_inference_name/1, subject_id/3, nb_remove_triple/4, id_triple_addition/4, terminus_instance_name/1, update/5, head/2, predicate_id/3, ... / ...|...].

Hope this helps.

9

I’ll never say something can’t be true, but use_module/1 uses the exported property (or, the same data as the exported property uses). I.e., use_module/1 and use_module/2 load the file if not loaded. The /2 version next imports the given predicates (if they are exported, otherwise still, but with a warning). The /1 version looks at the exported predicates, but at the module structure itself, not from the module/2 header.

:- module(r, [r/0]).
r :- writeln(r).

:- module(q, [q/0]).
:- reexport(r).
q :- writeln(q).

:- module(p, [p/0]).
:- use_module(q).

p :- q, r.

Saved into r.pl, q.pl and p.pl and loaded as swipl p.pl gives a properly working p/0 predicate.

10

How do you explain then:

init:  ?- predicate_property(terminus_instance_name(_), P).
false.

The module init calls the terminus_instance_name/1 predicate from one of its own local predicates.

11

I don’t know. I’ll try to have a look, but with my equipment here that is a bit hard. Does the minimal example not cover your claim? My claim is that at least in principle, reexported predicates behave the same as normally exported ones. I don’t have much clue why this specific predicate seems to escape this logic.

12

I also failed so far to find a minimal example that reproduces the reported problem.

But another interesting finding: if we modify the core/api/init.pl file as:

:- use_module(core(triple)).
:- use_module(core(triple), [terminus_instance_name/1]).  % added

We get:

pmoura@ubuntu18:~/terminus-server$ ./start.pl test
Warning: /home/pmoura/terminus-server/core/api/init.pl:11:
	import/1: triple:terminus_instance_name/1 is not exported (still imported into init)

But, as mentioned before, this is inconsistent with:

?- module_property(triple, exports(L)).
L = [terminus_inference_name/1, subject_id/3, nb_remove_triple/4, id_triple_addition/4, terminus_instance_name/1, update/5, head/2, predicate_id/3, ... / ...|...].
13

Enabling verbose loading (?- set_prolog_flag(verbose_load, true). gives a hint: ask.pl, which is using this terminus_instance_name/1 seems indirectly loaded from core/triple. It seems we are dealing with a case of cyclic module dependencies. use_module/1,2 should handle this correctly, but possibly reexport doesn’t.

Hopefully this gives a clue to @rob.

2 Likes
14

Still, if the root cause is reexport/1 not handling cyclic module dependencies, it’s still puzzling that module_property/2 is returning all the expected exports.

15

My guess is that the exports are used before they are complete. Will figure out later with a nice big machine and even more importantly a big screen.

2 Likes
16

We’ve now switched to re-exporting all imported predicates. This works, though it’s not the prettiest.

17

I understand it is an intermediate step. Note that there are plenty of tools in SWI-Prolog to compute the actually used dependencies. Any of these tools can make mistakes as Prolog allows constructing goals that are completely non-tracktable. Should be a good approximation though and if there is a good test suite it shouldn’t be too hard to fix the few possible omissions.

18

I’ve had the same experience maybe a year ago or so, using header like files with reexports. Also couldn’t get to the bottom of it or produce a toy example recreating it. Ended up with a big refactor, I think back then I tried using broadcast as an interface between “modules”, but that got refactored out too.

19

Even with those changes, I still see broken imports. For example:

$ swilgt start.pl
...
?- module_property(util, P).
P = class(user) ;
P = file('/Users/pmoura/terminus-server/core/util.pl') ;
P = line_count(1) ;
P = exports([terminus_schema_path/1, decimal/3, dateTime/11, time/8, truncate_list/4, exhaust/1, digit/3, ... / ...|...]) ;
P = exported_operators([op(601, xfx, @), op(601, xfx, ^^), op(920, fy, *), op(700, xfy, <>)]) ;
P = program_size(2448) ;
P = last_modified_generation(13327).

I then apply Logtalk tools, which compile the TerminusDB modules as objects, which does not modifying in any way those modules, and I get:

?- {loader}.
...
?- module_property(util, P).
P = class(user) ;
P = file('/Users/pmoura/terminus-server/core/util.pl') ;
P = line_count(1) ;
P = exports([terminus_schema_path/1, decimal/3, dateTime/11, time/8, truncate_list/4, exhaust/1, digit/3, ... / ...|...]) ;
P = exported_operators([op(601, xfx, @), op(601, xfx, ^^)]) ;
P = program_size(2448) ;
P = last_modified_generation(13327).

Notice the difference? Two of the util module exported operators are no longer reported.

20

Followup. Managed to isolate what’s causing this operator bug. When compiling the util module as an object, the resulting Prolog file is loaded using the goal load_files(Path, [derived_from(Source)| Options]). Here, Source is the path to the module file ('/Users/pmoura/terminus-server/core/util.pl') while Path is the plain Prolog file generated by the Logtalk compiler. If I remove the derived_from(Source) option, the bug no longer occurs. Thus, the handling of the derived_from /1 option seems to be causing the followup call to module_property/2 to report an empty list of exported operators. Hope this helps.