The dynamic database is evil and it is holding my program prisoner!

[stassa.p]

I have an automatically generated program that calls some automatically generated auxiliary programs. I assert the lot into the dynamic database in their own module, and call the top-level program in the new module.

The dynamic database is evil so I typically do that in a setup_call_cleanup/3 call, like this:

S = (assert(Maps:M0,Ref)
    % Top-level program:
    ,assert_program(program,Ss,Rs_S)
    % Auxiliary programs:
    ,assert_program(program,As,Rs_A)
     )
,G = (% The goal in E is s/2, the top-level program.
      call(program:E) 
     )
,C = (erase_program_clauses([Ref|Rs_S])
      ,erase_program_clauses(Rs_A)
     )
     ,setup_call_cleanup(S,G,C)

The goal call(program:E) fails. This is difficult to understand because one of the auxiliary programs’ clauses unifies with one of the body goals in the top-level program, s/2.

Here’s a protocol-log of a tracing session where I show this; I’ve listed s/2 and the auxiliary, step_right/1 in the tracer and I’m also spying them so I can leap to their goals:

[debug]  ?- spy(program:[s/2,step_right/2]).

% Spy point on program:s/2
% Spy point on program:step_right/2
true.

[debug]  ?- program([s/2],experiment_file,_Ps), solver_executor(_Ps,id(tessera_1),print,_As,_M), print_map(tiles,_M).

 * Call: (20) program:s([dyn1, 1/7, s, q0, [_125922, _125928|...], [_126022|...], [...|...]|...], [dyn1, 7/1, e, _124374, [], [], []|...]) ? bbreak
% Break level 1
[debug] [1]  ?- listing(program:s/2).

:- dynamic s/2.

s(A, B) :-
    step_down(A, B).
s(A, B) :-
    step_left(A, B).
s(A, B) :-
    step_right(A, B).
s(A, B) :-
    step_up(A, B).
s(A, B) :-
    step_down(A, C),
    s(C, B).
s(A, B) :-
    step_left(A, C),
    s(C, B).
s(A, B) :-
    step_right(A, C),
    s(C, B).
s(A, B) :-
    step_up(A, C),
    s(C, B).

true.

[debug] [1]  ?- 

% Exit break level 1
 * Call: (20) program:s([dyn1, 1/7, s, q0, [_125922, _125928|...], [_126022|...], [...|...]|...], [dyn1, 7/1, e, _124374, [], [], []|...]) ? lleap
 * Call: (21) program:step_right([dyn1, 1/7, s, q0, [_125922, _125928|...], [_126022|...], [...|...]|...], [dyn1, 7/1, e, _124374, [], [], []|...]) ? lleap
 * Fail: (21) program:step_right([dyn1, 1/7, s, q0, [_125922, _125928|...], [_126022|...], [...|...]|...], [dyn1, 7/1, e, _124374, [], [], []|...]) ? lleap
 * Call: (21) program:step_right([dyn1, 1/7, s, q0, [_125922, _125928|...], [_126022|...], [...|...]|...], _135120) ? wwrite
 * Call: (21) program:step_right([dyn1, 1/7, s, q0, [_125922, _125928, _125934, _125940, _125946, _125952, _125958, _125964, _125970, _125976, _125982, _125988, _125994, _126000, _126006, _126012], [_126022, _126028, _126034, _126040, _126046, _126052, _126058, _126064, _126070, _126076, _126082, _126088, _126094, _126100, _126106, _126112], [_126122, _126128, _126134, _126140, _126146, _126152, _126158, _126164, _126170, _126176, _126182, _126188, _126194, _126200, _126206, _126212], [_126222, _126228, _126234, _126240, _126246, _126252, _126258, _126264, _126270, _126276, _126282, _126288, _126294, _126300, _126306, _126312]], _135120) ? bbreak
% Break level 1
[debug] [1]  ?- listing(program:step_right/2).

:- dynamic step_right/2.

step_right([dyn1, 1/1, f, Q0, [Q0|Qs], [ppuu|Os], [right|As], [q1|Qs_]], [dyn1, 2/1, f, q1, Qs, Os, As, Qs_]).
step_right([dyn1, 1/7, s, Q0, [Q0|Qs], [upuu|Os], [right|As], [q1|Qs_]], [dyn1, 2/7, f, q1, Qs, Os, As, Qs_]).
step_right([dyn1, 2/1, f, Q0, [Q0|Qs], [upup|Os], [right|As], [q1|Qs_]], [dyn1, 3/1, f, q1, Qs, Os, As, Qs_]).
step_right([dyn1, 2/7, f, Q0, [Q0|Qs], [upup|Os], [right|As], [q1|Qs_]], [dyn1, 3/7, f, q1, Qs, Os, As, Qs_]).
step_right([dyn1, 3/1, f, Q0, [Q0|Qs], [ppup|Os], [right|As], [q1|Qs_]], [dyn1, 4/1, f, q1, Qs, Os, As, Qs_]).
step_right([dyn1, 4/1, f, Q0, [Q0|Qs], [upup|Os], [right|As], [q1|Qs_]], [dyn1, 5/1, f, q1, Qs, Os, As, Qs_]).
step_right([dyn1, 5/3, f, Q0, [Q0|Qs], [uppu|Os], [right|As], [q1|Qs_]], [dyn1, 6/3, f, q1, Qs, Os, As, Qs_]).
step_right([dyn1, 5/7, f, Q0, [Q0|Qs], [uppu|Os], [right|As], [q1|Qs_]], [dyn1, 6/7, f, q1, Qs, Os, As, Qs_]).
step_right([dyn1, 6/3, f, Q0, [Q0|Qs], [upup|Os], [right|As], [q1|Qs_]], [dyn1, 7/3, f, q1, Qs, Os, As, Qs_]).
step_right([dyn1, 6/7, f, Q0, [Q0|Qs], [upup|Os], [right|As], [q1|Qs_]], [dyn1, 7/7, f, q1, Qs, Os, As, Qs_]).

true.

[debug] [1]  ?- step_right([dyn1, 1/7, s, q0, [_125922, _125928, _125934, _125940, _125946, _125952, _125958, _125964, _125970, _125976, _125982, _125988, _125994, _126000, _126006, _126012], [_126022, _126028, _126034, _126040, _126046, _126052, _126058, _126064, _126070, _126076, _126082, _126088, _126094, _126100, _126106, _126112], [_126122, _126128, _126134, _126140, _126146, _126152, _126158, _126164, _126170, _126176, _126182, _126188, _126194, _126200, _126206, _126212], [_126222, _126228, _126234, _126240, _126246, _126252, _126258, _126264, _126270, _126276, _126282, _126288, _126294, _126300, _126306, _126312]], _135120) = step_right([dyn1, 1/7, s, Q0, [Q0|Qs], [upuu|Os], [right|As], [q1|Qs_]], [dyn1, 2/7, f, q1, Qs, Os, As, Qs_]).

_125922 = Q0, Q0 = q0,
_126022 = upuu,
_126122 = right,
_126222 = q1,
_135120 = [dyn1,2/7,f,q1,[_125928,_125934,_125940,_125946,_125952,_125958,_125964,_125970,_125976,_125982,_125988,_125994,_126000,_126006|...],[_126028,_126034,_126040,_126046,_126052,_126058,_126064,_126070,_126076,_126082,_126088,_126094,_126100|...],[_126128,_126134,_126140,_126146,_126152,_126158,_126164,_126170,_126176,_126182,_126188,_126194|...],[_126228,_126234,_126240,_126246,_126252,_126258,_126264,_126270,_126276,_126282,_126288|...]],
Qs = [_125928,_125934,_125940,_125946,_125952,_125958,_125964,_125970,_125976,_125982,_125988,_125994,_126000,_126006,_126012],
Os = [_126028,_126034,_126040,_126046,_126052,_126058,_126064,_126070,_126076,_126082,_126088,_126094,_126100,_126106,_126112],
As = [_126128,_126134,_126140,_126146,_126152,_126158,_126164,_126170,_126176,_126182,_126188,_126194,_126200,_126206,_126212],
Qs_ = [_126228,_126234,_126240,_126246,_126252,_126258,_126264,_126270,_126276,_126282,_126288,_126294,_126300,_126306,_126312].

[debug] [1]  ?- 

% Exit break level 1
 * Call: (21) program:step_right([dyn1, 1/7, s, q0, [_125922, _125928, _125934, _125940, _125946, _125952, _125958, _125964, _125970, _125976, _125982, _125988, _125994, _126000, _126006, _126012], [_126022, _126028, _126034, _126040, _126046, _126052, _126058, _126064, _126070, _126076, _126082, _126088, _126094, _126100, _126106, _126112], [_126122, _126128, _126134, _126140, _126146, _126152, _126158, _126164, _126170, _126176, _126182, _126188, _126194, _126200, _126206, _126212], [_126222, _126228, _126234, _126240, _126246, _126252, _126258, _126264, _126270, _126276, _126282, _126288, _126294, _126300, _126306, _126312]], _135120) ? 
creep
 * Fail: (21) program:step_right([dyn1, 1/7, s, q0, [_125922, _125928, _125934, _125940, _125946, _125952, _125958, _125964, _125970, _125976, _125982, _125988, _125994, _126000, _126006, _126012], [_126022, _126028, _126034, _126040, _126046, _126052, _126058, _126064, _126070, _126076, _126082, _126088, _126094, _126100, _126106, _126112], [_126122, _126128, _126134, _126140, _126146, _126152, _126158, _126164, _126170, _126176, _126182, _126188, _126194, _126200, _126206, _126212], [_126222, _126228, _126234, _126240, _126246, _126252, _126258, _126264, _126270, _126276, _126282, _126288, _126294, _126300, _126306, _126312]], _135120) ? aabort
% Execution Aborted
[debug]  ?- noprotocol.

Note that the first call to step_right/2 fails correctly. I don’t understand why the second call fails. The second call unifies with the second clause of step_right/2 in the dynamic database (the one that starts with step_right([dyn1, 1/7, s, Q0 ... ).

What is going on here? This is the first time this happens [edit: I mean that I’ve used this technique many times and it’s the first time it fails like this]. If I load s/2, step_right/2 and the other auxiliaries from a file, call(program:E) succeeds, so there’s nothing wrong with the query, as such.

My suspicion is that there is some sort of clash with further clauses of step_right/2 (and the other auxiliaries) that are loaded into the program database from a module file, before the above code is executed. However, that module has a different name than my on-the-fly populated, program module. Do I have to use a multifile directive somewhere, nonetheless?

I tried running call(program:E) inside a transaction just in case magick works but I guess it doesn’t.

Btw, I’m on Win 11 64 bits with SWI 9.3.7-1 (development version). I haven’t tried this on Linux yet. Will it make a difference?

[jan]

I doubt this is an OS specific issue. Debugging this is a bit hard. I’d list the dynamic program to a file, load that file and next debug it as static code using the graphical debugger.

[stassa.p]

Hi Jan, thanks for replying. My problem is that if I write the dynamic program to a file, my query succeeds. It’s only when I try to run it from the dynamic database that it fails. That’s what I don’t quite get.

In the tracing above -sorry, I know it’s a bit of a mess- I was trying to show that a goal that fails unifies with a clause in the database, and so shouldn’t be failing. It looks like Prolog can’t find the clause that unifies with that goal, unless I load that clause from a file, rather than having it in the dynamic database.

[stassa.p]

I tried to make a shorter, possibly more readable version of my tracing log from above. I added some '…'s manually to make lists easier to read but the success and failure results are from Prolog:

% All lists shortened manually

% Goal that fails:
* Call: (21) program:step_right([dyn1, 1/7, s, q0, [_125922, _125928|...], [_126022|...], [...|...]|...], _135120) ? wwrite

[debug] [1]  ?- listing(program:step_right/2).

% step_right/2 clause is in the database.
% ... 
step_right([dyn1, 1/7, s, Q0, [Q0|Qs], [upuu|Os], [right|As], [q1|Qs_]], [dyn1, 2/7, f, q1, Qs, Os, As, Qs_]).
% ... more clauses

% Unification succeeds:
[debug] [1]  ?- 
step_right([dyn1, 1/7, s, q0, [_125922, ...], [_126022, ...], [_126122, ...], [_126222, ...]], _135120) = step_right([dyn1, 1/7, s, Q0, [Q0|Qs], [upuu|Os], [right|As], [q1|Qs_]], [dyn1, 2/7, f, q1, Qs, Os, As, Qs_]).

_125922 = Q0, Q0 = q0,
_126022 = upuu,
_126122 = right,
_126222 = q1,
_135120 = [dyn1,2/7,f,q1,[_125928,..],[_126028,..],[_126128,..],[_126228,...]],
Qs = [_125928, ... ],
Os = [_126028, ... ],
As = [_126128, ... ],
Qs_ = [_126228,... ].

% But the call fails:
% Exit break level 1
 * Call: (21) program:step_right([dyn1, 1/7, s, q0, [_125922, ...], [_126022, ...], [_126122, ...], [_126222, ...]], _135120) ? 
creep
 * Fail: (21) program:step_right([dyn1, 1/7, s, q0, [_125922, ...], [_126022, ...], [_126122, ...], [_126222, ...]], _135120) ? aabort
% Execution Aborted

[peter.ludemann]

@stassa.p

My suspicion is that there is some sort of clash with further clauses of step_right/2 (and the other auxiliaries) that are loaded into the program database from a module file, before the above code is executed.

Are you perhaps running into logical update view vs immediate update view? Or, do you need transactions?

(Or there’s a bug in your assert_program and erase_program code?)

[stassa.p]

Hi Peter, thanks for the reply. I spent some time yesterday staring at the part of the documentation that you linked to (and friends), and I think this part:

Only clauses with a‘created’ …‘erased’interval that encloses the generation of the current goal are considered visible.

Should mean that calling my goal in the “call” part of setup_call_cleanup/3, after asserting the necessary clauses in the “setup” part should mean that those clauses “enclose” the goal’s generation. So they should be visible.

Just to check- I tried adding a listing/1 call right before the call to the program that fails, and the definition of both the main program and its auxiliaries are listed correctly:

        ,S = (assert(Maps:M0,Ref)
             ,assert_program(program,Ss,Rs_S)
             ,assert_program(program,As,Rs_A)
             )
        ,G = (listing(program:s/2)
             ,listing(program:step_down/2)
             ,call(program:E)
             )
        ,C = (erase_program_clauses([Ref|Rs_S])
             ,erase_program_clauses(Rs_A)
             ,untable(program:s/2)
             )
        ,setup_call_cleanup(S,G,C)

Output:

[debug]  ?- program([s/2],experiment_file,_Ps), solver_executor(_Ps,id(tessera_1),print,_As,_M), print_map(tiles,_M).
:- dynamic s/2.

s(A, B) :-
    step_down(A, B).
s(A, B) :-
    step_left(A, B).
s(A, B) :-
    step_right(A, B).
s(A, B) :-
    step_up(A, B).
s(A, B) :-
    step_down(A, C),
    s(C, B).
s(A, B) :-
    step_left(A, C),
    s(C, B).
s(A, B) :-
    step_right(A, C),
    s(C, B).
s(A, B) :-
    step_up(A, C),
    s(C, B).

:- dynamic step_right/2.

step_right([dyn1, 1/1, f, Q0, [Q0|Qs], [ppuu|Os], [right|As], [q1|Qs_]], [dyn1, 2/1, f, q1, Qs, Os, As, Qs_]).
step_right([dyn1, 1/7, s, Q0, [Q0|Qs], [upuu|Os], [right|As], [q1|Qs_]], [dyn1, 2/7, f, q1, Qs, Os, As, Qs_]).
step_right([dyn1, 2/1, f, Q0, [Q0|Qs], [upup|Os], [right|As], [q1|Qs_]], [dyn1, 3/1, f, q1, Qs, Os, As, Qs_]).
step_right([dyn1, 2/7, f, Q0, [Q0|Qs], [upup|Os], [right|As], [q1|Qs_]], [dyn1, 3/7, f, q1, Qs, Os, As, Qs_]).
step_right([dyn1, 3/1, f, Q0, [Q0|Qs], [ppup|Os], [right|As], [q1|Qs_]], [dyn1, 4/1, f, q1, Qs, Os, As, Qs_]).
step_right([dyn1, 4/1, f, Q0, [Q0|Qs], [upup|Os], [right|As], [q1|Qs_]], [dyn1, 5/1, f, q1, Qs, Os, As, Qs_]).
step_right([dyn1, 5/3, f, Q0, [Q0|Qs], [uppu|Os], [right|As], [q1|Qs_]], [dyn1, 6/3, f, q1, Qs, Os, As, Qs_]).
step_right([dyn1, 5/7, f, Q0, [Q0|Qs], [uppu|Os], [right|As], [q1|Qs_]], [dyn1, 6/7, f, q1, Qs, Os, As, Qs_]).
step_right([dyn1, 6/3, f, Q0, [Q0|Qs], [upup|Os], [right|As], [q1|Qs_]], [dyn1, 7/3, f, q1, Qs, Os, As, Qs_]).
step_right([dyn1, 6/7, f, Q0, [Q0|Qs], [upup|Os], [right|As], [q1|Qs_]], [dyn1, 7/7, f, q1, Qs, Os, As, Qs_]).

false.

This clause should be unifying with the call to step_right/2 in the 7’th clause of s/2 (one of the recursive ones):

step_right([dyn1, 1/7, s, Q0, [Q0|Qs], [upuu|Os], [right|As], [q1|Qs_]], [dyn1, 2/7, f, q1, Qs, Os, As, Qs_]).

It does when I unify it manually in a break level, and also when I load s/2, step_right/2 and co. from a static file. But it doesn’t unify when I have them only in the dynamic db.

I’ll try running everything in a transaction. I gave it a try yesterday but I think I did the wrong thing.

[stassa.p]

@peter.ludemann

(Or there’s a bug in your assert_program and erase_program code?)

Nothing is impossible but hey’re simple enough and I’ve used them for over 4 years so I should have found all the bugs by now.

Here’s assert_program/2. The numbervars/1’s are to avoid writing a clause if a more general version is already in the dynamic database:

%!	assert_program(+Module,+Program,-Clause_References) is det.
%
%	As assert_program/2 but also binds a list of Clause_References.
%
assert_program(M,Ps,Rs):-
	assert_program(M,Ps,[],Rs).

assert_program(_,[],Rs,Rs):-
	!.
assert_program(M,[A|P],Acc,Bind):-
	copy_term(A,A_)
	,numbervars(A_)
	,clause(M:A_,true)
	,!
	,assert_program(M,P,Acc,Bind).
assert_program(M,[C|P],Acc,Bind):-
	copy_term(C,H:-B)
	,numbervars(H:-B)
	,clause(M:H,B)
	,!
	,assert_program(M,P,Acc,Bind).
assert_program(M,[C|P],Acc,Bind):-
	assert(M:C,Ref)
	,assert_program(M,P,[Ref|Acc],Bind).

Here’s erase_program_clauses/1:

%!	erase_program_clauses(-Clause_References) is det.
%
%	Erase a list of Clause_References from the dynamic database.
%
%	Clause_References is meant to be a list of references of a
%	program's clauses asserted to the dynamic database with
%	assert_program/3.
%
erase_program_clauses([]):-
	!.
erase_program_clauses([Ref|Rs]):-
	erase(Ref)
	,erase_program_clauses(Rs).

[stassa.p]

And I just tried enclosing the steup_call_cleanup/3 call to a transaction/1 and it still can’t find the right clause. I can see it in there, though. How can it be hiding from me? My own clause? I generated it, you know.

[stassa.p]

Btw, this thing I’m trying to do, it’s cool beans. I have to share. Here’s a screenshot from a session where I’ve loaded the necessary clauses from a static file, like @jan suggested:

The first map titled “Searching map” is a model-free agent that searches an unseen map exhaustively. The agent is “model free” in that it doesn’t see the map, and doesn’t know how its actions affect the state of the map (i.e. the position of the agent on the map). The yellow arrows show the path of the agent through the map. Red tiles are walls (unpassable), green tiles are floor (passable).“S” is the starting location.

The second map, titled “Seeking start and exit tiles” is the same model-free agent that searches the same map for a start and exit tile, maked “S” and “E”. This has to be done in two steps because the agent literally can’t see the tiles it’s stepping on. It just marks its starting and final location as “S” and “E”

The third map, titled “Solving map”, is a model-based agent that solves the unseen map using a map derived by the model-free agent, by SLAM (Simultaneous Localisation and Mapping). This agent is model-based in that it has a representation of the state of the environment, and how its actions change the environment -that’s the step_right/2 etc clauses I show above. The “agent” is just the s/2 program, also listed above.

The final map is the SLAM map derived by the model-free agent in the first two runs, where it explores the map. Gray dots are unobservable locations. There’s a wall all around the map because the agent perceives areas outside the map as “unpassable”. This map is used to generate the model of the model-based agent.

So that’s going from a model-free agent, to a model-based agent. The fun part is that the model-free agent is trained by the model-based agent, on a different map, and it learns to solve basically any grid map. All done with ILP of course (and a lot of hard-coding for the SLAM).

[stassa.p]

Hey guys, just so you know, I figured it out. My program generator is generating clauses with Skolemised variables, like '$VAR'(As) etc, for pretty-printing to a file. Loaded from a file, they would be read as having variables rather than Skolem constants. That was the only way I used the generator originally. But when I started asserting the same clauses to the dynamic database, well, those Skolem constants were now ground terms and caused unification to fail as I showed above. Except of course, when I was tracing my program the Skolem constants looked just like variables in the tracer, whether I loaded them from a file or asserted them to the dynamic database. D’oh.

Btw I know I can set the tracer to show Skolem constants (numbered vars) but this tends to hurt my eyes so I usually turn it off. This one time it could have saved me a lot of confusion!

Anyway I haven’t fixed this yet, will need some elbow grease, but I figured it out. Thanks for the support!