Inspecting programs with attributed variables using clause/2

[stassa.p]

@ridgeworks

Dead end? Well, maybe not quite. For any given module, e.g., clpBNR, the module class can be modified using set_module/1, so it can be arranged that clpBNR can have class library and so look like any SWIP library module. Unfortunately, the circumstances under which this can be done is restrictive, i.e., called from inside the module at module initialization time based on experiments I’ve done.

That’s an interesting idea. I think it should be even possible to create a new, temporary module with the “library” class property from any imported library and a dummy module that only sets that property. I’d have to dig a bit in the module documentation to remember how but I think it’s possible to hack something up. Whether it’s a stable thing I have no idea.

Unfortunately…

@ridgeworks

Can I assume that any code not subject to meta-interpretation will be imported from somewhere, i.e., it will be constructed as a well formed module and not just random predicates dumped into the user context?

No

I don’t dare make such an assumption. You’d think and I had thought that modules would be widely adopted but it seems, no. Lots of legacy code and even modern code in ILP doesn’t use modules. Anyway it’s yet another restriction imposed on the user and I don’t want to do that.

I don’t know. Maybe I should consider it. I was just unpleasantly surprised at some point to realise how little Prolog code “in the wild” doesn’t use modules. I think SWI packs are an exception maybe. Don’t know for sure.

Unfortunately I have to accept that Prolog modules are … not great. I use them my self all the time, I mean I never write any Prolog code that isn’t in a module anymore but it seems there’s enough fiddliness around them that many people simply don’t bother.

[ridgeworks]

@stassa.p

I don’t dare make such an assumption. You’d think and I had thought that modules would be widely adopted but it seems, no. Lots of legacy code and even modern code in ILP doesn’t use modules. Anyway it’s yet another restriction imposed on the user and I don’t want to do that.

Really? We’re only talking about code that we don’t want to meta-interpret.

So the scenario is there’s a body of unstructured code in a set of files. Some of it we want to meta-interpret and some not. We can’t tell and you don’t want the user to deal with the question. Isn’t this a little unrealistic?

I might suggest that you’d only use such unstructured legacy code if you wanted to meta-interpret it for its “learning” value, but your call.

@stassa.p

Unfortunately I have to accept that Prolog modules are … not great. I use them my self all the time, I mean I never write any Prolog code that isn’t in a module anymore but it seems there’s enough fiddliness around them that many people simply don’t bother.

Now this surprises me. Modules (and interfaces) have been a part of the software engineering discipline, independent of programming language, for over 50 years. Any serious Prolog, and any currently active ones AFAIK, has had them for decades. (@jan has probably a better perspective than me.) People who don’t “bother” to use them are not writing serious software for public consumption, IMHO. Or they’re just lazy. Are they really producing anything you want to use?

As a dynamic language capable of introspection, existing Prolog module implementations have some complexities that other languages don’t have to deal with, but that’s no reason not to use them.

[ridgeworks]

@stassa.p

The new exception may just be because debugging makes everything more memory-intensive. After increasing the stack to 8GB I no longer get the exception. So probably a red herring but reporting it for the sake of completeness.

EDIT: oops, no, hang on. Yeah I still get the stack overflow with 8GB stack limit. I still think it’s a red herring.

Since executing the goal in Prolog terminates, it seems clear that your meta-interpreter is in an infinite loop; no amount of memory will fix that. I think you need to use the debugger to find out why, starting with the first call to prove. Stepping through it (it’s a small loop) should expose the issue fairly quickly.

You can also use the debugger to step through normal Prolog execution and see the difference. E.g.,

?- trace,(X::real,[A,B]::real(0,1)).
   Call: (15) clpBNR:(_25236::real) ? creep
   Call: (16) clpBNR:list(_25236) ? skip
   Fail: (16) clpBNR:list(_25236) ? creep
   Redo: (15) clpBNR:(_25236::real) ? creep
   Call: (16) clpBNR:g_read('$clpBNR:thread_init_done', _29586) ? skip
   Exit: (16) clpBNR:g_read('$clpBNR:thread_init_done', 1) ? creep
   Call: (16) var(real) ? creep
   Fail: (16) var(real) ? creep
   Redo: (15) clpBNR:(_25236::real) ? creep
   Call: (16) string(real) ? creep
   Fail: (16) string(real) ? creep
   Redo: (15) clpBNR:(_25236::real) ? creep
   Call: (16) real=..[_35734|_35736] ? creep
   Exit: (16) real=..[real] ? creep
   Call: (16) _37294=..[','] ? creep
   Exit: (16) (',')=..[','] ? creep
   Call: (16) clpBNR:int_decl_(real, ',', _25236) ? skip
   Exit: (16) clpBNR:int_decl_(real, ',', _39716{real(-1.0e+16,1.0e+16)}) ? creep
   Exit: (15) clpBNR:(_39716{real(-1.0e+16,1.0e+16)}::real) ? creep
   Call: (15) clpBNR:([_25244, _25250]::real(0, 1)) ? skip
   Exit: (15) clpBNR:([_42168{real(0,1)}, _42240{real(0,1)}]::real(0, 1)) ? creep
X::real(-1.0e+16, 1.0e+16),
A::real(0, 1),
B::real(0, 1).

Debugging is more memory intensive because things like last call optimization are disabled.

[jan]

@ridgeworks

The fact that it’s happening with tabling disabled should make it easier to find

I’ve missed most of the discussion while in the hospital. But, tabling and constraints do not play in SWI-Prolog. There is basic support for attributed variables in tabling, but that is just one step. So, if you combine them, make sure they only connect through normal Prolog terms.

[ridgeworks]

@jan

So, if you combine them, make sure they only connect through normal Prolog terms.

For those of us with only a superficial understanding of tabling, can you provide a simple example? Or point to an existing one?

Best wishes for a speedy recovery.

[jan]

@ridgeworks

So it would appear that a one line patch to clpBNR could solve the immediate problem. But maybe a more general purpose solution could be done, e.g., adding a module_property option to load_files/2 (analogous to option module(M) for renaming modules).

I don’t know. The notion has been introduced to restrict check/0 and similar analysis tools. Adding a class pack might be worth considering?

@stassa.p

The new exception may just be because debugging makes everything more memory-intensive

Not decisive, but this and the other stack traces suggests a loop in clpBNR that can be resolved by tabling, but then we get broken semantics due to the poor interaction of constraints and tabling. And yes, debugging disables last call optimization and ensures a choice point at the start of all calls, so we can redo. That uses a lot more memory.

@ridgeworks

For those of us with only a superficial understanding of tabling, can you provide a simple example? Or point to an existing one?

Not right now. Tabling is best applied to pure Prolog programs though. Commit (cut) easily causes tables to be incomplete (XSB has a runtime check for this, SWI-Prolog not yet). Tables do store attributes, but more is needed to fully support constraints and If I recall well, David Warren claimed a general solution does not exist. So, you can table a predicate that calls a constraint solver, as long as the same predicate resolves any pending constraints.

[ridgeworks]

@jan

I don’t know. The notion has been introduced to restrict check/0 and similar analysis tools. Adding a class pack might be worth considering?

A separate class value is probably the best solution. I would think modifying the class should be generally discouraged, otherwise it really doesn’t mean anything.

@jan

Not decisive, but this and the other stack traces suggests a loop in clpBNR that can be resolved by tabling, but then we get broken semantics due to the poor interaction of constraints and tabling.

To be determined, but I think the loop is in the (attempted) meta-interpretation of a clpBNR dependant goal. I’m not aware of any mechanism for creating a loop entirely within clpBNR, which is why I’m interested in tracking down the root cause.

@jan

Tables do store attributes, but more is needed to fully support constraints and If I recall well, David Warren claimed a general solution does not exist. So, you can table a predicate that calls a constraint solver, as long as the same predicate resolves any pending constraints.

I think that’s saying that attributed variables should not appear in the arguments of a tabled predicate. Maybe that can be relaxed a bit if the attributes are not changed during the execution of the predicate, but that hypothesis requires verification.

In the case of clpBNR, it may be that attributed variables are stored in the tables but unclear how that “data” stays connected to the rest of the interval constraint network. Probably it isn’t.

[stassa.p]

Sorry for being away for a while. The one above is the simplest example I could think of. It doesn’t produce the error in my program but after @jan’s posts I guess that doesn’t matter, I’ll just have to keep tabling disabled. I don’t think that’s going to be a big blocker for me.

@jan

I don’t know. The notion has been introduced to restrict check/0 and similar analysis tools. Adding a class pack might be worth considering?

I’d probably welcome it but like you said earlier I don’t think there’s ever going to be a complete solution.

I’m sorry to hear you were in hospital. I hope that was temporary and that you get better soon.

[stassa.p]

@ridgeworks

To be determined, but I think the loop is in the (attempted) meta-interpretation of a clpBNR dependant goal. I’m not aware of any mechanism for creating a loop entirely within clpBNR, which is why I’m interested in tracking down the root cause.

Oh, that’s a good point. My main project does not meta-interpret clpBNR goals (after adding a new check to avoid it as discussed above). My simple meta-interpreter does though. I’ll see if I can add a check to it too and if that changes something.

Edit: I changed my simple meta-interpreter like this:

:-module(meta_clp,[prove/1
                  ]).

:- table(prove/1).

prove(true).
prove((L,Ls)):-
        prove(L)
        ,prove(Ls).
prove(L):-
        L \== (_,_)
        ,L \== true
        ,guarded_clause(L,Bs)
        ,prove(Bs).

guarded_clause(L,true):-
        predicate_property(L,imported_from(clpBNR))
        ,debug(guarded_clause,'Calling ~w with Prolog',[L])
        ,call(L).
guarded_clause(L,Bs):-
        \+ predicate_property(L,imported_from(clpBNR))
        ,debug(guarded_clause,'Meta-interpreting ~w',[L])
        ,clause(L,Bs).

And this time it terminates although the guard doesn’t seem to work as expected:

% Just to be sure
122 ?- abolish_all_tables.
true.

123 ?- debug(guarded_clause).
true.

124 ?- G = (X::real, [A,B]::real(0,1)), meta_clp:prove(G).
% Calling _11964::real with Prolog
% Calling [_11972,_11978]::real(0,1) with Prolog
% Meta-interpreting _11964::real,[_11972,_11978]::real(0,1)
% Meta-interpreting call((_11964::real,[_11972,_11978]::real(0,1)))
G = (X::real, [A, B]::real(0, 1)),
X::real(-1.0e+16, 1.0e+16),
A::real(0, 1),
B::real(0, 1).

That’s difficult to understand as turning off tabling definitely eliminates the error in my main project. Also I guess this doesn’t change the fact that tabling and constraints don’t currently play well.

Btw, sorry to ask a silly question but what does the cut have to do with CLP?

[ridgeworks]

@stassa.p

And this time it terminates although the guard doesn’t seem to work as expected:

I think the guard does work as expected, but both the second and third clauses of prove are executed, i.e., (G1,G2) unifies with the head of both clauses. So best solution IMO is to put a cut after the neck in the second clause.

@stassa.p

Btw, sorry to ask a silly question but what does the cut have to do with CLP?

Which cut are you referring to? Where appropriate CLP predicates may leave choice points like any other Prolog code, e.g., searching for solutions consistent with defined constraints, like clpBNR/solve/1.

[peter.ludemann]

@jan

Commit (cut) easily causes tables to be incomplete

Does -> ; also cause tables to be incomplete? Should *-> ; be used instead?

[ridgeworks]

@ridgeworks

So best solution IMO is to put a cut after the neck in the second clause.

My mistake; I missed the additional guards in the third clause. But I think the second guard should be L \= (_,_), not L \==(_,_). (Anonymous variables will not be identical to anything in L .)

Now:

?- prove((X::real,[A,B]::real(-10,10))).
X::real(-1.0e+16, 1.0e+16),
A::real(-10, 10),
B::real(-10, 10) ;
false.

What the guards don’t do is remove the (spurious) choicepoints left by prove/1. I think only a cut (or SSU - single side unification) will deal with that. Note that guarded_clause/2 will also leave a (spurious) choicepoint.

You may also want to guard against ?- prove(L)..

[jan]

@peter.ludemann

Does -> ; also cause tables to be incomplete? Should *-> ; be used instead?

Not sure. I don’t recall whether *->/2 and delimited continuations play well. But, *->/2 is rarele what you want (but very useful in these rare cases). At some point we should implement the runtime check that verifies a commit may prune a table.

@ridgeworks

What the guards don’t do is remove the (spurious) choicepoints left by prove/1. I think only a cut (or SSU - single side unification) will deal with that. Note that guarded_clause/2 will also leave a (spurious) choicepoint.

There is '$meta_call'/1 defined in boot/init.pl that provides a complete meta call (call/1) implementation that you can use for inspriration on how to deal with complete meta interpretation for SWI-Prolog, including modules and the various control structures.

[stassa.p]

@ridgeworks

Which cut are you referring to? Where appropriate CLP predicates may leave choice points like any other Prolog code, e.g., searching for solutions consistent with defined constraints, like clpBNR/solve/1.

I was referring to @Jan’s comment about tabling and constraints, where he mentioned how cutting can cause tables to be left incomplete:

@jan

Not right now. Tabling is best applied to pure Prolog programs though. Commit (cut) easily causes tables to be incomplete (XSB has a runtime check for this, SWI-Prolog not yet). Tables do store attributes, but more is needed to fully support constraints and If I recall well, David Warren claimed a general solution does not exist. So, you can table a predicate that calls a constraint solver, as long as the same predicate resolves any pending constraints.

I took that to mean that constraints have something to do with the cut, or anyway with committing to choices. I confess that I am pretty much clueless about the internals of CLP (of all sorts) implementations or indeed the theory behind them (although from what I heard the theory was basically “Alain’s presentation slides” - meaning Alain Colmerauer).

[ridgeworks]

@stassa.p

I took that to mean that constraints have something to do with the cut, or anyway with committing to choices. I confess that I am pretty much clueless about the internals of CLP (of all sorts) implementations or indeed the theory behind them (although from what I heard the theory was basically “Alain’s presentation slides” - meaning Alain Colmerauer).

I think SWI-Prolog -- Constraint Logic Programming has a pretty good summary of CLP. Not sure understanding the implementation/internals of any particular variant is very helpful.