Use of `attribute_goals//1` to compose answers

[ridgeworks]

Apologies on the length of this post but think I need to provide some motivation.

Background: Attributed variables are a core feature supporting many (most?) CLP dialects on Prolog. By permitting logic variables to be “annotated”, they effectively permit constraint networks to be constructed without requiring a separately managed constraint store, really a brilliant idea IMHO.

The constraint networks built using attributed variables are terms which are typically cyclic (they’re graphs) and can get quite large. Each externally visible variable is a “handle” on a different part of the graph. All this works pretty well until you want to print the graph, or walk it, e.g., to make a copy or count the number of nodes, etc. To address these kinds of issues, attribute implementations are expected to support the attribute_goals//1 hook which, given an attributed variable, produces a list of goals which, if evaluated, would reproduce the attributed variable. This list is acyclic so much more amenable to processing.

In practice, the two common uses for this are 1) to support copy_term and friends, and 2) to produce terms that can be output in answers to a query (like the unification goals commonly seen).

My issue in clpBNR: For use with copy_term, you clearly need attribute_goals//1 to generate the full set of goals. However, for answer queries, this quickly results in information overload producing information related to variables internal to the constraint network (which is of little or no practical use to the average user), as well as constraint goals expressed as internal primitives rather than the original expression in source. A simple example:

?- X::real, {2*X**3-15*X**2+26*X+7==0}.
X::real(-4.820793239429888, 8.348881820487701),
(_A::real(-224.07092733268024, 1163.898039012228), {_A==2*_B, _A==_C+_D}),
(_B::real(-112.03546366634012, 581.949019506114), {_B==X**3}),
(_C::real(-224.07092733268024, 118.34062422517708), {-7==_C+_E}),
(_D::real(0, 1045.557414787051), {_D==15*_F}),
(_F::real(0, 69.70382765247005), {_F==X**2}),
(_E::real(-125.34062422517708, 217.07092733268024), {_E==26*X}).

?- 

Now imagine what the output would be for the full specification of a system of many such equations. All this may be of some interest to those supporting clpBNR, and is certainly required to support copy_term, but the user is probably only interested in the value of X (i.e., the variable in the query):

X::real(-4.820793239429888, 8.348881820487701)

Note that clpfd has a similar issue:

?- X in -1000..1000, 2*X^3-15*X^2+26*X+7 #= 0.
X in -1000..1000,
26*X#=_A,
X^2#=_B,
X^3#=_C,
_C in -13003..7512996,
2*_C#=_D,
_D in -26006..15025992,
_E+_F#=_D,
_B in 0..1000000,
15*_B#=_F,
_F in 0..15000000,
_E in -26007..25993,
_E+_A#= -7,
_A in -26000..26000.

?- 

clpfd users probably don’t notice it as much since much of the time clpfd domains narrow to a point, e.g., by labelling, so the attributes disappear. That essentially can’t happen with clpBNR reals with floating point bounds due to outward rounding (unless something else in the query actually unifies the variable with a numeric value).

To alleviate the problem, clpBNR uses a custom flag (clpBNR_verbose) to optionally provide an abbreviated form of answer output from attribute_goals//1. When set to false:

?- X::real,{2*X**3-15*X**2+26*X+7==0}.
X::real(-4.820793239429888, 8.348881820487701).

?- 

Using the expand_answer/2 hook, variables in the query can be further annotated such that attribute_goals can remove all the constraint goals and any “invisible” variables. But this requires that attribute_goals can detect whether it’s being used in “answer” mode or “copy” mode via some minimal state information. This is fairly easily “hacked” using a global variable when the query and the answer are processed in the same thread. It’s also not an issue when there is no answer generated, e.g., when used by the graphical debugger in the xpce thread. However, in the SWISH environment, the query execution and the answer generation are done in different threads. The only “hack” for this scenario is use the database, but that’s truly global, so not really useful in a server environment. (Global state affects every query.)

A solution: One possibility is to extend the existing attribute_goals hook with an additional argument which can be used to distinguish between “answer” mode and “copy” mode, default is “copy”. A slight variation is to provide an additional attribute_answer_goals\\1 hook (name TBD) which is specifically used to generate answers. If it fails, try attribute_goals\\1 next.

Any other ideas on how to crack this nut?

[jan]

Thanks. I think this is a nice and clean description of the problem. Changing attribute_goals//1 is not an option. It is as close we have gotten to establish a standard interface for constraints. Besides, its primary task is not not to generate nice toplevel output.

If I understand you correctly, in the normal toplevel you set some global variable. I suspect this variable provides attribute_goals//1 with the information as to which variables belong to the toplevel answer binding? You can’t do that in the SWISH case, but what you can do is to add another attribute to the variable that tells you this variable is part of the answer. You can use the same attribute as used for var_property/2 That approach should work for both SWISH and the normal toplevel.

Well, maybe I miss something. Another angle to think about is to have a hook for post-processing the constraints as part of the answer formulation? Possibly there can be a central solution (using a flag) to hide constraints in which no named toplevel variables appear?

[ridgeworks]

@jan

You can’t do that in the SWISH case, but what you can do is to add another attribute to the variable that tells you this variable is part of the answer.

It’s a little more subtle than that. From the “top level” perspective (application level query or SWISH), there’s a query phase followed an answer phase. During the query phase, any calls to attribute_goals are assumed to originate from copy_term (or I suppose a direct call via phrase), which requires a full representation, including primitive goals and internal variables. During the answer phase, a more abbreviated form is desirable in most cases.

So the problem boils down to how can attributes_goal distinguish between these two phases. Adding additional annotations to the top-level variables, e.g., a name, which can be done when the transition occurs between the two phases via ‘expand_answer’ or post_context, doesn’t really help because when given an “un-annotated” attributed variable, there’s no indication of the phase.

When the query and answer processing are done by the same thread, I can use a global variable to indicate the phase to global_attributes, but when they’re done in different threads, as in the SWISH case, the only way I can think of to do this is with a true global, i.e., the database, and that’s not server friendly.

@jan

Another angle to think about is to have a hook for post-processing the constraints as part of the answer formulation?

That’s the kind of thing I imagined doing using the suggested (new) attribute_answer_goals\\1 hook. It’s really attribute_goals with an implied phase=answer; current attribute_goals has an implied phase=query.

The only other thought I had was to somehow invoke attribute_goals in the same thread as the query, so the library could manage all this internally. This is how it’s currently done for the top level, but obviously doesn’t work in SWISH.

[jan]

@ridgeworks

the only way I can think of to do this is with a true global, i.e., the database, and that’s not server friendly.

Well, you can assert in the Pengine’s module. It is a temporary module anyway that is still present when doing the answer rendering (for the sake of operator declarations. etc.). But yes. cleanlyness is in the eyes of the beholder … And, I agree the problem is more general than your library and thus it deserves a cleaner solution.

@ridgeworks

\\

Would you mind using //? I thought only Windows users can’t tell the difference

Anyway, by vision was not to change attribute_goals//1. I propose to allow rewriting its output by a predicate that knows the (name of) the answer variables. Is something wrong with that?

@ridgeworks

but obviously doesn’t work in SWISH.

Yip. It could make sense to change that. That however would require changing the Pengine reply format to return a term for the bindings as well as the constraints. I have little clue about the consequences of such a design change. I’m afraid that constraints were not really part of the picture when Pengines were designed At least, it is fine to use constraints to solve goals, but not assumed one would communicate remaining constraints.

[ridgeworks]

@jan

Well, you can assert in the Pengine’s module.

And what is the “Pengine’s module” exactly? In a server context, does each query have its own Pengine module? If so, that would be a reasonable workaround pending a more general solution.

@jan

Would you mind using //?

Of course not, that was a typo; sorry.

@jan

Anyway, by vision was not to change attribute_goals//1.

I don’t think I was proposing changing attribute_goals//1. I was proposing adding an additional dcg predicate (attribute_answer_goals//1 or answer_goals//1) that is used for generating answer output. Sounds like we’re talking about the same thing, but maybe not.

@jan

Yip. It could make sense to change that. That however would require changing the Pengine reply format to return a term for the bindings as well as the constraints.

I suggested this as an alternative, having no clue as to how it might be implemented. Sounds like there are better alternatives on the table.

[jan]

@ridgeworks

And what is the “Pengine’s module” exactly? In a server context, does each query have its own Pengine module? If so, that would be a reasonable workaround pending a more general solution.

If a Pengine is created, it creates a temporary module for the code and a thread to run the code. The name of the module is normally a UUID, which is identical to the Pengine ID. Inside the Pengine, you can find the running Pengine using pengine_self/1.

Unfortunately, this module is not know when copy_term/3 is doing its work. I can try to find
a way around that.

@ridgeworks

Sounds like we’re talking about the same thing, but maybe not.

Not really. You are talking about two different attribute_goals//1-like predicates, I’m talking about post-processing the output of attribute_goals//1 (before handing it to the user).

@ridgeworks

Sounds like there are better alternatives on the table.

I don’t know (yet).

[ridgeworks]

I’ve written a “hack” which uses annotations in attributes via the hooks. It relies on the fact that in the answer phase, the initial call to attribute_goals//1 will be for a “named” (i.e., annotated) variable. (In copy phase, there will be no annotations.) Seems like a dubious assumption but appears to work in practise for clpBNR attributed variables.

The answers for set_prolog_flag(clpBNR_verbose,false)) are now the same for both top level and swish queries. However, when the flag is set to true, the HTML representation of the answer is somewhat garbled. Here’s the verbose top level answer for ?- X::real,{X**4-4*X**3+4*X**2-4*X+3==0}.:

X::real(-1.509169756145379, 4.18727500493995),
(_A::real(-9.036679024581517, 13.749100019759801), {_A==_B+_C, _A== -3+_D}),
(_E::real(0, 307.4156258686075), {_E==X**4, _E==_B+_F}),
_B::real(-79.16976689256097, 13.749100019759801),
(_F::real(-13.749100019759801, 293.66652584884764), {_F==4*_G}),
(_G::real(-3.4372750049399503, 73.41663146221191), {_G==X**3}),
(_C::real(0, 70.13308786797946), {_C==4*_H}),
(_H::real(0, 17.533271966994864), {_H==X**2}),
(_D::real(-6.036679024581516, 16.7491000197598), {_D==4*X}).

If I cut-and-paste the answer from SWISH, flatten it to remove unwanted inserted newlines, and comment it with variable names corresponding to the top level answer:

X::real(-1.509169756145379,4.18727500493995),
(_3800::real(0,307.4156258686075),{_1358==X**4,_1358==_1404+_1406}),  % _E
_1404::real(-79.16976689256097,13.749100019759801),                   % _B
(_1434::real(-9.036679024581517,13.749100019759801),{_1434==_1404+_1476,_1434 == -3+_1488}), % _A
(_1488::real(-6.036679024581516,16.7491000197598),{_1360==4*X}),      % _D
(_1358::real(0,70.13308786797946),{_1358==4*_1434}),                  % _C
(_1434::real(0,17.533271966994864),{_1434==X**2}),                    % _H
(_1356::real(-13.749100019759801,293.66652584884764),{_1356==4*_1396}), % _F
(_1396::real(-3.4372750049399503,73.41663146221191),{_1396==X**3}     % _G

Examples of issues:

1. Variable _E has one variable name in domain (_3800) and a different one in the goals (_1358)
2. Variable _1434 is used for both _A and _H.

Also, it’s hard to capture this, but the HTML formatting is wonky - the constraint sequence in {} is supposed to be visible inside the parenthesized definition as in the text above. But in several cases, the expression just ends with “,),” and the constraint goal appears in the following line, so it looks like this:

(_1356::real(-13.749100019759801,293.66652584884764),),
{_1356==4*_1396}

I’m willing to try to figure out how to fix this, but I’m not sure where in the SWISH code this functionality is implemented.

[ridgeworks]

Made some progress on the layout issue. To reproduce the problem on a small query, try this in SWISH:

X=(12345678901234567890123456789012345678901234567890,a).

So an expression in braces that exceeds some width limit, prints the closing ‘)’ on the same line as the large integer, and the atom ‘a’ appears on the line below it. This is caused by the html span element with
class="pl-op-seq pl-adaptive pl-level-0 vertical"

Note the “vertical”. This is added in term.js@line 150. If I comment out lines150-151 which set the class to include “vertical” (el.addClass("vertical")), the layout is fine. “vertical” apparently means that the contained elements are vertically stacked but the enclosing parentheses remain on the first line.

The comment on the function includes the text:

… As a result we must forward vertical to the adaptive node because this is not propagating.

so maybe this is the issue. In any case, some kind of fix seems necessary; either don’t set the class to include “vertical” or arrange that the closing parenthesis is placed correctly.

[ridgeworks]

Simple test case for bad variable names in SWISH using clpfd. Top level query/answer:

?- X^2-2*X+1#=0.
X in 1..sup,
2*X#=_A,
X^2#=_B,
_B in 1..sup,
-1+_A#=_B,
_A in 2..sup.

?-

Same query, but SWISH answer (with comments indicating problem):

X in 1..sup,
2*X#=_2804,        % _A
X^2#=_2828,        % _B
_2804 in 2..sup,   % _A in 2..sup (OK)
-1+_906#=_918,     % _906 \== _A, #918 \==  _B  (??)
_918 in 1..sup     % _918 \== _B   (??)   

[ridgeworks]

@ridgeworks

In any case, some kind of fix seems necessary; either don’t set the class to include “vertical” or arrange that the closing parenthesis is placed correctly.

I’m not a CSS expert, but probably the right place to fix this is in term.css. Currently style .pl-adaptive.vertical inherits the value of the CSS display property from pl-adaptive (which is currently inline-block); it should probably be set to block for vertical display.

So at term.css@79:

.pl-adaptive.vertical
{ vertical-align: top;
  display: block;  /* Add */
}

[jan]

@ridgeworks

Made some progress on the layout issue. To reproduce the problem on a small query, try this in SWISH:

X=(12345678901234567890123456789012345678901234567890,a).

This is not good. Needs some rethinking. As you say later changing inline-block to block is probably not the solution. These things can be nested. If I recall well, you can define something similar to portray/1 for the HTML renderer that allows you to control the output. That would apply to all SWISH applications though, so it is probably not what we want unless you want to control a very clpBNR specific term.

@ridgeworks

Same query, but SWISH answer (with comments indicating problem):

X in 1..sup,
2*X#=_2804,        % _A
X^2#=_2828,        % _B
_2804 in 2..sup,   % _A in 2..sup (OK)
-1+_906#=_918,     % _906 \== _A, #918 \==  _B  (??)
_918 in 1..sup     % _918 \== _B   (??)   

I think what we are dealing with is the fact that, unlike the toplevel, we currently do not give names to all variables before printing in SWISH. As a result, the same variable can have different identities (_NNN numbers) in different parts of the answer.

[ridgeworks]

@jan

As you say later changing inline-block to block is probably not the solution. These things can be nested.

Nesting doesn’t seem to be a problem, at least on the examples I tried. In some cases it produces stretchy parentheses, but it’s all generally better than the status quo. I also note that many other vertical classes (e.g., for compunds, lists and dictionaries) seem to use display: block;.

Do you have a specific test case in mind?

@jan

we currently do not give names to all variables before printing in SWISH. As a result, the same variable can have different identities (_NNN numbers) in different parts of the answer.

How, and where, is this done in SWISH? It appears that at least some of the HTML is generated in module term-html in package http.

[jan]

@ridgeworks

Do you have a specific test case in mind?

I did look at the code and found some issues. For local usage, please pull swish. You can see how it now works at https://swish2.swi-prolog.org. This fixes the initial layout, switching using the menu and placing the answer right of the variable rather than under it. Vertical operator sequences do use large parenthesis. Note that such a thing may appear inside a horizontal element, so normal reading order does not always apply when using vertical layout. The term is meant to be easy to interpret by the user. (sub)terms and answer variables have a menu that allows copying the term such that it can be pasted as value Prolog.

@ridgeworks

How, and where, is this done in SWISH? It appears that at least some of the HTML is generated in module term-html in package http.

It is done in pengines_io.pl from the Prolog libs. I have changed the reuse of the Prolog toplevel there, so this should work find after updating Prolog. That too has been done at the swish2 server.

Let me know whether these issues should be considered resolved now.

[ridgeworks]

@jan

Let me know whether these issues should be considered resolved now.

Layout looks fine. Variable naming is fixed for clpfd on swish2. My local server config still has old inconsistent numerical variables names, so I assume I’m missing something. (Can’t test clpBNR on swish2.)

I’ll do some further tests to see if anything else shakes out.

[jan]

@ridgeworks

My local server config still has old inconsistent numerical variables names, so I assume I’m missing something

You need to update Prolog to the current git for that.

[ridgeworks]

@jan

You need to update Prolog to the current git for that.

Done. So I think all the reported SWISH answer issues have been resolved (acceptable layout and consistent named variables). The SWISH answers aren’t identical to the toplevel (probably an ordering issue) but I believe they’re semantically equivalent - that’s good enough for me.

But I do have a new one that is clpBNR specific. In clpBNR non-verbose mode, I would like to output ranges of narrow intervals in ellipsis format, so X:: 1.000000000000000... rather than X::real(0.9999999999999999, 1.0000000000000002). This is currently done using the user:portray/1 hook which seems to be honoured in toplevel but not in SWISH, which outputs it as a compound term X :: ...(" 1.000000000000000").

I’m not thrilled about having to use the portray hook, but I haven’t found another way to do this (with answer value including necessary trailing digits and ‘…’ , unquoted).

I do know that this violates the principle of executable answers but it’s so much cleaner from a UI perspective that it’s worth this downside, IMO. (Can use verbose mode for outputting executable version, which also provides constraint expressions.)

[jan]

@ridgeworks

X:: 1.000000000000000...

The way to do this is to define the multifile hook term_html:portray//2.

That is not so nice as it is global. Cleaner is to define a SWISH render library. You find the examples in lib/render in SWISH. It allows for representing any Prolog term with a bit of HTML+CSS+JS. I guess you can figure out how it works from the examples.

Note that it is up to you to ensure the renderers are safe for the server as well as the user (though it is unlikely you can affect the user here).

[ridgeworks]

@jan

The way to do this is to define the multifile hook term_html:portray//2.

This was the low hanging fruit and all I needed. I actually think global is appropriate here, but I reduced the global footprint of the portrayed term to 'clpBNR...'(+String). It’s purely an answer output format, so if there’s a confilict with anybody else’s usage, I’d like to know about it.

No more UI issues from me so I’ll package all this up along with earlier suggestions into a new release.

[jan]

@ridgeworks

'clpBNR...'(+String).

That seems acceptable to me. Even better if you turn it into '$clpBNR'(Term), as $-terms are defined to be reserved. Does the next version also get rid of using the tracer hook? Looking forward to add this to SWISH

[ridgeworks]

@jan

Even better if you turn it into '$clpBNR'(Term), as $-terms are defined to be reserved.

Did not know that - will make this change.

@jan

Does the next version also get rid of using the tracer hook?

Yes - tracing now uses predicate wrapping. It’s global but tracing isn’t supported anyway (sandbox violation). (If thread-local debug topics are ever supported, I may have to revisit this.)