clpBNR flag management

Discussion
1

Just to illustrate the origin of the problem: my code above stays in a file named three_cubes.pl. Then

Welcome to SWI-Prolog (threaded, 64 bits, version 8.5.17)
SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software.
Please run ?- license. for legal details.

For online help and background, visit https://www.swi-prolog.org
For built-in help, use ?- help(Topic). or ?- apropos(Word).

?- edit(three_cubes).
true.

ok so far, the next ERROR is expected…


?- p(X,Y,Z).
ERROR: Unknown procedure: p/3 (DWIM could not correct goal)

so, switch to the editor window, and compile what is inside: I’m used to Ctrl-C Ctrl-B… then go back to the REPL, where ?- p(X,Y,Z). causes the error.

?- 
% *** clpBNR v0.10.1 ***.
?- p(X,Y,Z).
ERROR: Arithmetic: evaluation error: `float_overflow'
ERROR: In:
ERROR:   [22] clpBNR:lower_bound_val_(real,-1.0Inf,_4132)
ERROR:   [21] clpBNR:int_decl_(real,(-1.0Inf,1.0Inf),_4166) at c:/users/carlo/appdata/local/swi-prolog/pack/clpbnr/prolog/clpbnr.pl:430
ERROR:   [18] clpBNR:build_args_([_4214,...|...],[_4226|_4228],[real,real|...],_4244/_4246,_4210) at c:/users/carlo/appdata/local/swi-prolog/pack/clpbnr/prolog/clpbnr.pl:638
ERROR:   [17] clpBNR:build_(_4294**3+_4300**3,_4278,real,_4306/_4308,_4284) at c:/users/carlo/appdata/local/swi-prolog/pack/clpbnr/prolog/clpbnr.pl:633
ERROR:   [16] clpBNR:build_args_([... + ...,...],[_4368|_4370],[real,real],_4386/_4388,_4346) at c:/users/carlo/appdata/local/swi-prolog/pack/clpbnr/prolog/clpbnr.pl:638
ERROR:   [14] clpBNR:build_(... + ... == ... + -9,1,boolean,_4448/_4450,_4426) at c:/users/carlo/appdata/local/swi-prolog/pack/clpbnr/prolog/clpbnr.pl:633
ERROR:   [13] clpBNR:buildConstraint_(... + ... == ... + -9,_4506/_4508,_4484) at c:/users/carlo/appdata/local/swi-prolog/pack/clpbnr/prolog/clpbnr.pl:594
ERROR:   [11] clpBNR:{... + 9==_4554**3} at c:/users/carlo/appdata/local/swi-prolog/pack/clpbnr/prolog/clpbnr.pl:561
ERROR:   [10] three_cubes:p(_4586,_4588,_4590) at c:/users/carlo/documents/prolog/three_cubes.pl:10
ERROR:    [9] toplevel_call(user:user: ...) at c:/program files/swipl/boot/toplevel.pl:1169
ERROR: 
ERROR: Note: some frames are missing due to last-call optimization.
ERROR: Re-run your program in debug mode (:- debug.) to get more detail.
   Exception: (13) clpBNR:buildConstraint_(_1202{integer(0,239)}**3+_1124{integer(0,239)}**3==_1280{integer(0,239)}**3+ -9, _1350/_1350, _1354) ?

As we have seens, it goes away following the appropriate interface (?- use_module(...).)

2

I was wondering what could cause this, but this explains:

101 ?- thread_create(use_module(three_cubes), _, []).
true.

102 ?- % *** clpBNR v0.9.10alpha ***.
102 ?- 
|    p(X,Y,Z).
ERROR: Arithmetic: evaluation error: `float_overflow'
ERROR: In:
ERROR:   [22] clpBNR:lower_bound_val_(real,-1.0Inf,_9286)

I suspect that the problem is that clpBNR sets some Prolog flags that affect rational number and float behavior. As changing Prolog flags is thread-specific this does not work. Possibly clpBNR should manage these flags at runtime?

3

This was a bit of a surprise to me; I had always thought that Prolog environment flags were global but I now see:

A new thread (see section 10) copies all flags from the thread that created the new thread (its parent ).

Currently the necessary Prolog flag values are set at module initialization time:

set_prolog_flags :-
	set_prolog_flag(prefer_rationals, true),           % enable rational arithmetic
	set_prolog_flag(max_rational_size, 16),            % rational size in bytes before ..
	set_prolog_flag(max_rational_size_action, float),  % conversion to float

	set_prolog_flag(float_overflow,infinity),          % enable IEEE continuation values
	set_prolog_flag(float_zero_div,infinity),
	set_prolog_flag(float_undefined,nan).

So a problem can arise if a thread is created from a thread who is not a descendant of the one that loaded clpBNR. It seems like some sort of per-thread module initialization is needed here. A similar problem with global variables is managed using the undefined_global_variable exception. Any recommendations of how this per-thread flag management should be done?

Failing a general module level solution, I guess another option is to check the flag status whenever a public predicate which can create an interval is called - manageable but not very elegant.

Apologies for hijacking this thread (but thanks to @CapelliC for raising the issue).

4

The ideal is probably to fetch their current state and use setup_call_cleanup/3 to change and restore them around the clpBNR evaluation. That would also stop interaction with other code in the program that may have expectations about these flags. They may of course have too much impact on performance.

Flags are typically not module scoped. Some flags are, typically those that affect syntax. Possibly the flags that affect the semantics of arithmetic should be module scoped too. Changing this is not trivial though :frowning:

Moved to a new thread.

5

I’d like to avoid using setup_call_cleanup/3 for efficiency reasons. So a question:

When I start a separate thread and then use it to load clpBNR, it creates a few “private” clpBNR flags using create_prolog_flag/3 in that thread. Prior to loading, these flags (obviously) are undefined in the main thread. However after loading they are defined in the main thread, i.e., current_prolog_flag/2 now succeeds for these flags. So it appears that the created flags are “copied” to the main thread. Is this expected behaviour?

6

No. I can indeed confirm that this happens. This was not expected though. I guess this should be fixed unless we have a good reason to go with the current behavior. The docs say that the flag table is copied. The actual implementation uses a copy-on-write approach.

7

IMO the current behaviour is pretty confusing, i.e., the flags are copied at some future (undefined) time. In the meantime what are the values of the flags yet to be copied?

Hard for me to understand when that be the desired effect; the behaviour as documented seems much better.

8

I don’t think that is what is happening. I’ll check tomorrow what exactly is happening. And yes, I agree the documented behavior is cleaner. It could be that changing this raises issues though. Ideally it should not matter which threads loads some Prolog code. If the file introduces new Prolog flags though, it does matter if we “fix” this. Changing just about any Prolog flag is troublesome :frowning:

9

I’ll solve the clpBNR issue without requiring any changes to the current implementation. Current strategy is to use the undefined global var handler to trigger flag configuration. Checking the globals existence will cause this to occur once on every thread using clpBNR. Doesn’t guard against inadvertent or malicious changing of the flags, but that’s part of the API “contract”.

10

Should work. Would be nice if there was a cleaner solution :frowning:

11

From a programming perspective, I think a cleaner solution would be to allow modules to define a hook which gets called at thread initialization time. That’s essentially what I’m doing with the undefined global but I have to trigger it myself by accessing the global on each relevant predicate call. (The initialization directive can also call the hook when the module is loaded.)

12

That is possible, use the thread_initialization/1 directive. It is still not clean though. You are doing stuff the target thread may not need. The worst part IMO is that Prolog flags affect the arithmetic in all code while it should only affect the code used in your module.

13

This would be helpful, but I tried it as a directive:

init_clpBNR_thread :-
	print_message(informational, clpBNR(thread_init)).
	
prolog:message(clpBNR(thread_init)) -->
	[ '*** clpBNR thread init ***'-[] ].
	
:- thread_initialization(init_clpBNR_thread).

And it doesn’t seem to be called. Is there an example of usage somewhere that I can reference?

That’s a much bigger fish but aren’t there lots of flags with this issue, e.g., occurs_check, optimise, last_call_optimisation, encoding, iso, determinism_error…

14

Sorry, my problem. Too many “experimental” versions kicking around.

15

Pretty close. The one use case that is questionable is when a thread loads the module which a pre-existing thread then calls. That will generate an exception because the flags have not been configured properly in the pre-existing thread. (The module has been loaded and is visible to the thread, but hasn’t run the thread initialization code.)

However, if the pre-existing thread re-loads the module before making the call, it seems to work correctly. But it must not depend on things like current_module/1 to conditionally load it. Alternatively, it can explicitly call the thread initialization (clpBNR:init_clpBNR_thread/0) before calling any of the other predicates.

Although it’s not quite as robust, I think I prefer the thread_initialization/1 approach. The affected code is more contained and it seems to me that the problematic case is a bit of an outlier - a thread starting another thread to load a module that the first thread then uses. And if that case ever arises, there is a workaround.

But maybe there’s some other pitfall I’m not seeing.

16

That is exactly the situation that created this thread: compiling from the development tools which normally run in their own thread. This also applies to several other development tools such as the Eclipse based PDT and @oskardrums’s new sweep mode for Emacs.

Note that a use_module/1 is no guarantee as the module may already have been loaded, turning the use_module/1 into a no-op. Basically there is no way to modify flags globally and the only ways I can see is to verify and adjust the flags on demand, (cleaner) save/modify/restore the flags for the involved code or go for serious changes and make it possible to compile code in such a way that the arithmetic behavior is as required regardless of these flags.

For the latter we could envision a data structure that captures the math mode. This data could be attached to the module or even to calls to the arithmetic evaluation. Such a change is rather involved though. It should also consider cases where we want to globally affect arithmetic (if these exist).

17

OK, so can’t depend on thread_initialization/1 for this in the near term. I’ll stick with the global var approach for now.

Explicitly saving/restoring a TBD math mode is also a reasonable option if it can be done fairly efficiently, perhaps by adding a builtin so it could be done in a single step (something similar to working_directory/2):

math_mode(Old,New),
.... do some arithmetic
math_mode(_,Old),

In general, clpBNR requires no errors to be generated, so use IEEE continuation values and prefer rationals subject to the tripwire. So the math mode should allow something equivalent to:

	set_prolog_flag(prefer_rationals, true),           % enable rational arithmetic
	(current_prolog_flag(max_rational_size,_)
	 -> true                                           % already defined, leave as is
	 ;  set_prolog_flag(max_rational_size, 16)         % rational size in bytes before ..
	),
	set_prolog_flag(max_rational_size_action, float),  % conversion to float

	set_prolog_flag(float_overflow,infinity),          % enable IEEE continuation values
	set_prolog_flag(float_zero_div,infinity),
	set_prolog_flag(float_undefined,nan),

I think something like this would allow clpBNR to execute independent of global flags which would be a welcome improvement.

18

A couple of ideas, maybe they are useful (maybe not):

  • SWI-Prolog could provide a enable_flag_broadcast(Flags) predicate
  • enable_flag_broadcast/1 would enable SWI-Prolog to call broadcast_request/1 when any of the registered flags change
  • the interested module(s) would use listen/3 to take appropriate action and notify SWI-Prolog about what to do next

If there is worry about too many events flying around this could be used:

  • provide a ‘different_from’ option to enable_flag_broadcast/2 so you could do
    enable_flag_broadcast(Flags,[different_from([flag1=value1,flag2=value2,...]) in this case
    the broadcast_request/1 will be done only if the new flag value is different from the ones
    specified.
  • some systems make a hash of groups of configuration flags (or capabilities) and use this to
    make sure the flags are all consistent.
19

Just to see what would happen, I tried the most obvious thing. This results in the timing below, or 2us per call. This is obviously far too expensive for putting around a simple is/2, but might be fine around some interval solving code.

?- time(forall(between(1, 1 000 000, _), clpBNR_call(true))).
% 28,000,001 inferences, 1.852 CPU in 1.852 seconds (100% CPU, 15118657 Lips)
program 
clpBNR_flag_state(state(PreferRats,
                        MaxRatSize,
                        MaxRatSizeAction,
                        FloatOverflow,
                        FloatDivZero,
                        FloatUndefined)) :-
    current_prolog_flag(prefer_rationals, PreferRats),
    (   current_prolog_flag(max_rational_size, MaxRatSize)
    ->  true
    ;   MaxRatSize = 100 000 000
    ),
    current_prolog_flag(max_rational_size_action, MaxRatSizeAction),
    current_prolog_flag(float_overflow, FloatOverflow),
    current_prolog_flag(float_zero_div, FloatDivZero),
    current_prolog_flag(float_undefined, FloatUndefined).

clpBNR_restore_state(state(PreferRats,
                           MaxRatSize,
                           MaxRatSizeAction,
                           FloatOverflow,
                           FloatDivZero,
                           FloatUndefined)) :-
    set_prolog_flag(prefer_rationals, PreferRats),
    set_prolog_flag(max_rational_size, MaxRatSize),
    set_prolog_flag(max_rational_size_action, MaxRatSizeAction),
    set_prolog_flag(float_overflow, FloatOverflow),
    set_prolog_flag(float_zero_div, FloatDivZero),
    set_prolog_flag(float_undefined, FloatUndefined).

clpBNR_set_flags :-
    set_prolog_flag(prefer_rationals, true),
    (   current_prolog_flag(max_rational_size,_)
    ->  true
    ;   set_prolog_flag(max_rational_size, 16)
    ),
    set_prolog_flag(max_rational_size_action, float),
    set_prolog_flag(float_overflow,infinity),
    set_prolog_flag(float_zero_div, infinity),
    set_prolog_flag(float_undefined, nan).

clpBNR_set_flags(State) :-
    clpBNR_flag_state(State),
    clpBNR_set_flags.

clpBNR_call(Goal) :-
    setup_call_cleanup(
        clpBNR_set_flags(State),
        Goal,
        clpBNR_restore_state(State)).

Note that their doesn’t seem to be a way to lift the max_rational_size. Guess that should change, which would also get rid of the ugly if-then-else.

The problem is that flags are (except from a few) independent from modules, i.e., they are globally scoped settings. The read-only flags are of course fine. The others have global impact on various things, ranging from syntax to semantics of certain predicates. Some of this seems ok, e.g., setting resource limits. Others are problematic, such as affecting the behavior of arithmetic evaluation. A broadcast when the flag changes does not help (I think). Note that the broadcast library does not communicate between threads. You can use this, but that really seems asking for trouble :slight_smile:

thread_signal(Thread, set_prolog_flag(...)),
1 Like
20

On 8.5.16, MacOS Intel:

?- time(forall(between(1, 1 000 000, _), clpBNR_call(true))).
% 28,000,001 inferences, 8.025 CPU in 8.908 seconds (90% CPU, 3489295 Lips)
true.

which is too much IMO. Other than possible resource errors, a working version of clpBNR should never cause an exception, so I also tested with:

clpBNR_call_noclean(Goal) :- 
        clpBNR_set_flags(State),
        Goal,
        clpBNR_restore_state(State).

clpBNR_flags_mgmt :-
        clpBNR_set_flags(State),
        clpBNR_restore_state(State).

Results:

?- time(forall(between(1, 1 000 000, _), clpBNR_call_noclean(true))).
% 25,000,001 inferences, 2.728 CPU in 2.729 seconds (100% CPU, 9164610 Lips)
true.

?- time(forall(between(1, 1 000 000, _), clpBNR_flags_mgmt)).
% 24,000,001 inferences, 2.547 CPU in 2.548 seconds (100% CPU, 9421090 Lips)
true.

Obviously, I’d like to do better (spends most of it’s time traversing the Prolog-C boundary?) but this is borderline acceptable for my purposes. More testing on my part required.