Memory management and garbage_collect_clauses/0

General
1

Hi,

when implementing memoization via assert/retract to solve a problem I noticed that SWI-Prolog does not always reclaim the memory of retracted clauses right away. According to the manual, one can then use garbage_collect_clauses/0, but this does not seem to work (see minimal example below) when invoked from a rule. Running garbage_collect_clauses/0 from the query window works, but seems cumbersome if one is interested in solving many problems after each other.

:- dynamic t/2.

fill(N) :- forall(between(1, N, I), (F is random_float, assert(t(I,F)))).
	

test :-
	fill(1000000),
	retractall(t(_,_)),
	garbage_collect_clauses.
3

Thanks for the pointer! I my scenario I run several instances of the problem after each other, and quickly run out of (allowed) memory. I thought this could be fixed my manually using garbage_collect_clauses/0 but it doesn’t seem to do the trick in my case.

4

Your problem needs more memory in total and the clauses are not the issue?

You are not cleaning something?

You are using up something that is not garbage collected?

There is a bug somewhere, maybe on your platform only?

I tested your code on a laptop (Some cheap 6-7 year old Intel Lenovo with Linux on it), and it “just works”. For me calling garbage_collect_clauses had no obvious effect on RES memory reported by top. This is how I tested, leaving out explicit garbage collection:

$ cat grbg.pl 
:- dynamic t/2.

fill(N) :-
    forall(between(1, N, I),
        (   F is random_float,
            assert(t(I,F))
        )).

test :-
	fill(1 000 000),
	retractall(t(_,_)).

$ swipl
Welcome to SWI-Prolog (threaded, 64 bits, version 9.3.13-18-g9ba133d9f)
SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software.
Please run ?- license. for legal details.

For online help and background, visit https://www.swi-prolog.org
For built-in help, use ?- help(Topic). or ?- apropos(Word).

?- [grbg].
true.

?- forall(between(1, 100, X), (test, writeln(X))).
1
2
<snip>
99
100
true.

swipl memory consumption grows very slowly until it reaches about 1.2 GB RES memory as reported by top, and then it goes down again.

5

For the above code I get something similar (but peaking at a slightly higher memory usage). Using garbage_collect_clauses/0 in test/0 does not seem to affect the memory usage at. The reason for why this is actually a problem in my case is that I solve a batch of problems where I in every iteration come very close to a memory limit, so it would be good to force garbage collection of the retracted clauses before starting working on the next problem.

However, I think that I managed to identify the problem: by using set_prolog_gc_thread(false) it’s possible to force garbage collection to run in the current thread. This seems to completely resolve the issues.

7

Yes, for single threaded applications this may improve the situation, in particular if you call GC after completing a batch where it is guaranteed there are no accesses to the predicate. Otherwise GC runs asynchronously and is likely not able to remove all clauses immediately. Note that set_prolog_gc_thread(false) still allows for full multi-threading, causing one of the normal threads to deal with clause GC at some point. That may not happen at the time you’d like and it may not happen on the thread you prefer.

Ultimately the system should probably implement different memory management for “thread local” predicates. As no other threads are involved,reclaiming all memory immediately can be done. As is, all clause reclaiming goes through the same lock free GC mechanism that needs to consider other threads may be using the predicate.