Dynamic predicates and physical memory in use

Help!
#1

I have got a problem with the physical memory in use by SWI prolog when asserting and then retracting clauses. In a nutshell, when I retract asserted clauses, the physical memory in use does not decrease. This is a problem when asserting and retracting a large number of clauses and the program is not meant to terminate (in an online setting using a server for instance).

Here an example:

:- dynamic myRelation/2.

test(N) :-
    retractall(myRelation(_, _)),
    loopAssert(N),
    loopFind(N),
    retractall(myRelation(_, _)), !.

loopAssert(0).
loopAssert(N) :-
    get_time(Time),
    asserta(myRelation(N, Time)),
    M is N-1,
    loopAssert(M).

loopFind(0).
loopFind(N) :-
    myRelation(N, _Time),
    M is N-1,
    loopFind(M).

If I query “test(10000000)”, then I can see that at the termination of the program the physical memory in use does not decrease. By opening windows 10 Task Manager, I can see that 2,076 MB of memory remains in use at termination, and it stays like this. May I kindly as: why? And is there any workaround?

I’m using: SWI-Prolog version 8.2.3.

test_memory
test_memory1241×618 21.1 KB

#2

What happens if you call garbage_collect_clauses/0 ?

You can also see what SWI-Prolog thinks has been used with statistics/0 or statistics/2 (e.g., cgc_gained, heapused).

#3

If I call garbage_collect_clauses/0, then the problem is solved :slight_smile:

Thank you so much Peter!

Edit: Actually, it works “sometimes”! Sometimes the memory in use decreases (back to normal I guess), sometimes it does not. That is very mysterious.

Interestingly, if I make another query such as “allo.” after the first query then the clause is said to be unknown of course, but the thread/program seems to “wake up” in some sense and the physical memory in use decreases. Is there any explanation for that?

#4

O’Keefe has a section on memory use an deallocation in the crafts book and distinguishes between structure sharing and structure copying prolog systems.

In structure sharing systems deallocation is more complex and sometimes unclear.

Which, kind of system is swi-prolog?

Dan

#5

Well, it calls the very same garbage collector automatically at times. This is based on memory statistics and runtime counting of erased clauses that slow down clause lookup. If either is above some threshold a signal is sent to the gc thread to perform clause GC. Something similar happens for atoms. The total result in OS terms also depends a lot on the malloc() implementation used. See old posts on tcmalloc on this forum. There are, AFAIK, no permanent leaks related to the dynamic database.

#6

Thank you for the explanation, and the tcmalloc pointer.

I understand that there are no simple solutions here, or are there any workarounds to this problem? What would be the simplest thing to do? What would you recommend?

#7

Typically, everything should just work. On some workloads you may need a better allocator than the default OS one. In some very rare cases you may wish to call garbage_collect_clauses/0 to collect more aggressively than the default.

Note that very intensive use of assert/retract is typically not a good idea. In most cases there are better alternatives.