Is there a way to allow unbound prolog variables to escape into an FFI context and later be reintroduced? Currently this works downwards, due to the way the stack works, but it doesn’t work upward. It seems term refs can’t be used as identifiers because they don’t actually point to or identify the variable, but to a local shadow stack slot instead.
A large pool of term refs could be made at the top level of the program entry point, but there doesn’t seem to be a way to reference count or garbage collection them.
Could you elaborate a bit more what you are after? It seems you pretty well understand how term references work. I don’t understand what “escape into an FFI context and later be reintroduced” means though.
Preallocating term references is also done by the system itself to keep exception terms, the constraint wakeup list and some similar Prolog terms at well known positions. You can only do this with a fixed number of them as you observed.
I’m trying to support prolog variables escaping into Chez Scheme’s runtime, and then being introduced back into SWI at later points that don’t necessarily follow an ordered stack discipline. I suppose what I need is a way to promote a variable to the global SWI heap, then obtain a reference to the underlying object of a term_t, and then pin the object to prevent garbage collection by SWI, and then finally add a GC finalizer to the object in Chez so that it gets unpinned in SWI when the object goes out of scope in Chez.
Does the global interface provide anything that a pre-allocated top-level term ref doesn’t?
Some ways I can think of to solve the problem:
If I were to preallocate a fixed amount of term refs at the top level, and assuming SWI doesn’t use anything like a cactus stack, this pool can’t grow at all correct? This isn’t so great because the user has to guess how many refs will be needed.
A workaround to this could be to preallocate a single top-level term ref ‘root’, and fill it with a compound term with a large amount of arg slots. Then the first slot of the compound can be used to link to a second page, thereby creating a linked list of large arrays. Then all the FFI conversion routines could just reference these slots with a two-dimensional integer index when converting prolog variables back and forth across Chez/SWI. This effectively provides a stable indirect pointer to the underlying data, bypassing the local term ref stack, and forces SWI’s GC to retain the object.
Since it is only really Chez that needs these stable pointers to prolog variables, once they go out of scope in Chez the arg slot in SWI can be reset, and the SWI GC will release the underlying object at a later point.
It would be much easier if the SWI runtime supported pinning objects directly though, instead of having to use a large table to maintain the integer<->object mapping.
So, you are looking for a way to make Prolog terms accessible through the FFI without the scoping limitations imposed by term_t handles? How is this supposed to cooperate with Prolog backtracking?
An indirect trick using a compound may work. Note that you can also use a variation on a trick I learned for dynamic arrays: have a small set of root pointers, the first pointing at an array of 1, second 2, 3rd 4, etc. You’d still need some way to reset an argument in an compound to a variable, but that is easy.
In fact, this may be a better implementation than what is now used for global variables which is a hash table that contains references into the global stack. The problem with that is that these pointers live outside the Prolog stacks and for garbage collection we (temporary) have to allocate them onto the stacks.
Having a pool of global terms is certainly also doable. We also already have code to shift the stacks. It should be possible to reuse most of that and create a hole near the base of the stack.
Might be better to discuss this in a telco? If you think so too, drop me a personal message to make an appointment.