# Foreign interface: BUF\_DISCARDABLE seems to be non-working **URL:** \ **Category:** General\ **Created:** [September 29, 2024, 3:23pm UTC](https://swi-prolog.discourse.group/t/foreign-interface-buf-discardable-seems-to-be-non-working/8505 "2024-09-29T15:23:16Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![AlexeyKosarchuk](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/alexeykosarchuk/32/4466_2.png) [@AlexeyKosarchuk](https://swi-prolog.discourse.group/u/AlexeyKosarchuk)\ **Post date:** [September 29, 2024, 3:23pm UTC](https://swi-prolog.discourse.group/t/foreign-interface-buf-discardable-seems-to-be-non-working/8505/1 "2024-09-29T15:23:16Z")
Stress-testing of my version of Lesta’s C# wrapper for Prolog foreign interface revealed somtheing strage. Repeated call to PL\_get\_wchars eventually produce an error: SWI-Prolog: [FATAL ERROR: at Sun Sep 29 18:13:27 2024 Too many stacked strings] First, I tried by wrapping calls into foreign frame fuctions as described here: [SWI-Prolog -- Discarding Data](https://www.swi-prolog.org/pldoc/man?section=foreign-discard-term-t) It didn’t work at all, which is strange by itself. Then I’ve tried to use BUF\_DISCARDABLE with PL\_get\_wchars, which also didn’t work. I looked through the code of PL\_get\_wchars and internal call to PL\_save\_text and failed to find codepath that uses BUF\_DISCARDABLE at all. It seems to check of BUF\_MALLOC and fallback to BUF\_STACK algorithm otherwise. Documentation for PL\_get\_chars contains very interestring framgent for other flag: **CVT\_WRITE** Convert any term that is not converted by any of the other flags using [write/1](https://www.swi-prolog.org/pldoc/man?predicate=write/1). If no `BUF_*` is provided, `BUF_STACK` is implied. But BUF\_DISCARDABLE is defined as 0x0, so providing it equals to not providing BUF\_\* at all. Very confusing. So, open questions are: 1. Why using and discarding frames did’t prevent strings stack leackage? 2. How to use BUF\_DISCARDABLE correctly with BUF\_DISCARDABLE? ---
**Author:** ![jan](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/jan/32/4_2.png) [@jan](https://swi-prolog.discourse.group/u/jan)\ **Post date:** [September 29, 2024, 4:23pm UTC](https://swi-prolog.discourse.group/t/foreign-interface-buf-discardable-seems-to-be-non-working/8505/2 "2024-09-29T16:23:32Z")
> [@AlexeyKosarchuk](#): > > - Why using and discarding frames did’t prevent strings stack leackage? > - How to use BUF\_DISCARDABLE correctly with BUF\_DISCARDABLE? Surely, the answer to the first is in [SWI-Prolog -- String buffering](https://www.swi-prolog.org/pldoc/man?section=foreign-strings). I’m not sure about the current state of `BUF_DISCARDABLE`. I think you can still use it and it may avoid copying in some cases, probably in particular getting the C text from a Prolog string if the representation matches. If there is no C string with matching representation, it will use `BUF_STACK`. That will run out if you do not take precautions. Older versions used a ring, but that could result in rather hard to track bugs. And no, the foreign frame interface only deals with `term_t` handles and backtracking. Guess the docs need some work ☹ ---
**Author:** ![peter.ludemann](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/peter.ludemann/32/48_2.png) [@peter.ludemann](https://swi-prolog.discourse.group/u/peter.ludemann)\ **Post date:** [September 30, 2024, 4:04am UTC](https://swi-prolog.discourse.group/t/foreign-interface-buf-discardable-seems-to-be-non-working/8505/3 "2024-09-30T04:04:04Z")
`BUF_DISCARDABLE` is defined as 0, so it’s the default, isn’t it? The C++ API uses `PL_get_chars()` (and wchars) in a few places, and I _think_ the test cases exercise that. Also, `PlTerm::get_nchars()` explicitly turns off `BUF_STACK|BUF_MALLOC|BUF_ALLOW_STACK` and it works AFAICT (but inside a string buffer). Do I need to revisit that code? ---
**Author:** ![jan](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/jan/32/4_2.png) [@jan](https://swi-prolog.discourse.group/u/jan)\ **Post date:** [September 30, 2024, 7:18am UTC](https://swi-prolog.discourse.group/t/foreign-interface-buf-discardable-seems-to-be-non-working/8505/4 "2024-09-30T07:18:57Z")
> [@peter.ludemann](#): > > Do I need to revisit that code? I think all should be fine as long as the code is between PL\_STRINGS\_MARK()/PL\_STRINGS\_RELEASE(). Strings will be stacked if a copy needs to be made to create a different representation. You only get the native representation if you ask for an ISO\_LATIN\_1 string from a string or atom or you ask for a wide string from a string or atom that happens to contain code points `>255`. In all other cases, the call will create a string for you and stores that on the string stack. The two above cases are the current situation, there is still a plan to replace the double representation by UTF-8, which would mean getting UTF-8 and ASCII are without copying. ---
**Author:** ![peter.ludemann](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/peter.ludemann/32/48_2.png) [@peter.ludemann](https://swi-prolog.discourse.group/u/peter.ludemann)\ **Post date:** [September 30, 2024, 3:45pm UTC](https://swi-prolog.discourse.group/t/foreign-interface-buf-discardable-seems-to-be-non-working/8505/5 "2024-09-30T15:45:30Z")
There appear to be some errors in the C++ API, at least with calls to `PL_get_chars()`. I’ll try to do a cleanup (and double-check the calls to `PL_get_nchars()` and `PL_get_wchars()`), but it might be a few weeks before I can do that. @AlexeyKosarchuk – I’m curious to see what the C# wrapper looks like, to see if any of its design decisions would improve the C++ wrapper … where can I see the C# wrapper? (The links at [SWI-Prolog interface to C# and F#](https://www.swi-prolog.org/contrib/CSharp.html) didn’t work for me) ---
**Author:** ![peter.ludemann](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/peter.ludemann/32/48_2.png) [@peter.ludemann](https://swi-prolog.discourse.group/u/peter.ludemann)\ **Post date:** [October 1, 2024, 6:26pm UTC](https://swi-prolog.discourse.group/t/foreign-interface-buf-discardable-seems-to-be-non-working/8505/6 "2024-10-01T18:26:38Z")
I did a search for `PL_*()` functions that update a `char*` or `wchar_t*` and found the following that are not documented as needing `PL_STRINGS_MARK` … can any of them generate a temporary copy? (If so, I’ll update the documentation and the C++ API) ```cpp PL_atom_mbchars(atom_t a, size_t *len, char **s, unsigned int flags) PL_get_atom_chars(term_t t, char **a) PL_get_atom_nchars(term_t t, size_t *len, char **a) PL_get_string(term_t t, char **s, size_t *len) // DEPRECATED PL_get_list_chars(term_t l, char **s, unsigned int flags) PL_get_list_nchars(term_t l, size_t *len, char **s, unsigned int flags) PL_get_file_name(term_t n, char **name, int flags) PL_get_file_nameW(term_t n, wchar_t **name, int flags) PL_cvt_i_string(term_t p, char **c) PL_cvt_i_codes(term_t p, char **c) ``` And here are the functions that are documented as requiring `PL_STRINGS_MARK`: ```cpp PL_get_chars(term_t t, char **s, unsigned int flags) PL_get_nchars(term_t t, size_t *len, char **s, unsigned int flags) PL_get_wchars(term_t l, size_t *length, pl_wchar_t **s, unsigned flags) ``` Note that the C++ API tries to avoid the need for `PL_STRINGS_MARK` (which it abstracts as the RAII class `PlStringBuffers`) by returning `std::string` rather than `char*`. ---
**Author:** ![AlexeyKosarchuk](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/alexeykosarchuk/32/4466_2.png) [@AlexeyKosarchuk](https://swi-prolog.discourse.group/u/AlexeyKosarchuk)\ **Post date:** [October 2, 2024, 9:31am UTC](https://swi-prolog.discourse.group/t/foreign-interface-buf-discardable-seems-to-be-non-working/8505/7 "2024-10-02T09:31:11Z")
Here: [Алексей Косарчук / swipl-cs-2 · GitLab](https://gitlab.com/alkid1/swipl-cs-2) Requires Visual Studio 22 & .NET SDK 6. Tested for 64 bit. It is my fork from old Lesta’s C# wrapper you’ve mentioned here. It basicly works, but I still have some leaks and (rarely) random crashes. I solved major string stack leakage by using BUF\_MALLOC and PL\_Free after unmarshaling strings, but somwhere strings are still stacked and under stress-testing I still manage to get “too many stacked strings” ---
**Author:** ![jan](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/jan/32/4_2.png) [@jan](https://swi-prolog.discourse.group/u/jan)\ **Post date:** [October 2, 2024, 12:27pm UTC](https://swi-prolog.discourse.group/t/foreign-interface-buf-discardable-seems-to-be-non-working/8505/8 "2024-10-02T12:27:58Z")
Good to hear we will have a functioning C# binding again 🙂 > [@AlexeyKosarchuk](#): > > I solved major string stack leakage by using BUF\_MALLOC and PL\_Free after unmarshaling strings, but somwhere strings are still stacked and under stress-testing I still manage to get “too many stacked strings” Possibly you found a bug. If you can reduce this to specific API calls, I’m happy to have a look. It is wise to use the stacked string API though as that avoids a lot of malloc()/free() calls. This saves time and memory fragmentation. Note that while the PL\_STRINGS\_MARK()/PL\_STRINGS\_RELEASE() API is C specific, but the underlying PL\_mark\_string\_buffers() and PL\_release\_string\_buffers\_from\_mark() are not. PL\_mark\_string\_buffers() is side-effect free. If you call it before and after some operation, its state should not change. You can also use ``` :- set_prolog_flag(string_stack_tripwire, 10). ``` to get errors if the stack accumulates only 10 stacks. 10 is enough for the built-in predicates. That should allow for reproducing with less stress 🙂 ---
**Author:** ![AlexeyKosarchuk](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/alexeykosarchuk/32/4466_2.png) [@AlexeyKosarchuk](https://swi-prolog.discourse.group/u/AlexeyKosarchuk)\ **Post date:** [October 2, 2024, 4:16pm UTC](https://swi-prolog.discourse.group/t/foreign-interface-buf-discardable-seems-to-be-non-working/8505/9 "2024-10-02T16:16:10Z")
Thanks, I will investigate further and try to distill reproduction to minimal program. ---
**Author:** ![AlexeyKosarchuk](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/alexeykosarchuk/32/4466_2.png) [@AlexeyKosarchuk](https://swi-prolog.discourse.group/u/AlexeyKosarchuk)\ **Post date:** [October 2, 2024, 5:32pm UTC](https://swi-prolog.discourse.group/t/foreign-interface-buf-discardable-seems-to-be-non-working/8505/10 "2024-10-02T17:32:22Z")
Okay. it seems that careful usage of mark/release strings solved the problem. The catch was that even if called with BUF\_MALLOC PL\_get\_wchars allocated strings on stack. Possibly it was some internal intermediate results. PL\_atom\_wchars also requires usage of mark/release. With these fixes and careful usage of frames my code runs indefinitely with no increase of memory footprint and prolog errors. ---
**Author:** ![peter.ludemann](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/peter.ludemann/32/48_2.png) [@peter.ludemann](https://swi-prolog.discourse.group/u/peter.ludemann)\ **Post date:** [October 2, 2024, 6:06pm UTC](https://swi-prolog.discourse.group/t/foreign-interface-buf-discardable-seems-to-be-non-working/8505/11 "2024-10-02T18:06:53Z")
I’ve been reading the code at `PL_get_nchars()`, which calls `PL_get_text()`, `PL_mb_text()`, etc. … It seems that if you _know_ that the value is pure ASCII or not a bignum, then it’s safe to not surround with `PL_STRINGS_MARK()`/`PL_STRINGS_RELEASE()` but otherwise, some values might be allocated on the buffer stack (it’s not clear to me when `BUF_MALLOC` is used – there appear to be places where the flags are ignored and `BUF_STACK` is used). @jan – should all calls to the [`PL_*()` functions that pass a pointer to `char*`](https://swi-prolog.discourse.group/t/foreign-interface-buf-discardable-seems-to-be-non-working/8505/6) be surrounded with mark/release? If you don’t know off-hand, then I’ll read the code and update the docs with what I find. ---
**Author:** ![jan](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/jan/32/4_2.png) [@jan](https://swi-prolog.discourse.group/u/jan)\ **Post date:** [October 3, 2024, 8:01am UTC](https://swi-prolog.discourse.group/t/foreign-interface-buf-discardable-seems-to-be-non-working/8505/12 "2024-10-03T08:01:42Z")
> [@peter.ludemann](#): > > @jan – should all calls to the [`PL_*()` functions that pass a pointer to `char*`](https://swi-prolog.discourse.group/t/foreign-interface-buf-discardable-seems-to-be-non-working/8505/6) be surrounded with mark/release? It you use BUF\_MALLOC(), this should not be needed. The examples by @AlexeyKosarchuk seem to indicate a bug. `BUF_STACK` surely needs it 🙂`BUF_DISCARDABLE` may need it, i.e., under some conditions you can do without but typically you’ll need it. It could be that you get a pointer to a fragile `char*`, such as a Prolog string on the stacks. Almost any stack related API call may cause a garbage collection or stack shift, invalidating this pointer. So, use one of - BUF\_MALLOC and PL\_free() - BUF\_STACK and PL\_STRINGS\_MARK()/PL\_STRINGS\_RELEASE(). Note that this is typically more efficient as small strings do not use malloc()/free(). It will also use a direct pointer to Prolog’s internal when safe (for example when getting the data from an atom) - BUF\_DISCARDABLE and PL\_STRINGS\_MARK()/PL\_STRINGS\_RELEASE(). This is similar than the above, but you must consume the data before making any other Prolog API call (to keep it simple, some are in fact safe). It may avoid a copy compared to BUF\_STACK. I think this needs some documentation updates and a fix for the first, which seems to be not always satisfied. ---
**Author:** ![AlexeyKosarchuk](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/alexeykosarchuk/32/4466_2.png) [@AlexeyKosarchuk](https://swi-prolog.discourse.group/u/AlexeyKosarchuk)\ **Post date:** [October 3, 2024, 8:16am UTC](https://swi-prolog.discourse.group/t/foreign-interface-buf-discardable-seems-to-be-non-working/8505/13 "2024-10-03T08:16:18Z")
> [@jan](#): > > The examples by @AlexeyKosarchuk seem to indicate a bug. Just to clarify a bit: if I use PL\_get\_wchars with BUF\_MALLOC, then result is actually located on the heap and is correctly freed by PL\_free. But stacked strings still appear somewhere under the hood and without mark/release stack runs out. ---
**Author:** ![jan](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/jan/32/4_2.png) [@jan](https://swi-prolog.discourse.group/u/jan)\ **Post date:** [October 3, 2024, 9:04am UTC](https://swi-prolog.discourse.group/t/foreign-interface-buf-discardable-seems-to-be-non-working/8505/14 "2024-10-03T09:04:56Z")
> [@AlexeyKosarchuk](#): > > But stacked strings still appear somewhere under the hood and without mark/release stack runs out. Yes, and that should not be the case ☹ ---
**Author:** ![jan](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/jan/32/4_2.png) [@jan](https://swi-prolog.discourse.group/u/jan)\ **Post date:** [October 3, 2024, 12:49pm UTC](https://swi-prolog.discourse.group/t/foreign-interface-buf-discardable-seems-to-be-non-working/8505/15 "2024-10-03T12:49:45Z")
> [@AlexeyKosarchuk](#): > > The catch was that even if called with BUF\_MALLOC PL\_get\_wchars allocated strings on stack. > Possibly it was some internal intermediate results. Indeed. Fixed. > [@AlexeyKosarchuk](#): > > PL\_atom\_wchars also requires usage of mark/release. This one is documented to use the string stack. It has no argument to specify how the result must be buffered, so that is it. I think this should be considered deprecated. PL\_get\_wchars() can do the same for you. PL\_atom\_wchars() is an older API. ---
**Author:** ![peter.ludemann](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/peter.ludemann/32/48_2.png) [@peter.ludemann](https://swi-prolog.discourse.group/u/peter.ludemann)\ **Post date:** [October 3, 2024, 4:38pm UTC](https://swi-prolog.discourse.group/t/foreign-interface-buf-discardable-seems-to-be-non-working/8505/16 "2024-10-03T16:38:01Z")
It appears that some of the code paths can temporarily use the string stack, even if `BUF_MALLOC` is specified. Maybe this is guaranteed to be safe because it’s guaranteed to be only one string? - but it seems that there could be leaks if mark/release stack isn’t used. ---
**Author:** ![jan](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/jan/32/4_2.png) [@jan](https://swi-prolog.discourse.group/u/jan)\ **Post date:** [October 3, 2024, 7:23pm UTC](https://swi-prolog.discourse.group/t/foreign-interface-buf-discardable-seems-to-be-non-working/8505/17 "2024-10-03T19:23:51Z")
See my patch to PL\_get\_wchars(). If BUF\_MALLOC is specified and we cannot otherwise guarantee there are no stacked intermediate results, the implementation should internally use the stack interface to guarantee no stacked strings are left. ---
**Author:** ![peter.ludemann](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/peter.ludemann/32/48_2.png) [@peter.ludemann](https://swi-prolog.discourse.group/u/peter.ludemann)\ **Post date:** [October 4, 2024, 5:24am UTC](https://swi-prolog.discourse.group/t/foreign-interface-buf-discardable-seems-to-be-non-working/8505/18 "2024-10-04T05:24:25Z")
I was thinking about the various calls to `findBuffer(BUF_STACK)`, regardless of what the flags are (mostly in `os/pl-text.c` but also in other places). ---
**Author:** ![jan](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/jan/32/4_2.png) [@jan](https://swi-prolog.discourse.group/u/jan)\ **Post date:** [October 4, 2024, 8:18am UTC](https://swi-prolog.discourse.group/t/foreign-interface-buf-discardable-seems-to-be-non-working/8505/19 "2024-10-04T08:18:56Z")
PL\_get\_wchars() uses one of these. I now fixed that using a mark/release when BUF\_MALLOC is requested, but possibly there is a better route by mallocing the value earlier during the conversion. Roughly, we need three steps: - Get a string from the Prolog term. This may or may not require buffering. - Get it in the right representation (ISO latin 1, UTF-8, locale multibyte or wchar\_t\*). - Get it stored in the right way. Preferably we want to avoid as much as possible malloc() as well as copying. Might need some reviewing … ---
**Author:** ![peter.ludemann](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/peter.ludemann/32/48_2.png) [@peter.ludemann](https://swi-prolog.discourse.group/u/peter.ludemann)\ **Post date:** [October 4, 2024, 5:02pm UTC](https://swi-prolog.discourse.group/t/foreign-interface-buf-discardable-seems-to-be-non-working/8505/20 "2024-10-04T17:02:19Z")
Shouldn’t `if(flags&BUF_MALLOC)` be `if(flags&~BUF_STACK)`? (to include `BUF_DISCARDABLE`) So, can `PL_get_text()` assume it’s inside a `PL_mark_string_buffers()`? And this is also true for other functions in `pl-text.c`, such as `PL_mb_text()`, `PL_text_recode()`, etc.? [Next page](https://swi-prolog.discourse.group/t/foreign-interface-buf-discardable-seems-to-be-non-working/8505.md?page=2)