Exception when closing a stderr(pipe) after process_create failure

Help!
1

I would appreciate any insight regarding a problem I’m experiencing. An exception when I close the stderr pipe via

process:process_create(‘/tmp/duma.sh’, [stdout(pipe(Out)), stderr(pipe(Err))])

The file duma.sh does not exist.

The following is correctly handled and stderr pipe is closed because dum.sh exists and writes one line to each of stdout and one to stderr.

process:process_create('/tmp/dum.sh', [stdout(pipe(Out)), stderr(pipe(Err))]), writeln(Out), writeln(Err), read_line_to_string(Out,OutStr), read_line_to_string(Err,ErrStr), (nonvar(Out) -> close(Out);true), (nonvar(Err) -> close(Err);true).
Out = <stream>(0x3dd9c00),
Err = <stream>(0x3dd9d00),
OutStr = "From dum to stdout",
ErrStr = "From dum to stderr".

However if the file doesn’t exist, an exception is generated during closure of stderr (though it is closed)

process:process_create('/tmp/duma.sh', [stdout(pipe(Out)), stderr(pipe(Err))]), writeln(Out), writeln(Err), read_line_to_string(Out,OutStr), read_line_to_string(Err,ErrStr), (nonvar(Out) -> close(Out);true), (nonvar(Err) -> close(Err);true).
ERROR: Process "/tmp/duma.sh": exit status: 1
ERROR: In:
ERROR:   [11] close(<stream>(0x3ddf100))
ERROR:   [10] '<meta-call>'(user:user: ...) <foreign>
ERROR:    [9] toplevel_call(user:user: ...) at /usr/local/lib/swipl/boot/toplevel.pl:1158
   Exception: (11) close(<stream>(0x3ddf100)) ? creep

To avoid the exception and don’t close the stderr pipe, we end up with hundreds of pipes. (Each run is about 30K processes). swipl is a very long running process.

Versions: 8.4.3 and 9.0.4

PS Yes the absolute_file_name check in process_create/3 mitigates this particular problem, but I’ve tried to simplify the issue in preference to more significant execution failures. I do use catch for exception handling, but I’d rather understand the problem :slightly_smiling_face:

2

prolog streams have their own exception state, separate from the ‘normal’ engine state. Operations on streams will often raise these as ordinary prolog exceptions whenever they encounter a stream in this state. That is what is happening on your close here.
you can close a stream with the ‘force’ option like this:

close(Err, [force(true)]).

That should supress any errors.

You can also check error state with

stream_property(Err, error(HasError)).
1 Like
3

If you add the option process(PID), you get the process status with process_wait/2. If not, a non-zero exit of the process is translated to an exception when closing the I/O streams.

Notably for long running processes you should use setup_call_cleanup/3 for almost all external resources such as file handles.

Note that calling process:process_create/2 calls a private predicate. No guarantees are made about the existence or behavior of private predicates.

In newer versions you can set the Prolog flag agc_close_streams to true, which causes the garbage collector to close forgotten file handles. Note that GC has no way to propagate errors, so a possible error is printed but otherwise ignored.

4

Thankyou very much for your advise.

  • Mitigation 1. process_wait paired with process_create options: detached(true), process(PID)

  • Mitigation 2. implement the close option, force(true)
    Both mitigations produce the same outcome when applied against a successful external program or via an induced error. Also confirmed by current_stream(X,Y,Z), format(“~w ~w ~w\n”, [X,Y,Z]), fail.

I use delaying process_create completion to acquire status information elsewhere, but not in this use-case where I need to audit stderr output. I wasn’t aware of the side-effects of redirected stream closure, so this is enlightening. :slight_smile: Maren pointing me to force closing, is the option for this particular case.

Jan, the ‘non-zero process status’ forcing an exception on stream closure is instructional. I was concerned that my implementation was buggy as the cause/effect appear distant.

Its good advice to use setup_call_cleanup/3 its extremely useful and I do use extensively.

The new set_prolog_flag option of agc_close_streams is intriguing, though I think I should be more disciplined in file handling. :slight_smile:

You were both very helpful.

Aside, using catch around close(stderr) occludes the problem but also quite ugly.

Kind regards, Dewayne.
PS I do use the latest swipl 9.2.2 available on FreeBSD14 but there I use CFLAGS -O3 -flto=full -D_FORTIFY_SOURCE=3 -fstrict-flex-arrays=3 … which may distract from my older vanilla builds.

5

Glad to have been of help :slight_smile:

Are you sure you want this? This is there to create a new process group. Something that is traditionally used to create a service process from behind the terminal, i.e., it allows the process to continue after you log off.

6

:slight_smile: Good question and I understand where you’re coming from, and appreciate you emphasising the behaviour. Yes, I don’t require a terminal association for most uses. I use s6 and s6-rc from skarnet to perform process and service management while swipl (will) oversee the service manager on my HardenedBSD machines. swipl starts at boot as a standalone, its really quite fun - if it weren’t for the large number of library dependencies swipl standalone would be process 1 :wink: . I’ve used PROLOG since 80’s though this is my first foray into processes and threads within its context.

1 Like
7

LOL :slight_smile: I’ve seen many things SWI-Prolog is used for, but this is new. I know it is being used to model software component dependencies and asses the impact of failing components and find ways to get as much as possible services usable.

If you use -DUSE_GMP=OFF, the core only depends libz. The stuff in the clib package only depends on the standard C lib of the system (hence the name). Using -DSTATIC_EXTENSIONS=ON the packages are merged into the main executable and if you link statically to libz and optionally to libc, you can have as few runtime dependencies as you like.

In other words, it takes some tweaking, but you can create a single file SWI-Prolog executable with a selection of the bundled extensions and your application that is completely stand-alone. There is also a little known option to store the “state” in an embedded string in the executable rather than adding it to the end of the executable. That allows dropping read access from the executable.