Connecting streams to process_create/3

Help!
1

I’m using SWI-Prolog version 8.1.6

Is it possible to connect an already-open stream to the input/output/error of a process using process_create/3? I see the stdin/stderr/stdout arguments, but they only give “fresh” pipes.

i.e., is there some way to do something like


test(Status) :-
    setup_call_cleanup(
        ( open("foo_out", write, Out),
          open("foo_in", read, In),
          open("foo_err", write, Err) ),
        ( process_create(path(cat), [],
                         [ process(Pid),
% below gives
% error: `process_stream' expected, found <stream>(...)
% if I try stdout(pipe(Out)), then it's "Uninstantiated argument expected"
                           stdin(In),
                           stdout(Out),
                           stderr(Err) ]),
          process_wait(Pid, Status) ),
        ( close(Out), close(In), close(Err) )
    ).

and have the files foo_out, foo_in, and foo_err used as the output, input, and error of some process?
Failing this working directly, is there some simple way to just connect the streams returned by giving stdout(pipe(NewOut)) to the stream I’ve opened?

2 Likes
2

The API comes from SICStus. If the streams come from a file (or socket) we should technically be able to pass this on to the child without Prolog involvement. In the general case we cannot as the stream may be connected to some memory area in the Prolog process. If it concerns files you can in most cases pass the proper arguments to the process (always if you use the shell as helper).

The typical solution is to use threads to send or read the various inputs. That works for the general case, but is inefficient for files/sockets.

I would not be against adding such functionality. It is not on my shortlist though.

1 Like
3

Created a pull request to allow passing already-opened file streams to process_create/3: https://github.com/SWI-Prolog/packages-clib/pull/26

1 Like
4

Thanks. Merged!

1 Like
5

And Jan has merged a unit test case:

1 Like
6

This doesn’t seem to work under Windows (10 Pro 64-bit), where the unit test is skipped (no sh.exe).
Coding this (working) cmd (whose files typically exist on a Windows installation):

C:/Windows/System32/findstr.exe iana < C:/Windows/System32/drivers/etc/protocol

as

open('C:/Windows/System32/drivers/etc/protocol', read, Sa),
process_create('C:/Windows/System32/findstr.exe', ['iana'], [
	process(PId),
	stdin(stream(Sa)),
	stdout(pipe(Sb)),
	stderr(pipe(Sb))
]),
process_wait(PId, Status)

binds Status to exit(1) which apparently means

a match is not found in any line of any file, (or if the file is not found at all)

and nothing has been read from Sa, which has a plausible small integer FileNo from

stream_property(Sa, file_no(FileNo))

Cygwin’s grep.exe more helpfully reports

grep: (standard input): Bad file descriptor

and both fail similarly with a stream from stdout(pipe(S)) from another process_create/3 (NB is it otherwise even feasible to create process pipelines under Windows?).

7

I’m not a windows programmer, but as far as I know it isn’t really - the nearest equivalent is named pipes, which I don’t think really fit this use-case. Anyway, as I recall, this change wasn’t implemented for Windows.

8

A quick scan and test tells me using stream(S) works find for stdout on Linux, but also seems to fail for stdin as you use it. Looks like a bug. Using the stream(S) binding does have a Windows implementation. Needs some work, but I need to do other stuff now.

If someone wants to explore, see packages/clib/process.c

9

I had a closer look at using Linux for this. That failed too. The issue here was that open/4 as a text stream (default) tries to detect the encoding by means of the BOM marker, which implies it fills the buffer, after which the data is no longer available on the OS stream. In this case, opening as binary fixes this and is best option anyway. So, this works, returning the output of grep in Output and the exit status in Status.

mygrep(File, Pattern, Output, Status) :-
    open(File, read, Sa, [type(binary)]),
    process_create(path(grep), [Pattern],
                   [ process(PId),
                     stdout(pipe(Sb)),
                     stdin(stream(Sa))
                   ]),
    close(Sa),
    read_string(Sb, _, Output),
    process_wait(PId, Status).

You can also chain processes. Consider this (using >> rather than > because > cannot be chained due to operator definition):

rp :-
    run_pipe(- >>
             process(path(echo), ['Hello world']) >>
             process(path(tr), ['a-z', 'A-Z']),
             -).

run_pipe(P1 >> process(Exe, Args), Out) =>
    run_pipe(P1, Out0),
    (   var(Out)
    ->  Opt = [stdout(pipe(Out))]
    ;   Out == (-)
    ->  Opt = []
    ),
    process_create(Exe, Args,
                   [ stdin(stream(Out0))
                   | Opt
                   ]).
run_pipe(-, Out) =>
    Out = user_input.
run_pipe(File, Out) =>
    open(File, read, Out, [type(binary)]).

Now, after update to the latest GIT as file descriptor 0 (user_input) was not handled properly

?- rp.
HELLO WORLD
true.

Haven’t tested on Windows. Most likely chaining processes will not work. No idea whether it can work.

10

Pushed a fix that makes this work both for file streams and process streams. The single issue was that you have to enable inheritance of the I/O handle explicitly. Thus, you can now create a pipeline of processes using process_create/3 without using a shell and the code should run on Unix-like systems as well as Windows.

Whether it makes much sense to do so is another matter :slight_smile: I can see some advantages wrt portability and quoting arguments to be processed correctly by an intermediate shell.

1 Like