Windows, open(pipe(...)) does not seem to work

Discussion
1

I noticed that one of the tests fails on a standard Windows installation of SWI-prolog. The relevant code is copied below from pl-test.pl, with a few modifications in the attempt to debug it. The file is empty, no hello world in it.

Minor: The original code may cause unwanted backtracking if A == Text fails.

popen(cat-1) :-
        (   current_prolog_flag(windows, true)
        ->  Cmd = 'cmd /c type con'             % "type con" is cmd-speak for "cat /dev/stdin"
        ;   Cmd = cat
        ),
        current_prolog_flag(pid, Pid),
        format(atom(File), 'pltest-~w.txt', [Pid]),
        Text = 'Hello World',
        atomic_list_concat([Cmd, ' > ', File], Command),
        atom_string(Command, CommandStr),
        writeln("opening file"),
        open(pipe(CommandStr), write, Fd),
        writeln(Fd, Text),               % nothing is written here
        flush_output(Fd),
        close(Fd),
        writeln("file has been written":File),
        !,
        writeln("opening file for reading":File),
        open(File, read, Fd2),
        collect_data(Fd2, String),
        close(Fd2),
        writeln("file has been read"),
        % delete_file(File),
        atom_codes(A, String),
        % A == Text,
        writeln(result:A).

collect_data(Fd, String) :-
        get0(Fd, C0),
        collect_data(C0, Fd, String).

collect_data(-1, _, []) :- !.
collect_data(C, Fd, [C|T]) :-
        get0(Fd, C2),
        collect_data(C2, Fd, T).
2

I looked into this a bit more, using pipes with process_create:

pipe1 works fine, assuming that there is a cat.exe at the specified path.
pipe2 works fine, as well, findstr is the Windows version of grep
pipe3 does not work. the program hangs, and one has to kill the cmd.exe using the task manager.

Conclusion: cmd /c etc. does not like pipes.

I suggest to change the unit test, at least for windows.

pipe1(Lines) :-
        process_create("c:/msys64/usr/bin/cat.exe", [],
                       [ stdin(pipe(In)), stdout(pipe(Out)) ]),
        writeln(In, "xxx1"),
        writeln(In, "xxx2"),
        writeln(In, "xxx3"),
        writeln(In, "yyy"),
        close(In),
        read_lines(Out, Lines),
        close(Out).

read_lines(Out, Lines) :-
        read_line_to_codes(Out, Line1),
        read_lines(Line1, Out, Lines).

read_lines(end_of_file, _, []) :- !.
read_lines(Codes, Out, [Line|Lines]) :-
        atom_codes(Line, Codes),
        read_line_to_codes(Out, Line2),
        read_lines(Line2, Out, Lines).

pipe2(Lines) :-
        process_create(path("findstr.exe"), [xxx],
                       [ stdin(pipe(In)), stdout(pipe(Out)) ]),
        writeln(In, "xxx1"),
        writeln(In, "xxx2"),
        writeln(In, "xxx3"),
        writeln(In, "yyy"),
        close(In),
        read_lines(Out, Lines),
        close(Out).

pipe3(Lines) :-
        process_create(path("cmd.exe"), ["/c", "type", "con"],
                       [ stdin(pipe(In)), stdout(pipe(Out)) ]),
        writeln(In, "xxx1"),
        writeln(In, "xxx2"),
        writeln(In, "xxx3"),
        writeln(In, "yyy"),
        close(In),
        read_lines(Out, Lines),
        close(Out).
3

Thanks. So, we need real executables. I’ve made an attempt to use Prolog itself, but that also causes problems. No clue why. Might be related to the interactive mode it tries to create when running inside a console, although that should be disabled if there is no console connected to I/O.

4

Unsure. It rather appears that one can open a pipe for reading but not for writing:

This works fine

popen(pwd-1) :-
	(   current_prolog_flag(windows, true)
	->  Command = 'cmd /c cd'
	;   Command = pwd
	),
	open(pipe(Command), read, Fd),
	collect_line(Fd, Codes),
	close(Fd),
	string_codes(String, Codes),
        writeln(String).

This one does not, hallo.txt is empty. (dd.exe is found in the local folder)

popen(dd-1) :-
	(   current_prolog_flag(windows, true)
	->  Command = 'dd.exe of=hallo.txt'
	;   Command = 'cat'
	),
        open(pipe(Command), write, Fd),
	writeln(Fd, xxx1),
	flush_output(Fd),
	close(Fd),
	setup_call_cleanup(
	    open("hallo.txt", read, Fd2),
	    collect_data(Fd2, String),
	    close(Fd2)),
	delete_file("hallo.txt"),
	writeln(String).
5

That’s a nasty one.

This little program works fine in msys2:

#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>

int main(void)
{
  FILE* output = popen("dd.exe of=hallo1.txt", "w") ;
  write(fileno(output), "matthias", 9) ;
  pclose(output) ;
  return 0 ;
}

However, if I copy the three lines (FILE* …, write…, close…) into pl-stream.c, e.g. somewhere into Sopen_pipe, a file hallo1.txt is created in the working directory, but it’s empty. errno, return values, everything seems to work, nothing suspicious.

1 Like
6

There must be a hint in that :slight_smile: No time now …

7

Ok, three things were needed to get the pipes work on msys2/mingw64. Since the problem is also seen on the normal swipl for Windows, it may be worth a try.

  1. I disabled the code in src/os/windows/popen.c that seemed to create the problem. pl-stream.h or .c includes this file under Windows (I commented it out) and at some point invoked an initialize function (I also commented it out). As a consequence, the normal popen is now used.

  2. I think the normal popen cannot deal with > redirecting output under Windows. Except for sort.exe, no Windows command line program accepts input from stdin and can, at the same time, redirect output on a file by a command-line switch /o file. Unfortunately, sort also exists in a bash environment, and the bash-sort does not like slash-switches. It may be worth trying your swipl.exe solution with tell-copy-told again :slight_smile:

  3. For now, I created a little bat-file with @findstr .* > %1 as a replacement for cat > out.txt.

popen(cat-1) :-
        current_prolog_flag(pid, Pid),
        format(atom(File), 'pltest-~w.txt', [Pid]),
        (   current_prolog_flag(windows, true)
        ->  format(atom(Cmd), 'cmd /c typecon.bat ~w', [File]),
            format(atom(Bat), 'pltest-~w.bat', [Pid]),
            open(Bat, write, Fd1),
            writeln(Fd1, '@findstr .* > %1'),
            close(Fd1)
        ;   format(atom(Cmd), 'cat > ~w', [File])
        ),
        Text = 'Hello World',
        open(pipe(Cmd), write, Fd2),
        format(Fd2, '~w~n', [Text]),
        close(Fd2),
        open(File, read, Fd3),
        collect_data(Fd3, String),
        close(Fd3),
        delete_file(File),
        (   current_prolog_flag(windows, true)
        ->  delete_file(Bat)
        ),
        !,
        atom_codes(A, String),
        format(atom(A), '~w~n', [Text]).

At least the pipes are working again. If you wish, I can make a PR. I don’t know how much things break if the code in src/os/windows/popen.c is removed, but given that it doesn’t work anyway…

2 Likes
8

Thanks for the PR! I’ve moved the updated tests to a plunit test file and modernized them. Now passes on Windows :slight_smile:

I’ve also updated the docs of open/3,4 to deprecate the pipe(Command) feature. process_create/3 is much more versatile and portable, at least to SICStus.