Windows, CRLF unit test for cgi

Help! Predicate
1

The unit test for cgi fails under Windows, and I think it is because of cgi_open behavior. Somewhere deep in packages/http/cgi_stream.c, a hook is created that does several useful things, but, on Windows, replaces \n by \r\n even in the data segment. This causes some of the unit tests to fail, since they pipe an atom with ascii codes from 1 to 128 through the test and check if it returns exactly like that.

One can observe the problem by tracing through packages/http/test_cgi_stream.pl and investigating the temporary file using a hexeditor right here:

test(traditional,
     [ forall(current_data(Name)),
       Reply == Data,
       setup(open_dest(TmpF, Out)),
       cleanup(free_dest(TmpF))
     ]) :-
    data(Name, Data, ContentType),
    cgi_open(Out, CGI, cgi_hook, []),
    format(CGI, 'Content-type: ~w\n\n', [ContentType]),
    format(CGI, '~w', [Data]),
    close(CGI),
    close(Out),
    here!,
    http_read_mf(TmpF, Header, Reply),
    assert_header(Header, status(_, ok, _)).

It has the numbers 01 02 03 … 09, then 0D 0A 0B etc. The 0D is the \r. The problem can be fixed by telling the hook to use NL_POSIX, see below:

packages/http/cgi_stream.c

static foreign_t pl_cgi_open(term_t org, term_t new, term_t closure, term_t options)
{
...skipping a few lines...

line 800ff:
  s2->encoding = ENC_ASCII;             /* Header is ASCII only */
  s2->newline = SIO_NL_POSIX;    /* added this! */
  ctx->parent_encoding = s->encoding;
  s->encoding = ENC_OCTET;
  ctx->cgi_stream = s2;
  if ( PL_unify_stream(new, s2) )
  { Sset_filter(s, s2);
    silent_release_stream(s);
    LOCK();
    ctx->id = ++current_id;
    UNLOCK();

    return TRUE;
  } else
  { return instantiation_error();
  }
}

Thank you for your consideration.

2

Which test are we talking about? I get this for the CGI tests. Note there are two, one in clib for writing CGI scripts in Prolog and one in HTTP that deals with the output of the HTTP handlers which also use (most of) the CGI conventions to talk to the HTTP server infrastructure.

janw (build.win64; master *) 3_> ctest -R cgi
Test project /home/janw/src/swipl-devel/build.win64
    Start 42: http:cgi_stream
    Start 31: clib:cgi
1/2 Test #31: clib:cgi .........................   Passed    0.00 sec
2/2 Test #42: http:cgi_stream ..................   Passed    0.01 sec

100% tests passed, 0 tests failed out of 2

Total Test time (real) =   0.01 sec
3

Actually, both, but I was referring to the second one, test_cgi_stream.

?- test_cgi_stream.
% PL-Unit: cgi_stream .
ERROR: c:/users/matth/documents/prolog/test_cgi_stream.pl:189:
        test traditional (forall bindings = [ascii]): wrong answer (compared using ==)
ERROR:     Expected: '\x1\\x2\\x3\\x4\\x5\\x6\\a\b\t\n\v\f\r\xE\\xF\\x10\\x11\\x12\\x13\\x14\\x15\\x16\\x17\\x18\\x19\\x1A\\x1B\\x1C\\x1D\\x1E\\x1F\ !"#'...[126 codes]...'\\]^_`abcdefghijklmnopqrstuvwxyz{|}~'
ERROR:     Got:      '\x1\\x2\\x3\\x4\\x5\\x6\\a\b\t\r\n\v\f\r\xE\\xF\\x10\\x11\\x12\\x13\\x14\\x15\\x16\\x17\\x18\\x19\\x1A\\x1B\\x1C\\x1D\\x1E\\x1F\ !"'...[127 codes]...'\\]^_`abcdefghijklmnopqrstuvwxyz{|}~'
.

You can see the \n replaced by \r\n in the output.

1 Like
4

I see. The problem was that running ctest on my host machine got the paths from the docker container I used to build and wine some-non-existing-exe prints an error, but succeeds … So, no tests were executed :frowning: Worked around that. Applied your patch. The http pack now tests clean on Windows (for me).

Also ran SWISH using this binary. Seems all fine.

Thanks!

5

The other problem (clib:cgi) is related to environment variables.


% c:/users/matth/documents/prolog/test_cgi compiled into test_cgi 0.00 sec, -3 clauses

?- test_cgi.

% PL-Unit: cgi

ERROR: c:/users/matth/documents/prolog/test_cgi.pl:60:

test atom: received error: environment `'QUERY_STRING'' does not exist

I think it is related to a getenv call in line 393 of clib/form.c


else if ( (s = getenv("QUERY_STRING")) )

{ if ( lenp )

*lenp = strlen(s);

*data = s;

*must_free = FALSE;

return TRUE;

}

which should use the „self-written“ Getenv from src/pl-os.h, a bit like this


else if ( (s = Getenv("QUERY_STRING", buf, 1000)) )

Background: The thing is that handling of environment variables doesn’t really work under Windows.


$ cat env.c

#include <stdio.h>

#include <stdlib.h>

int main()

{

char* s = getenv("TEMP") ;

printf("TEMP=%s\n") ; <<<<<<< returns NULL

putenv("QUERY_STRING=name=value") ;

s = getenv("QUERY_STRING") ;

printf("QUERY_STRING=%s\n", s) ;

return 0 ;

}

I am sure you know this since you have written the workaround with the uppercase Setenv and Getenv.

6

The SWI-Prolog code has a lot of stuff to work around missing or incorrect POSIX functions in Windows. Some of that is still valid. In some cases the work-around is no longer needed and in a few cases such as the popen() you discovered the Windows runtime function is correct and the emulation is wrong :frowning: SWI-Prolog was ported to Windows in the Windows '95 days using Windows NT-3.1 :slight_smile:

As for this, we cannot call Getenv() as that is private to libswipl.dll. I think we have two options. One is to accept that we cannot test this library this way and we must either test it differently (by running the test in a newly launched swipl), disable the test for Windows or decide we want to allow SWI-Prolog plugins to share environment variables with plugins in a portable way and expose Getenv() as PL_getenv().

I think my preference goes to disabling this test for Windows. This library is rarely used these days anyway.

7

Do I understand correctly that using the environment variable in the unit test is a bit of a hack, in other words, that the code in clib/form.c

else if ( (s = getenv("QUERY_STRING")) )
{ if ( lenp )
*lenp = strlen(s);

only serves for testing? That would indeed argue in favor of disabling the test.

On the other hand, having a cross-platformly functioning PL_getenv (and PL_setenv und PL_unsetenv, while we are at it) could be of use elsewhere. I’ll give it a try, and if it doesn’t work within reasonable time, you’ll receive a PR that disables the unit test on Windows :slight_smile:

8

No. The CGI standard makes the HTTP query string available to the CGI script using this environment variable. That works fine, also under Windows. The problem is that setting an environment variable using Prolog setenv/2, which in the end results in calling some Win32 API, doesn’t make the variable accessible through Windows CRT getenv() function. So, we cannot test the CGI interface by setting the CGI environment variables and passing some data through it in the same process. Instead, we should create a process environment, for example using process_create/3, calling SWI-Prolog as a child process.

Adding PL_setenv() and friends is, I think, a bridge too far. If we start providing emulations for OS calls, where do we stop? The CGI stream may also work if we use the Win32 environment functions. That might be a better route.

9

Ok, I’ll give up. Although I was really close… I just added PL_setenv & co. to SWI-Prolog.h and tinkered the respective calls of Setenv & Co. into pl-fli.c (plus a bit of documentation).

But I see your point. Lemma to Greenspun 10: „Any sufficiently complicated cross-platform C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp the POSIX standard.“

1 Like