# Automatic tests can succeed with syntax errors

**URL:** <https://swi-prolog.discourse.group/t/automatic-tests-can-succeed-with-syntax-errors/3891>\
**Category:** Discussion\
**Tags:** bug\
**Created:** [April 30, 2021, 5:10pm UTC](https://swi-prolog.discourse.group/t/automatic-tests-can-succeed-with-syntax-errors/3891 "2021-04-30T17:10:07Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![peter.ludemann](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/peter.ludemann/32/48_2.png) [@peter.ludemann](https://swi-prolog.discourse.group/u/peter.ludemann)\
**Post date:** [April 30, 2021, 5:10pm UTC](https://swi-prolog.discourse.group/t/automatic-tests-can-succeed-with-syntax-errors/3891/1 "2021-04-30T17:10:08Z")

</div>

The test suite for building SWI-Prolog is run by the command `ctest`, which uses the file `packages/cmake/PrologPackage.cmake` to run tests defined by

```cmake
  add_test(NAME "${SWIPL_PKG}:${test_name}"
	   COMMAND ${PROG_SWIPL} -p "foreign=${pforeign}"
			 -f none --no-packs -s ${test_source}
			 -g "${test_goal}"
			 -t halt)

```

Unfortunately, this doesn’t catch syntax errors in the tests, as demonstrated here (outside the `ctest` framework, because I’m working in a separate module:

```prolog
$ swipl "-p" "foreign=" "-f" "none" "--no-packs" "-s" test_protobufs.pl "-g" "test_protobufs" "-t" "halt" ; echo $?
ERROR: /home/peter/src/contrib-protobufs/test_protobufs.pl:267:22: Syntax error: Operator expected
% PL-Unit: protobuf_message ....... done
% PL-Unit: protobuf_segment_convert ....... done
% All 14 tests passed
0

```

Question: is there an option to `swipl` that will cause a failure if a file is loaded that has a syntax error?

By the way, here’s the same thing done using `ctest`, by introducing a deliberate syntax error into the current source (which doesn’t use `plunit`, so it produces different output):

```prolog
$ ctest -R protobufs
Test project /home/peter/src/swipl-devel/build
    Start 49: protobufs:protobufs
1/1 Test #49: protobufs:protobufs .............. Passed 0.18 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) = 0.19 sec

```

```prolog
$ ctest -V -R protobufs
UpdateCTestConfiguration from :/home/peter/src/swipl-devel/build/DartConfiguration.tcl
UpdateCTestConfiguration from :/home/peter/src/swipl-devel/build/DartConfiguration.tcl
Test project /home/peter/src/swipl-devel/build
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 49
    Start 49: protobufs:protobufs

49: Test command: /home/peter/src/swipl-devel/build/src/swipl "-p" "foreign=" "-f" "none" "--no-packs" "-s" "/home/peter/src/swipl-devel/packages/protobufs/test_protobufs.pl" "-g" "test_protobufs" "-t" "halt"
49: Test timeout computed to be: 10000000
49: ERROR: /home/peter/src/swipl-devel/packages/protobufs/test_protobufs.pl:308:5: Syntax error: Operator expected
49: Loading Google's Golden Wirestream (2.5.0)...OK
49: Unifying canned Golden Message with canned Golden Template...OK
49: Unifying canned Golden Message with Google's Golden Wirestream...OK
49: Parsing Google's Golden Wirestream to canned Golden Template...OK
49: Comparing canned Golden Message to parsed Golden Template...OK
49: Serializing canned Golden Message to Codes...OK
49: Comparing Google's Golden Wirestream to Codes...OK
49: Parsing Codes to canned Golden Template...OK
49: Comparing canned Golden Message to parsed Golden Template...OK
49: All tests passed.
1/1 Test #49: protobufs:protobufs .............. Passed 0.18 sec

The following tests passed:
	protobufs:protobufs

100% tests passed, 0 tests failed out of 1

Total Test time (real) = 0.18 secc

```

---

<div class="post-metadata">

**Author:** ![rla](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/rla/32/6863_2.png) [@rla](https://swi-prolog.discourse.group/u/rla)\
**Post date:** [May 1, 2021, 12:12pm UTC](https://swi-prolog.discourse.group/t/automatic-tests-can-succeed-with-syntax-errors/3891/2 "2021-05-01T12:12:59Z")

</div>

I have also been bitten by this behavior but in a server environment with automatic code deployment. Code was deployed and started successfully despite some modules having loading errors. The errors would cause issues later when specific code path was run.

I defined a dynamic flag “started” to counter this. The flag was set after the server was started. In your case you could set a flag right after loading but before running the tests. Any error or warning emitted before the flag is set would be considered as a loading error and would terminate the process with non-zero status code (make sure this hook comes before other code):

```prolog
user:message_hook(Term, Type, _):-
    \+ started,
    ( Type = error ; Type = warning ),
    message_to_string(Term, String),
    writeln(user_error, String),
    halt(1).

```

In development you could disable the hook to make the “make”-based editing not exit the process.

---

<div class="post-metadata">

**Author:** ![jan](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/jan/32/4_2.png) [@jan](https://swi-prolog.discourse.group/u/jan)\
**Post date:** [May 1, 2021, 3:04pm UTC](https://swi-prolog.discourse.group/t/automatic-tests-can-succeed-with-syntax-errors/3891/3 "2021-05-01T15:04:52Z")

</div>

Probably the best way out would be to provide some library that you can load at the start of your program or even using a commandline argument. As compiling and running are not well separated stages in SWI-Prolog there are many complicated cases though. For tests even more as there are tests that validate that compiling some code produces some specific message … We probably need some flag that says “errors expected”

---

<div class="post-metadata">

**Author:** ![peter.ludemann](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/peter.ludemann/32/48_2.png) [@peter.ludemann](https://swi-prolog.discourse.group/u/peter.ludemann)\
**Post date:** [May 1, 2021, 4:31pm UTC](https://swi-prolog.discourse.group/t/automatic-tests-can-succeed-with-syntax-errors/3891/4 "2021-05-01T16:31:50Z")

</div>

It gets worse: a syntax error doesn’t result in an error from the compiler (`swipl --stand_alone=true --undefined=error --verbose=false --foreign=save -o xxx.qlf -c xxx.pl`).

I hadn’t noticed this before because a syntax error usually cascades to other errors that do stop the compiler, Why is syntax error special in not giving a bad return code? (For example, `--undefined=error` sets the return code the way I expect.)

(I tried @rla 's message\_hook trick and it did stop a compilation with a return code, which is nice.)

---

<div class="post-metadata">

**Author:** ![peter.ludemann](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/peter.ludemann/32/48_2.png) [@peter.ludemann](https://swi-prolog.discourse.group/u/peter.ludemann)\
**Post date:** [May 3, 2021, 8:04pm UTC](https://swi-prolog.discourse.group/t/automatic-tests-can-succeed-with-syntax-errors/3891/5 "2021-05-03T20:04:59Z")

</div>

> <https://github.com/SWI-Prolog/swipl-devel/issues/826>
>
> The compiler should return a non-zero return code if there's any error, includin…g syntax errors. (For example, it has an option to return non-zero if there's an undefined predicate.)
> \`\`\`
> $ swipl --stand\_alone=true --undefined=error --verbose=false --foreign=save -o xxx.qlf -c xxx.pl ; echo rc=$?
> ERROR: /tmp/xxx.pl:1:5: Syntax error: Operator expected
> % Disabled autoloading (loaded 28 files)
> % Disabled autoloading (loaded 2 files)
> % Disabled autoloading (loaded 0 files)
> rc=0
> \`\`\`
> 
> and the source for xxx.pl is:
> \`\`\`
> this has a syntax error.
> \`\`\`

---

<div class="post-metadata">

**Author:** ![jan](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/jan/32/4_2.png) [@jan](https://swi-prolog.discourse.group/u/jan)\
**Post date:** [May 4, 2021, 11:31am UTC](https://swi-prolog.discourse.group/t/automatic-tests-can-succeed-with-syntax-errors/3891/6 "2021-05-04T11:31:05Z")

</div>

I’m still pretty unsure on how to address this. Yes, it is a problem in unattended builds. A robust solution that will not cause basically ok programs that print error messages to exit with status 1 is far from trivial. One option might be to do this only for `swipl -c file.pl` as that is not supposed to run anything except stuff that supports the compilation process.

For general case it is mostly initialization goals that cause problems as these may support the compilation (should be considered an error) but may also run the application. Many people use initialization/1 to start applications. On SWI-Prolog that is not the best solution and one should use initialization/2 to be more precise on when the initialization hook is executed.

Another problem is what is an error? We could state any call to print\_message/2 using `error`. However, some programs deliberately capture these messages with the purpose to suppress them. So, print\_message/2 that is not successfully hooked? But then we have programs that hook to redirect messages to some channel such as a logging facility.

This seems to be the price of integrating an incremental compiler into the runtime environment and allow running arbitrary goals as a side effect of compiling a file.

---

<div class="post-metadata">

**Author:** ![peter.ludemann](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/peter.ludemann/32/48_2.png) [@peter.ludemann](https://swi-prolog.discourse.group/u/peter.ludemann)\
**Post date:** [May 5, 2021, 7:15pm UTC](https://swi-prolog.discourse.group/t/automatic-tests-can-succeed-with-syntax-errors/3891/7 "2021-05-05T19:15:42Z")

</div>

> [@jan](#):
>
> One option might be to do this only for `swipl -c file.pl`

This (partial) solution would at least produce “less surprise” when generating .qlf files in Makefiles. Perhaps controlled by a flag `--error=error` (to match `--undefined=error`)? I presume that if someone wants to suppress a specific error, they can use the user:message\_hook/2 trick.  
(Might also want a `--warning=error` flag, similar to g++'s `-Wall`).

Question: does the user:mesage\_hook/2 trick work across all imported files by just defining user:message\_hook/2 in the main module, or would I have to add it to each imported file? (e.g., to avoid the problem that @rla described)

---

<div class="post-metadata">

**Author:** ![peter.ludemann](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/peter.ludemann/32/48_2.png) [@peter.ludemann](https://swi-prolog.discourse.group/u/peter.ludemann)\
**Post date:** [May 7, 2021, 4:32pm UTC](https://swi-prolog.discourse.group/t/automatic-tests-can-succeed-with-syntax-errors/3891/8 "2021-05-07T16:32:05Z")

</div>

Where in the source is the compiler called? (I tried tracing through the top-level intialization, but I got lost).

Would I be better off by creating a user:message/3 hook, and then running qsave\_program/2, rather than using `swipl -c`? But I’d need to somehow remove the user:message/3 hook from the saved state …

---

<div class="post-metadata">

**Author:** ![jan](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/jan/32/4_2.png) [@jan](https://swi-prolog.discourse.group/u/jan)\
**Post date:** [May 7, 2021, 5:00pm UTC](https://swi-prolog.discourse.group/t/automatic-tests-can-succeed-with-syntax-errors/3891/9 "2021-05-07T17:00:08Z")

</div>

The `-c` invocation is handled by ‘$compile’/0 in `boot/toplevel.pl`. One solution to get rid of the hook is initialization/2 with _When_ set to `restore_state`. Alternatively just before creating the state. Not sure anything can go wrong that doesn’t lead to failure anyway. Or create a generic handler that always fails except while creating the state.

---

<div class="post-metadata">

**Author:** ![peter.ludemann](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/peter.ludemann/32/48_2.png) [@peter.ludemann](https://swi-prolog.discourse.group/u/peter.ludemann)\
**Post date:** [May 7, 2021, 5:53pm UTC](https://swi-prolog.discourse.group/t/automatic-tests-can-succeed-with-syntax-errors/3891/10 "2021-05-07T17:53:15Z")

</div>

That sounds like a non-trivial amount of work, so I guess we’ll just have to leave this as an open issue in github.

For my purposes, it’s sufficient to have a user:message\_hook/3 that I can abolish after loading the saved state - and it appears that user:message\_hook/3 has global effect, so it seems that I can trap syntax errors with a single hook definition.

For now, I’ll just put this into my code (it’s probably insufficient - e.g., doesn’t handle a missing fie or some other “consult” errors):

```prolog
user:message_hook(Term, error, Lines) :-
    Term = error(syntax_error(_Msg), _Details),
    print_message_lines(user_error, 'ERROR: ', Lines),
    halt(1).

```

---

<div class="post-metadata">

**Author:** ![peter.ludemann](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/peter.ludemann/32/48_2.png) [@peter.ludemann](https://swi-prolog.discourse.group/u/peter.ludemann)\
**Post date:** [May 8, 2021, 7:14pm UTC](https://swi-prolog.discourse.group/t/automatic-tests-can-succeed-with-syntax-errors/3891/11 "2021-05-08T19:14:43Z")

</div>

It seems that there’s a solution, and a rather elegant one at that. Thank-you @jan!

> [@Changes in error handling](https://swi-prolog.discourse.group/t/changes-in-error-handling/3930):
>
> As a followup to [Automatic tests can succeed with syntax errors](https://swi-prolog.discourse.group/t/automatic-tests-can-succeed-with-syntax-errors/3891) and similar topics raised in the past I’ve pushed a number of changes: Add global and thread statistics/2 keys errors and warnings to count errors and warnings. This only counts errors and warnings printed using print\_message/2 that are not intercepted using message\_hook/3. Add an option --on-error= and --on-warning= and related Prolog flags on\_error and on\_warning that can all have the values print (default), status (continu…