Problems with call_with_time_limit and test

General
1

Hi

I have some problems with call_with_time_limit… not honoring the time limit. Note that it may be in conjunction with some I/O, but I do not fathom how blocking could be involved given that a (short, 3Kb) file is read and then processed (and it looks like the reading finishes). The processing may be long and c_w_t_l seems to work from the command line.

However, my calls to c_w_t_l are wrapped in ‘test’ from the PLunit library and they do not seem to work. This is my wrapper.

testing(Query) :-

    testing_timeout(Timeout),
    catch(call_with_time_limit(Timeout, Query),
	  Exception,
	  (
              !,
              Exception = time_limit_exceeded ->
              fail
              ;
              throw(Exception)
          )
         ).

which I usually call as

test(foo, [det]) :- testing(thetestedpredicate(42)).

Any hint about what could go wrong?

Note that I have little control over the actual code tested, but it should not be so off a chart.

Thanks

Marco

4

Did you really want unification =/2? Comparison is ==/2.

5

Well… does it really matter in this case?

6

Probably not but one less thing to worry about.

Sometimes when the code changes, one forgets that it is a unification and not a comparison and then time is spent chasing it down. That happened to me twice and took about an hour each time because there were no test cases for that so I religiously change them as needed when I find them.

7

Hard to tell. There are some reasons for call_with_time_limit/2 to fail to abort a query. One is blocking foreign code that cannot be unblocked (for example some of the thread synchronization primitives such as mutex locking; the set depends on the OS), another are foreign code that keeps running without calling PL_handle_signals() occasionally and finally there may be low level catch/3 calls that ignore the timeout.

P.s. To turn them into failures, simply use

  catch(call_with_time_limit(Timeout, Query),
        time_limit_exceeded,
        fail).

If your code catches errors, make sure to use error(Formal,Context) as catch/3 pattern, so other exceptions are propagated.

8

Thanks… But it seems that the issue is still that the timeout does not get sent. If there is some blocking foreign code, it is in SWI Prolog itself; for sure it is not C code.

9

Unless your version is buggy, it is rather unlikely that the timeout is not sent. Well, I’m unsure about really short timeouts. I guess we need something to reproduce this.

10

Hi

the problem is that I use this scheme to test students’ code. Of course they are very… creative in their solutions, but that simply means that it is difficult to extract a proper test case. The code I posted is essentially what I use. I have a hunch that the problem may be with programs that “burn” stacks a lot. In the specific case, the students were asked to build a JSON parser; I see the problem on a longish file. The students’ parser starts and becomes insensitive to timeouts.

Go figure… (Which means that I will have to do deep debugging)

All the best

Marco

11

Still rather weird. SWISH processes zillions of student programs and uses timeouts. What OS are you using to run the tests? What is the timeout? Do you start/stop Prolog to test a single student query each time?

12

I use both quite vanilla W11 and MacOS; I could try Linux. I start Prolog (latest stable release) separately for each student and load and start the test suite.

The following line is in a bash script.

${swipl} -g "${prolog_code}" \
-t halt 2>&1 | tee -a "${summary_file}" >> "${result_file}" 2>&1

The relevant variable is

prolog_code="consult(\"${source}\"), consult(\"${prolog_test_source}\")."

source is the student’s code (a single file) and prolog_test_ source just loads a .plt file and then runs the tests.

I know it is weird and I am stumped.

Marco

13

Windows timeout handling is a bit different from the rest, but MacOS and Linux are pretty much the same.

I guess waiting until it happens and then share the specific case is as good as it gets. You can send them to me by private mail/message as I guess you don’t want the student stuff and the tests on this website.