Rtools42, Unit test for pcre

mgondan1 · April 5, 2022, 5:12pm

Rtools42 is an MinGW distribution for package development in R. Compilation of swipl-devel is successful, and so is ?- check_installation., but I get an error in ctest for pcre, related to missing re_config(jittarget(X)) if there is no PCRE JIT compilation, that is, re_config(jit(false)).

Unsure what to do. Currently, re_config(jittarget(X)) fails, although it might also raise an error. In test_pcre.pl, it seems that there may have been an error message in a previous version. I could also think of a scenario in which jittarget(X) returns something like X = “JIT is not enabled”, or just “NA”.

Maybe someting like this (test_pcre.pl, line 350).

re_test(config_all3, DocSorted = ConfigsSorted) :-
    Doc = [bsr2(_),
           compiled_widths(_),
           depthlimit(_),
           heaplimit(_),
           jit(_),
           jittarget(_),
           linksize(_),
           matchlimit(_),
           never_backslash_c(_),
           newline2(_),
           parenslimit(_),
           stackrecurse(_),
           unicode(_),
           unicode_version(_),
           version(_)
           ],
    sort(Doc, DocSorted),
    bagof(C, re_config_for_doc(C), Configs), %%%%%%%%%%%%% here
    assertion(ground(Configs)),
    msort(Configs, ConfigsSorted), % note msort/2, in case of dups
    % For more easily finding the first mismatch:
    maplist(assertion_eq, DocSorted, ConfigsSorted).

re_config_for_doc(jittarget(none)) :-
    re_config(jit(false)).

re_config_for_doc(X) :-
    re_config(X).

And a similar thing below (added the condition):

re_test(config_jittarget, [condition(re_config(jit(true)))]) :-
    re_config(jittarget(V)),
    must_be(atom, V).

unicode and unicode_version may have the same issue.

peter.ludemann · April 5, 2022, 6:08pm

Is this on Windows?

I’ll need to look a bit into how check_installation works, and get back to you.

mgondan1 · April 5, 2022, 6:11pm

It’s on windows, but please disregard check_installation. That is working fine. It’s the unit test that fails, test_pcre.

It boils down to a decision whether re_config(jittarget(X)) should just fail or raise an error if re_config(jit(false)).

peter.ludemann · April 5, 2022, 6:20pm

Oops, I made a mistake in reading your email … I’ll have to put together a test for the combination you gave me. I presume that the version of pcre2 that you have doesn’t have JIT enabled.

It looks like there’s an error in the test (a left-over from some other things I did).

mgondan1 · April 5, 2022, 6:23pm

Yes, indeed. JIT is not available on the libpcre2 on my system.

peter.ludemann · April 5, 2022, 6:26pm

I used to have invalid config queries throwing an exception, but I changed that to failure.
e.g. re_config(foo(X)) used to be an error, but now it’s fail.
I’ll take a look at the tests and put together a PR.

mgondan1 · April 5, 2022, 6:29pm

Thanks, Peter. A not much elegant solution is found in my post above.

peter.ludemann · April 5, 2022, 6:53pm

Sorry for the strange answers … I was replying from gmail and it wasn’t showing the entire message that you sent.

Please look at the prec2.h file and tell me if it has the following #defines:
PCRE2_NO_JIT
PCRE2_CONFIG_JIT
PCRE2_CONFIG_JITTARGET

mgondan1 · April 5, 2022, 7:00pm

#define PCRE2_NO_JIT                      0x00002000u  /* Not for pcre2_dfa_match() */
#define PCRE2_CONFIG_JIT                     1
#define PCRE2_CONFIG_JITTARGET               2

peter.ludemann · April 5, 2022, 7:10pm

Ignore that last message – if those items weren’t defined, you’d get a compile error.

peter.ludemann · April 5, 2022, 8:25pm

Please compile and run this code: Output some PCRE2 values for debugging PCRE2 with MinGW · GitHub

On my system, I built it and ran with this:

$ gcc pcre2_mingw.c -lpcre2-8 && ./a.out
VERSION (len=17 rc=17) 0x0000000b: '10.34 2019-11-21'
JITTARGET (len=38 rc=38) 0x00000002: 'x86 64bit (little endian + unaligned)'
UNICODE_VERSION (len=7 rc=7) 0x0000000a: '12.1.0'
JIT (len=4 rc=0) 0x00000001: 0x00000001

Also, please run this

$ swipl
Welcome to SWI-Prolog (threaded, 64 bits, version 8.5.9-44-g69745c4bd4-DIRTY)

?- forall(member(V, [version(_),jit(_),jittarget(_),unicode_version(_)]), (re_config(V), format('~q~n', [V]))).
version('10.34 2019-11-21')
jit(true)
jittarget('x86 64bit (little endian + unaligned)')
unicode_version('12.1.0')

Also, assuming you’re synced to the latest version of package-pcre:

?- forall(re_config(V), format('~q~n', [V])).
bsr2(unicode)
compiled_widths(7)
depthlimit(10000000)
heaplimit(20000000)
jit(true)
jittarget('x86 64bit (little endian + unaligned)')
linksize(2)
matchlimit(10000000)
never_backslash_c(false)
newline2(lf)
parenslimit(250)
stackrecurse(false)
unicode(true)
unicode_version('12.1.0')
version('10.34 2019-11-21')

mgondan1 · April 6, 2022, 6:29am

Here you go:

Matthias@DESKTOP-A2T8IFC UCRT64 ~
$ cd /home/matthias/

Matthias@DESKTOP-A2T8IFC UCRT64 /home/matthias
$ mkdir pcre2

Matthias@DESKTOP-A2T8IFC UCRT64 /home/matthias
$ cd pcre2/

Matthias@DESKTOP-A2T8IFC UCRT64 /home/matthias/pcre2
$ notepad pcre2_mingw.c

Matthias@DESKTOP-A2T8IFC UCRT64 /home/matthias/pcre2
$ gcc pcre2_mingw.c
C:/rtools42/ucrt64/bin/../lib/gcc/x86_64-w64-mingw32/11.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:\rtools42\tmp\ccdgKHzR.o:pcre2_mingw.c:(.text+0x1c): undefined reference to `__imp_pcre2_config_8'
C:/rtools42/ucrt64/bin/../lib/gcc/x86_64-w64-mingw32/11.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:\rtools42\tmp\ccdgKHzR.o:pcre2_mingw.c:(.text+0x31): undefined reference to `__imp_pcre2_config_8'
C:/rtools42/ucrt64/bin/../lib/gcc/x86_64-w64-mingw32/11.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:\rtools42\tmp\ccdgKHzR.o:pcre2_mingw.c:(.text+0xb4): undefined reference to `__imp_pcre2_config_8'
C:/rtools42/ucrt64/bin/../lib/gcc/x86_64-w64-mingw32/11.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:\rtools42\tmp\ccdgKHzR.o:pcre2_mingw.c:(.text+0xc9): undefined reference to `__imp_pcre2_config_8'
collect2.exe: error: ld returned 1 exit status

Matthias@DESKTOP-A2T8IFC UCRT64 /home/matthias/pcre2
$ gcc pcre2_mingw.c -lpcre2-8

Matthias@DESKTOP-A2T8IFC UCRT64 /home/matthias/pcre2
$ ./a.exe
VERSION (len=17 rc=17) 0x0000000b: '10.39 2021-10-29'
FAILED: JITTARGET len=-34 rc=-34
UNICODE_VERSION (len=7 rc=7) 0x0000000a: '14.0.0'
JIT (len=4 rc=0) 0x00000001: 0x00000000

Matthias@DESKTOP-A2T8IFC UCRT64 /home/matthias/pcre2
$

mgondan1 · April 6, 2022, 6:45am

Part 2

$ swipl
Welcome to SWI-Prolog (threaded, 64 bits, version 8.5.9-46-g26ce7cb97-DIRTY)
SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software.
Please run ?- license. for legal details.

For online help and background, visit https://www.swi-prolog.org
For built-in help, use ?- help(Topic). or ?- apropos(Word).

forall(member(V, [version(_),jit(_),jittarget(_),unicode_version(_)]), (re_config(V), format('~q~n', [V]))).

version('10.34 2019-11-21')
version('10.39 2021-10-29')
jit(false)
false.

forall(re_config(V), format('~q~n', [V])).
bsr2(unicode)
compiled_widths(7)
depthlimit(10000000)
heaplimit(20000000)
jit(false)
linksize(2)
matchlimit(10000000)
never_backslash_c(false)
newline2(lf)
parenslimit(250)
stackrecurse(false)
unicode(true)
unicode_version('14.0.0')
version('10.39 2021-10-29')
true.

jan · April 6, 2022, 7:51am

It seems the unit tests are a bit too strict and cannot deal with different configurations/versions of pcre. This is what I get on an Apple M1 running the latest everything, using pcre2 from Macports
(pcre2 @10.39 (devel))

% PL-Unit: pcre .....................................................................................
ERROR: /Users/jan/src/swipl-devel/packages/pcre/test_pcre.pl:350:
	test config_all3: assertion failed
	Assertion: jittarget(_12516)=linksize(2)
ERROR: /Users/jan/src/swipl-devel/packages/pcre/test_pcre.pl:350:
	test config_all3: assertion failed
	Assertion: linksize(_12526)=matchlimit(10000000)
ERROR: /Users/jan/src/swipl-devel/packages/pcre/test_pcre.pl:350:
	test config_all3: assertion failed
	Assertion: matchlimit(_12536)=never_backslash_c(false)
ERROR: /Users/jan/src/swipl-devel/packages/pcre/test_pcre.pl:350:
	test config_all3: assertion failed
	Assertion: never_backslash_c(_12546)=newline2(lf)
ERROR: /Users/jan/src/swipl-devel/packages/pcre/test_pcre.pl:350:
	test config_all3: assertion failed
	Assertion: newline2(_12556)=parenslimit(250)
ERROR: /Users/jan/src/swipl-devel/packages/pcre/test_pcre.pl:350:
	test config_all3: assertion failed
	Assertion: parenslimit(_12566)=stackrecurse(false)
ERROR: /Users/jan/src/swipl-devel/packages/pcre/test_pcre.pl:350:
	test config_all3: assertion failed
	Assertion: stackrecurse(_12576)=unicode(true)
ERROR: /Users/jan/src/swipl-devel/packages/pcre/test_pcre.pl:350:
	test config_all3: assertion failed
	Assertion: unicode(_12586)=unicode_version('14.0.0')
ERROR: /Users/jan/src/swipl-devel/packages/pcre/test_pcre.pl:350:
	test config_all3: assertion failed
	Assertion: unicode_version(_12596)=version('10.39 2021-10-29')
ERROR: /Users/jan/src/swipl-devel/packages/pcre/test_pcre.pl:350:
	test config_all3: failed

.....
ERROR: /Users/jan/src/swipl-devel/packages/pcre/test_pcre.pl:388:
	test config_jittarget: failed

................................................................. done
% one test is blocked:
% /Users/jan/src/swipl-devel/packages/pcre/test_pcre.pl:748:
	test wb_2: javascript_compat
% 9 assertions failed
% 2 tests failed
% 155 tests passed

peter.ludemann · April 6, 2022, 4:34pm

Please run the C code and forall/format query on the M1? (I don’t have easy access to a Mac of any kind)

gist.github.com

https://gist.github.com/kamahen/9f54a160773125f382c81d2b5470a872

pcre2_mingw.c

#include <stdio.h>
#define PCRE2_CODE_UNIT_WIDTH 8
#include <pcre2.h>

static void config_uint(const char *descr, uint32_t flag) {
  uint32_t value;
  uint32_t len = pcre2_config_8(flag, NULL);
  int rc = pcre2_config_8(flag, &value);
  if (rc >= 0) {
    printf("%s (len=%d rc=%d) 0x%08x: 0x%08x\n", descr, len, rc, flag, value);

This file has been truncated. show original

jan · April 6, 2022, 4:55pm

Got this

VERSION (len=17 rc=17) 0x0000000b: '10.39 2021-10-29'
FAILED: JITTARGET len=-34 rc=-34
UNICODE_VERSION (len=7 rc=7) 0x0000000a: '14.0.0'
JIT (len=4 rc=0) 0x00000001: 0x00000000

There is no JIT for the M1 yet (it seems) and as we have seen, not all pcre2 libs are build with JIT. SWI-Prolog runs on a lot of hardware and old as well as new OS versions. We need to be pretty tolerant wrt. our dependencies.

peter.ludemann · April 6, 2022, 4:57pm

The problem seems to be that pcre2_config() returns PCRE2_ERROR_BADOPTION for PCRE2_CONFIG_JITTARGET if JIT support isn’t there (that is PCRE2_CONFIG_JIT returns 0).

I’ll need to make some small changes to re_confog_() to handle this properly and not introduce other potential problems (JW put a patch in the code that he cryptically (and correctly) marked as “dubious”); plus I need to make some changes to test_pcre.pl.

jan · April 6, 2022, 6:26pm

FYI, all tests still pass on pcre2 10.31 (amd64) I plan to release 8.5.10 shortly as there are quite a few important fixes. This does need to be resolved for that … Sorry …

peter.ludemann · April 6, 2022, 7:15pm

I should be able to do at least a quick fix for the tests by the end of today (Wed Apr 6, Pacific Daylight Time)

peter.ludemann · April 7, 2022, 4:32pm

My fix is intended to be “permanent”.

@mgondan1 - pleaes verify that it works for MinGW.

Topic		Replies	Views
Java/jpl cmake test errors General	20	2089	April 20, 2020
Rtools42, Unit test for sgml General	7	451	April 7, 2022
Debugging failing SSL test Installing	4	805	April 1, 2020
Automatic tests can succeed with syntax errors Discussion bug	10	865	May 8, 2021
Plunit fixme and assertion/1 Tools bug	1	533	January 8, 2022

Rtools42, Unit test for pcre

Related Topics