# UTF-8 encode and decode (Discussion) **URL:** \ **Category:** Discussion\ **Created:** [February 21, 2020, 8:14am UTC](https://swi-prolog.discourse.group/t/utf-8-encode-and-decode-discussion/3209 "2020-02-21T08:14:33Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![jan](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/jan/32/4_2.png) [@jan](https://swi-prolog.discourse.group/u/jan)\ **Post date:** [February 21, 2020, 8:14am UTC](https://swi-prolog.discourse.group/t/utf-8-encode-and-decode-discussion/3209/1 "2020-02-21T08:14:33Z")
Can you create a PR to add them to the tests in `src/Tests/library/`? The more tests the better! ---
**Author:** ![pmoura](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/pmoura/32/11_2.png) [@pmoura](https://swi-prolog.discourse.group/u/pmoura)\ **Post date:** [February 22, 2020, 12:51pm UTC](https://swi-prolog.discourse.group/t/utf-8-encode-and-decode-discussion/3209/2 "2020-02-22T12:51:31Z")
QuickCheck may be useful here. Using Logtalk 3.35.0 with SWI-Prolog 8.1.22 to test `utf8_codes//1` on macOS 10.14.6: ``` ?- {lgtunit(loader)}. ... % (0 warnings) true. ?- [user]. |: property(Atom) :- |: atom_codes(Atom, Codes), |: phrase(utf8_codes(Codes), UTF8), |: phrase(utf8_codes(Codes2), UTF8), |: Codes == Codes2. |: ^D% user://1 compiled 0.01 sec, 1 clauses true. ?- lgtunit::quick_check(property(+non_empty_atom(unicode_bmp)), [n(1000)]). % 1000 random tests passed true. ?- lgtunit::quick_check(property(+non_empty_atom(unicode_full)), [n(1000)]). % 1000 random tests passed true. ``` According to Jan observations, the second test is expected to fail when running on Windows, correct? Anyone on Windows that can try the queries above? @EricGT maybe you can try something similar with `plunit` + `quickcheck` pack? ---
**Author:** ![EricGT](https://avatars.discourse-cdn.com/v4/letter/e/f1d935/32.png) [@EricGT](https://swi-prolog.discourse.group/u/EricGT)\ **Post date:** [February 22, 2020, 1:03pm UTC](https://swi-prolog.discourse.group/t/utf-8-encode-and-decode-discussion/3209/3 "2020-02-22T13:03:37Z")
> [@pmoura](#): > > According to Jan observations, the second test is expected to fail when running on Windows, correct? Not sure exactly which test you mean? If you are asking if any Unicode code point greater than U+FFFF will cause an error on Windows with SWI-Prolog when trying to be created as a character or in a string, then that is what I am finding. ([ref](https://swi-prolog.discourse.group/t/unicode-code-point-for-u-10000-use-in-string-results-in-illegal-character-code/1896)) But as I noted in this [reply](https://swi-prolog.discourse.group/t/unicode-code-point-for-u-10000-use-in-string-results-in-illegal-character-code/1896/3), using SWI-Prolog on Ubuntu on WSL on Windows is working for Unicode code points greater than U+FFFF as far as I can tell, but I literally just started looking into this. > [@pmoura](#): > > @EricGT maybe you can try something similar with `plunit` + `quickcheck` pack? OK, I’ll take a look. EDIT I tried to install the package [quickcheck](https://www.swi-prolog.org/pack/list?p=quickcheck) but it doesn’t find MinGW for some reason, and I don’t want to spend the day chasing this problem down. Running this by hand did work. ```prolog ?- mingw_root(X). Correct to: "prolog_pack:mingw_root(X)"? yes X = 'C:/MinGW'. ``` 🙁 ---
**Author:** ![pmoura](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/pmoura/32/11_2.png) [@pmoura](https://swi-prolog.discourse.group/u/pmoura)\ **Post date:** [February 22, 2020, 3:43pm UTC](https://swi-prolog.discourse.group/t/utf-8-encode-and-decode-discussion/3209/4 "2020-02-22T15:43:54Z")
> [@EricGT](#): > > Not sure exactly which test you mean? The one with the `unicode_full` argument. P.S. The main reason I suggested QuickCheck was that I noticed your hard-coded test cases (defined using the `unicode_string_utf8_test_case_generator/2` and `unicode_utf8_test_case_generator/2` predicates). ---
**Author:** ![EricGT](https://avatars.discourse-cdn.com/v4/letter/e/f1d935/32.png) [@EricGT](https://swi-prolog.discourse.group/u/EricGT)\ **Post date:** [February 22, 2020, 4:02pm UTC](https://swi-prolog.discourse.group/t/utf-8-encode-and-decode-discussion/3209/5 "2020-02-22T16:02:25Z")
Glad you noticed. My view on [them](https://github.com/SWI-Prolog/swipl-devel/blob/a2fa41e5d2a01092ec26a64a1d6173fd349c3240/src/Tests/library/test_utf8.pl) at present is they are better than what we had which was nothing, and that they are just a start for checking on Unicode with SWI-Prolog. If your goal is to add more test cases in this area to the master source code then we are on the same path. 🙂 ---
**Author:** ![pmoura](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/pmoura/32/11_2.png) [@pmoura](https://swi-prolog.discourse.group/u/pmoura)\ **Post date:** [February 22, 2020, 6:42pm UTC](https://swi-prolog.discourse.group/t/utf-8-encode-and-decode-discussion/3209/6 "2020-02-22T18:42:30Z")
I’m planning to add some QuickCheck-based tests to the Logtalk distribution that verify Unicode support in backend Prolog systems. Other Prolog systems already run the Logtalk tests suites routinely. SWI-Prolog can do it too as a second line of testing. I do it myself everytime I commit and push to GitHub using the SWI-Prolog stable Docker image. Adding the specific tests I posted to the master source would require, however, solving the odd issue with the `quickcheck` pack you mentioned and defining the types used (which are parametrized by the char set). ---
**Author:** ![jan](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/jan/32/4_2.png) [@jan](https://swi-prolog.discourse.group/u/jan)\ **Post date:** [February 23, 2020, 11:16am UTC](https://swi-prolog.discourse.group/t/utf-8-encode-and-decode-discussion/3209/7 "2020-02-23T11:16:48Z")
> [@pmoura](#): > > Other Prolog systems already run the Logtalk tests suites routinely. SWI-Prolog can do it too as a second line of testing. That makes sense. Last time I looked at it (some years ago), my complaint was that running the test suite is pretty verbose and it fails on several ISO tests where I deliberately decided not to follow the ISO standard. Going through that once is fine, but for routine testing I just want an _all passed_ or _test XYZ failed_ and otherwise pretty quiet operation. Is that (now) possible? CMake’s `ctest` has that pretty neatly solved, were ctest only prints some summary and progress reports for normal operation and the full output for a failing test. It also runs tests concurrently, so I can run the whole suite in 10 seconds elapsed time on a quad core i7 machine. Part of the speed is due to ctest starting multiple Prolog processes where each process runs a fair number of individual tests controlled by `plunit`. I’m glad you quickly run the tests on suspicious updates from my side. It did good work for the rational syntax. Otherwise the coverage of the bundled tests seems pretty much ok as your testing rarely finds something. ---
**Author:** ![pmoura](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/pmoura/32/11_2.png) [@pmoura](https://swi-prolog.discourse.group/u/pmoura)\ **Post date:** [February 23, 2020, 6:34pm UTC](https://swi-prolog.discourse.group/t/utf-8-encode-and-decode-discussion/3209/8 "2020-02-23T18:34:08Z")
> [@jan](#): > > That makes sense. Last time I looked at it (some years ago), my complaint was that running the test suite is pretty verbose … The _default_ output (from the `logtalk_tester` automation script) is verbose (but made of 100% recycled bits). But you also have TAP and xUnit output options. The xUnit output in particular can be easily converted into HTML (e.g. [https://logtalk.org/files/blog/2019\_11\_06/xunit\_report.html](https://logtalk.org/files/blog/2019_11_06/xunit_report.html) which provide a nice way to dig into the test results). Note that all output from `lgtunit` uses the message printing mechanism and thus easy to customize (that’s how TAP and xUnit outputs are implemented). > [@jan](#): > > … and it fails on several ISO tests where I deliberately decided not to follow the ISO standard. You don’t need to run all test suites. E.g. when using TravisCI via GitHub to check Logtalk commits using SWI-Prolog stable Docker image, the Prolog compliance suite is not run (see the [https://github.com/LogtalkDotOrg/logtalk3/blob/3b77b374128ee69a4ede0faccf857a490fbb2cb8/.travis.yml](https://github.com/LogtalkDotOrg/logtalk3/blob/3b77b374128ee69a4ede0faccf857a490fbb2cb8/.travis.yml) file). > [@jan](#): > > Going through that once is fine, but for routine testing I just want an _all passed_ or _test XYZ failed_ and otherwise pretty quiet operation. Is that (now) possible? One option would be to divert the output of the `logtalk_tester` automation script (to a file or `dev/null`) and use its exit status (see [logtalk\_tester man page](https://logtalk.org/man/logtalk_tester.html) for the different values and their meanings) to tell if all tests pass or what kind of failure was found and, in the later case, just browse the generated HTML report(s). This is easy to script. > [@jan](#): > > CMake’s `ctest` has that pretty neatly solved, were ctest only prints some summary and progress reports for normal operation and the full output for a failing test. It also runs tests concurrently, so I can run the whole suite in 10 seconds elapsed time on a quad core i7 machine. Part of the speed is due to ctest starting multiple Prolog processes where each process runs a fair number of individual tests controlled by `plunit` . Running Logtalk test suites concurrently is supported. See the [https://github.com/LogtalkDotOrg/logtalk3/blob/3b77b374128ee69a4ede0faccf857a490fbb2cb8/library/parallel\_logtalk\_processes\_setup.pl](https://github.com/LogtalkDotOrg/logtalk3/blob/3b77b374128ee69a4ede0faccf857a490fbb2cb8/library/parallel_logtalk_processes_setup.pl) file for the required configuration. ---
**Author:** ![pmoura](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/pmoura/32/11_2.png) [@pmoura](https://swi-prolog.discourse.group/u/pmoura)\ **Post date:** [March 3, 2020, 4:00pm UTC](https://swi-prolog.discourse.group/t/utf-8-encode-and-decode-discussion/3209/9 "2020-03-03T16:00:59Z")
> [@jan](#): > > That makes sense. Last time I looked at it (some years ago), my complaint was that running the test suite is pretty verbose In the just released Logtalk 3.36.0 version, you can do: ```bash $ logtalk_tester -o minimal ``` To run test suites concurrently: ```bash $ logtalk_tester -i '-f /path/to/library/parallel_logtalk_processes_setup.pl' ```