The documentation for normalize_space/2 says:
All non-empty sequences for Unicode white space characters are replaced by a single space (\u0020) character.
but
atom_codes(A, [0x00A0]),normalize_space(atom(X),A).
A = X, X = ’ '.
Thanks
Mike
Thanks. The space detection was reusing the Prolog parser separator detection. That detection changed with recent Unicode updates to follow the Unicode recommendations for separators in source code. Now it uses the locale dependent code_type/2 space class.
Thanks Jan. However this didn’t solve my problem and now I am confused.
You say “now it uses the locale dependent code_type/2 space class”.
The doc for code_type/2 says:
The types are sensitive to the active locale, see setlocale/3.
My locale: setlocale(ctype,A,A).
A = 'en_US.UTF-8’.
According to Google:
In the en_US.UTF-8 locale, there are exactly 29 characters recognized as white space under standard Unicode rules and POSIX wide-character classification (iswspace). [1, 2].
…
3. C1 Control and Latin-1 Spacing
Multi-byte spaces derived from the extended Latin-1 standard block. [1]
| Code Point | Character Name | UTF-8 Bytes |
|---|---|---|
U+0085 |
Next Line (NEL) | C2 85 |
U+00A0 |
No-Break Space (NBSP) | C2 A0 |
So I expect NBSP to be included but code_type/2 doesn’t agree:
code_type(A,space),format(‘0x~16R~n’,[A]),fail.
0x9
0xA
0xB
0xC
0xD
0x20
0x1680
0x2000
0x2001
0x2002
0x2003
0x2004
0x2005
0x2006
0x2008
0x2009
0x200A
0x2028
0x2029
0x205F
0x3000
It took a little while. With help from Claude the character classification is now detached from libc and controlled by our own tables generated from the Unicode data. This makes the classification locale independent and portable. As is, NBSP is classified as blank in BSD Unix (derived) systems and not in Linux. There is something to say for that, but I think it is a good idea that SWI-Prolog takes a platform independent decision (which is to make them blank).
Wow! The only way around was through. Thanks Jan.