# Unicode symbols, and a possible numerical paradox!

**URL:** <https://swi-prolog.discourse.group/t/unicode-symbols-and-a-possible-numerical-paradox/5533>\
**Category:** Discussion\
**Created:** [June 21, 2022, 3:31pm UTC](https://swi-prolog.discourse.group/t/unicode-symbols-and-a-possible-numerical-paradox/5533 "2022-06-21T15:31:13Z")\
**Posts on this page:** 8\
**Page:** 2

<div class="post-metadata">

**Author:** ![peter.ludemann](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/peter.ludemann/32/48_2.png) [@peter.ludemann](https://swi-prolog.discourse.group/u/peter.ludemann)\
**Post date:** [June 25, 2022, 12:51am UTC](https://swi-prolog.discourse.group/t/unicode-symbols-and-a-possible-numerical-paradox/5533/38 "2022-06-25T00:51:30Z")

</div>

> [@kuniaki.mukai](#):
>
> I agree. But office workers might think differently, because now they may be busy toward paperless offices.

In general, it’s a mistake to depend on language features for normalizing data (even when setting the localisation environment). I would expect a Japanese-oriented application to normalize full-width numbers to ASCII (e.g., `１２３` would be normalized to `123`) and also to normalize 半角 (han-kaku) to 全角 (zen-kaku) (e.g. `ﾊﾝｶｶｸ` to `ハンカク`), and probably a lot of other normalizations. Presumably there are libraries for handling this. (Normalization rules for other languages have their own nuances; for example Québec and France have slightly different rules for capitalizing a name with accents, such as “Lenôtre”.)

---

<div class="post-metadata">

**Author:** ![kuniaki.mukai](https://avatars.discourse-cdn.com/v4/letter/k/c67d28/32.png) [@kuniaki.mukai](https://swi-prolog.discourse.group/u/kuniaki.mukai)\
**Post date:** [June 25, 2022, 2:27am UTC](https://swi-prolog.discourse.group/t/unicode-symbols-and-a-possible-numerical-paradox/5533/39 "2022-06-25T02:27:36Z")

</div>

I fully agree with your suggestion. In fact I have been prcticing in that way since a long time ago. I believe also most people are doing so. In fact, for example, it was widely advised by network administrators not to use half-width characters in the mail title, because of high risk on troubles on send/receiveing such series of chars.  
There might be many R & D topics related to encoding, which might be still active really. However I escaped from such important and heavy issues. I hope so as I was close to them in the sense of “social distance”. Charcter encoding did not look attractive for me, but now it may be more important technology than I thought. Meta font technology might concern glyphs, for example.

---

<div class="post-metadata">

**Author:** ![kuniaki.mukai](https://avatars.discourse-cdn.com/v4/letter/k/c67d28/32.png) [@kuniaki.mukai](https://swi-prolog.discourse.group/u/kuniaki.mukai)\
**Post date:** [June 25, 2022, 11:02am UTC](https://swi-prolog.discourse.group/t/unicode-symbols-and-a-possible-numerical-paradox/5533/42 "2022-06-25T11:02:33Z")

</div>

TexEdit seems to open both files in utf8 and utf16 without warings as you told. Thanks.

```prolog
% echo 漢字 > a 
% nkf -w16 a > b
% open -a TextEdit a
% open -a TextEdit b

```

---

<div class="post-metadata">

**Author:** ![kuniaki.mukai](https://avatars.discourse-cdn.com/v4/letter/k/c67d28/32.png) [@kuniaki.mukai](https://swi-prolog.discourse.group/u/kuniaki.mukai)\
**Post date:** [June 25, 2022, 12:32pm UTC](https://swi-prolog.discourse.group/t/unicode-symbols-and-a-possible-numerical-paradox/5533/44 "2022-06-25T12:32:51Z")

</div>

Do you see what you want. I see something without meaning 😅

```prolog
% hexdump a
0000000 bce6 e5a2 97ad 000a                    
0000007

% hexdump b
0000000 fffe 226f 575b 0a00                    
0000008
%

```

---

<div class="post-metadata">

**Author:** ![jan](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/jan/32/4_2.png) [@jan](https://swi-prolog.discourse.group/u/jan)\
**Post date:** [June 25, 2022, 3:05pm UTC](https://swi-prolog.discourse.group/t/unicode-symbols-and-a-possible-numerical-paradox/5533/46 "2022-06-25T15:05:47Z")

</div>

> [@anon95304481](#):
>
> [SWI-Prolog](https://www.swi-prolog.org/pldoc/doc_for?object=manual) could also maybe find a good soul, that would add a SHIFT-JIS implementation to this list here, if there is a demand for it? Remark: Encodings might also affect \< 8 bit.

Roughly, that would require making something like libiconv or librecode part of the system. While this is typically installed already on most Unix like systems, it is a big new dependency, about as big as SWI-Prolog itself ☹

When I added Unicode support I went for built-in support for ISO Latin 1, UTF-8 and USC-2 (big and little endian). In addition there is the `text` encoding, which uses whatever the locale tells the multibyte routines of the C runtime to support. So, you have always access to files in your default locale, UTF-8 and USC-2 files. As fairly all non-Windows system already use UTF-8 as their primary encoding, this seems the right direction. Windows seems to be moving in the same direction.

Once upon a time UCS-2 should be extended to UTF-16, i.e. we must support surrogate pairs. As is, this is not that useful as we find UTF-16 mostly on Windows and SWI-Prolog does not support code points \> 0xffff on Windows.

---

<div class="post-metadata">

**Author:** ![jan](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/jan/32/4_2.png) [@jan](https://swi-prolog.discourse.group/u/jan)\
**Post date:** [June 26, 2022, 8:23am UTC](https://swi-prolog.discourse.group/t/unicode-symbols-and-a-possible-numerical-paradox/5533/50 "2022-06-26T08:23:56Z")

</div>

> [@anon95304481](#):
>
> ```prolog
> ?- X = 'Flower 💐', Y = 'Flower \xd83d\\xdc90\', X == Y.
> false.
> 
> ```

On a Unix system the console produces a multibyte string in the current locale encoding, these days almost invariable UTF-8. That is read and represented as `wchar_t` array, which is effectively UCS-4 on anything but Windows. An atom is sequence of Unicode code points, and thus query boils down to

```
?- X = 'Flower \U0001f490', Y = 'Flower \xd83d\\xdc90\', X == Y.

```

On Windows the story is different because `wchar_t` is 2 bytes using UTF-16 encoding. SWI-Prolog doesn’t know about this and interprets it as UCS-2, considering the surrogate pairs as characters as reserved/unassigned.

It might make sense to stop escaping the surrogate pairs. It is still fundamentally wrong though. An atom should logically be a sequence of code points, not some internal encoding. It is not really clear whether stopping to escape surrogate pairs makes things better. It looks better, but hides the internal problem.

The real solution is to get rid of the UCS-2 on Windows. There are two obvious routes for that: go UTF-8 on all platforms internally or go UCS-1+UCS-4 as we use on Unix also on Windows. For most of the code using UTF-8 is probably the simplest way out and, as it makes all platforms internally the same, probably also the safest route. It also seems to be the choice of most new languages/implementations. Luckily SWI-Prolog has always hidden encoding from the user, so we can change it without affecting applications.

The main problem are predicates such as sub\_atom/5. These get seriously slower on long atoms/strings and more complicated. Using UTF-16 on Windows is another route, introducing the same complications for sub\_atom/5 and other routines that handle strings as an array of code points.

---

<div class="post-metadata">

**Author:** ![jan](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/jan/32/4_2.png) [@jan](https://swi-prolog.discourse.group/u/jan)\
**Post date:** [June 26, 2022, 10:49am UTC](https://swi-prolog.discourse.group/t/unicode-symbols-and-a-possible-numerical-paradox/5533/52 "2022-06-26T10:49:21Z")

</div>

> [@anon95304481](#):
>
> Does it silently only take the lower 16-bit on Windows from  
> the given code value? It seems so:

That made me wonder. The case conversion uses towupper() and towlower(). These take `wint_t` input and output. Now, on Windows `sizeof(wint_t)` is guess guess guess … yes! 2 bytes! That is the case for all the isw\* and tow\* functions. So, none of the Windows C runtime wide character routines can reason about code points \> 0xffff. Here surrogate pairs are not going to help ☹

The native Windows API provides CharLowerW() which takes a wchar\_t\*, with the very strange semantics that if the pointer is \<= 0xffff the argument is taken to be a character, returning a “pointer” that actually contains a code point … That doesn’t help either. Maybe it works correctly if you actually use the string API and give it a surrogate pair? The docs do not give a hint and seem to suggest Microsoft is not aware of surrogate pairs. Well, I know it is not _that_ bad.

There is more wrong with Windows than one would hope ☹

---

<div class="post-metadata">

**Author:** ![SHYAM](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/shyam/32/4492_2.png) [@SHYAM](https://swi-prolog.discourse.group/u/SHYAM)\
**Post date:** [June 27, 2022, 7:27pm UTC](https://swi-prolog.discourse.group/t/unicode-symbols-and-a-possible-numerical-paradox/5533/54 "2022-06-27T19:27:09Z")

</div>

```latex
\documentclass[12pt]{article}

\usepackage{xltxtra}
\usepackage[none]{hyphenat}
\usepackage{hyperref}
\usepackage{tikz}
\usepackage{amssymb}
\usepackage{polyglossia}
 \setmainlanguage{english}
 \setotherlanguages{sanskrit}
 \newfontfamily\devanagarifont[Script=Devanagari]{Noto Serif Devanagari}

\newcommand{\zdn}[1]{\begin{sanskrit}#1\end{sanskrit}}
\newcommand{\zcode}[1]{\textasciigrave{}#1\textasciigrave{}}
\newcommand{\crossschwa}[1]{\lower.05cm\hbox{\rotatebox{45}{\begin{tikzpicture}[#1]
 \draw (-1,1) -- (-1,0) -- (1,0) -- (1,-1);
 \draw (-1,-1) -- (0,-1) -- (0,1) -- (1,1);
 \fill (-0.5,-0.5) circle[radius=2.25pt];
 \fill (-0.5,0.5) circle[radius=2.25pt];
 \fill (0.5,-0.5) circle[radius=2.25pt];
 \fill (0.5,0.5) circle[radius=2.25pt];
\end{tikzpicture}}}} %https://tex.stackexchange.com/q/263269

\title{issue(115).}
\author{\crossschwa{scale=0.16} \zdn{\char"0936\char"094D\char"092F\char"093E\char"092E} \crossschwa{scale=0.16}}
\date{HE 12022.06.28}

\begin{document}
\maketitle
\abstract{\zdn{\char"0950} In this \XeLaTeX{} document, I present my work in brainstorming how Prolog handles Indic scripts; for a second time, in response to ``the daily build swipl-w64-2022-06-24.exe''. Noto Serif Devanagari is the best font in the Raspberry Pi repos. All content used for academic purposes.}
\tableofcontents

\section{I remain skeptical \#commentreasure} % @the_speed_of_light @lightspeed @ftl}
% ^^^^^ ^ ^ ^
% │││││ │ │ └ some incursive recursive madness i'll take care of later; make a date 'n' every"-thing" self settles
% │││││ └───────────┴ my future selves's optimised models
% ││││└ my derelict dialect; codenamed "mirage", pronounced "मी रैज् ॥"
% ││└┼ me cast adrift; "are we there yet?"@log,_like_a_pro;_err…
% │└─┴ space
% └ airlock
This seems shallow.
What happens for \texttt{`is/2`}?
I can't imagine anyone wanting to input one script, and outputting another; from a hacking perspective.
My RFC was they should be equal (\texttt{`==/2`}), but should not unify (\texttt{`=/2`}); now I realise this is a deeper issue, but the (more) informative error message from my RFC still stands, or the inconsistency from the prior section looks buggy, but Perl taught me bugs are byproducts of features, and now I've adapted this philosophy to Prolog.

\subsection{Compromisation}
My compromise is to just use Prolog the same as before I started learning Sanskrit, and do some"-thing" along the lines of this:
\\\texttt{devnum(D,N) :- ground(N), nth0(N,[}\zdn{०}\texttt{,}\zdn{१}\texttt{,}\zdn{२}\texttt{,}\zdn{३}\texttt{,}\zdn{४}\texttt{,}\zdn{५}\texttt{,}\zdn{६}\texttt{,}\zdn{७}\texttt{,}\zdn{८}\texttt{,}\zdn{९}\texttt{],D). \%commentreasure} %D never needs grounding, since D becomes N through unification; I'm not convinced this will even work as intended, but was planning on using atom_chars(Atomic_D,Lithp_D)

\section{I reject your' reality, and substitute my own!!! :D \#eject\_2:\_infinity,\_and\_beyond!!!\_:D}
I'm going to hack my own self-hosted dialect; reinventing each, and every, (key) component part of each, and every, wheel, while inventing some new ones, and \texttt{s/getting/setting/} the time right (this time), whilst I am, locatively, at it; future proofing it, so it is (well, and truly) ``good for over a hundred years'', or even ``good forever''; even after the heat death of the multiverse, and into the following cyclic multiverses.

\end{document}

```

[Previous page](https://swi-prolog.discourse.group/t/unicode-symbols-and-a-possible-numerical-paradox/5533.md?page=1)