Emacs on win11 with prolog-mode and swipl.exe and indic script output in the toplevel buffer

I am using emacs on win11 with prolog-mode and swipl.exe. I am trying to writeln devenagari script with:

?- string_codes(S, [0x0915, 0x094D, 0x0937, 0x093F]), writeln(S).

but get:

\u0915\u094D\u0937\u093F
S = \u0915\u094D\u0937\u093F.

but do not get the devanagari glyph. I have set the buffer to utf-8 without success.

the same correctly outputs the glyph outside emacs in a win11 cmd console.

क्षि

Copilot thinks this is a systemic limitation of my setup that cannot be overcome in win11.

Is this correct?

Thanks.

FWIW, this works fine for me on Linux (Chromebook) – in a terminal window, in an emacs shell, and with swipl-win. I don’t think I’ve done anything special in my emacs setup.

But, what does Prolog think about the encoding? Try

?- stream_property(user_output, encoding(E)).

On most non-Windows systems, text encoding defaults to UTF-8 these days. I think you can also setup Windows that way, but it is not default. You can change the encoding using set_stream/2 or set the Prolog flag encoding (this only applies to new streams).

1 Like

I got the following:
?- stream_property(user_output, encoding(E)).
E = text.

I have the following for prolog buffer in emacs startup, but it does not seem to take, I got the suggestion from chatGPT :frowning: so maybe it is only for unix environment.

(add-hook 'inferior-prolog-mode-hook
(lambda ()
;; Force UTF-8 encoding for SWI-Prolog process
(set-buffer-process-coding-system 'utf-8-unix 'utf-8-unix)))

I use win11 and have never used WSL so fear I may run into more problems setting that up.
It works for me in the CMD window with utf-8 encoding, and in swipl-win, but not in emac prolog-mode toplevel, or in emacs shell buffer.

That leaves the encoding to the C library, which on Windows probably implies using the current code page. Try

?- set_stream(user_output, encoding(utf8)).

And probably also for user_error and user_input. Not sure where to put this to make this permanent. One option is in your Prolog init.pl file. You probably want that only under Emacs, so you get e.g.

:- if(current_prolog_flag(emacs_inferior_process, true)).
:- set_stream(user_input, encoding(utf8)).
:- set_stream(user_output, encoding(utf8)).
:- set_stream(user_error, encoding(utf8)).
:- endif.
1 Like

Got the following:

?- set_stream(user_output, encoding(utf8)).
true.

?- string_codes(S, [0x0915, 0x094D, 0x0937, 0x093F]), writeln(S).
क्षि
S = क्षि.

?- stream_property(user_output, encoding(E)).
E = utf8.

copy/paste change the actual glyphs printed above but something garbled like that, I do have a font that supports that script.

Can you show the SWI-Prolog banner for more info?


For the SWI-Prolog version 9.2.9 on Windows 11 this appears correct.

Welcome to SWI-Prolog (threaded, 64 bits, version 9.2.9)
SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software.
Please run ?- license. for legal details.

For online help and background, visit https://www.swi-prolog.org
For built-in help, use ?- help(Topic). or ?- apropos(Word).

?- string_codes(S, [0x0915, 0x094D, 0x0937, 0x093F]), writeln(S).
क्षि
S = “क्षि”.


For the SWI-Prolog version 10.0.0 on Windows 11 this appears incorrect.

However pasting the copied text here does show the character correctly.

Welcome to SWI-Prolog (threaded, 64 bits, version 10.0.0)
SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software.
Please run ?- license. for legal details.

For online help and background, visit https://www.swi-prolog.org
For built-in help, use ?- help(Topic). or ?- apropos(Word).

4 ?- string_codes(S, [0x0915, 0x094D, 0x0937, 0x093F]), writeln(S).
क्षि
S = “क्षि”.


While this is a guess, in building SWI-Prolog on Windows 11 using MSVC, the ctest failed and Claude did focus on some utf-8 problems.

Only providing these mostly for Jan because they do point out C level code changes with details, in other words you won’t have to waste time on figuring out if this is of value. Still not sure myself if these will matter in the long run. Feel free to ignore.

read_pending_input_utf8_investigation.md (4.2 KB)

re_buffer_fix_summary.md (5.0 KB)


Before others ask about what the MSVC build shows

Using swipl.exe

Using swipl-win.exe

2 Likes

with Jan’s suggested additions to init.pl, emacs restart, M-x run-prolog I get the following
So it is “working” now.
I also made the following changes to emacs startup suggested by chatGPT:
Probably both emacs config and swipl config changes were needed.

;; UTF-8 everywhere
(set-language-environment “UTF-8”)
(prefer-coding-system 'utf-8-unix)

(setq default-process-coding-system '(utf-8-unix . utf-8-unix))
(setq locale-coding-system 'utf-8-unix)

1 Like

Not sure what the minimal config changes are.

The display indicates that the font does not know these code points. That is dictated by the resource font.pango_families. The default for mono on Windows is

Consolas,Courier New,monospace

Possibly there is a better default. Anyway, you can edit the GUI preferences. The system default file has this block:

! XPCE uses `sans`, `serif` and `mono` for all its named fonts
! using the table above.   Before these are passed to Pango, the
! table @font_families is consulted that is initialized from the
! resource below.  The system default table depends on the OS.
! You can use XPCE's font viewer to find a pleasant font and
! modify this table.  The value is a ,-separated list of Pango
! font families that are tried in order. "Noto" is the default
! on Linux.

! font.pango_families:	[ sans  := "Noto Sans,sans", \
!			  serif := "Noto Serif,serif", \
!			  mono  := "Noto Sans Mono,monospace" \
!			]