Ann: SWI-Prolog 10.1.6

[jan]

Dear SWI-Prolog user,

SWI-Prolog 10.1.6 is ready for download. The main theme of this release
is improved Unicode support, from the stream layer all the way up to the
terminal. Highlights:

Building!

• There is a new dependency. Our unmaintained copy of utf8proc.c, underlying library(unicode) has been removed. All major platforms now support this package and thus we use the platform version. Install using

  • Debian based systems: apt-get install libutf8proc-dev
  • Redhat/Fedora based systems: dnf install utf8proc-devel
  • Macports: port install utf8proc

Core

• Unicode-aware stream position. Supdatepos() now counts East-Asian
  wide characters (CJK, emoji) as two columns and combining marks as
  zero, using wcwidth() (with a bundled mk_wcwidth.c fallback on
  platforms that lack it). format/2’s tab-stop arithmetic and the
  new write_size/4 predicate inherit the correct column widths.
• New write_size(+Term, -Width, -Height, +Options) replaces the
  now-deprecated write_length/3. It reports both the longest
  output line and the number of lines a term would occupy when
  printed, respects Unicode column widths, and accepts
  max_width(+N) / max_height(+N) options that bound the runtime
  on huge terms by aborting the write as soon as a threshold would
  be exceeded.
• code_type(Code, width(W)) exposes wcwidth() at the Prolog level.

Library(unicode) / package utf8proc

• The package no longer vendors its own copy of utf8proc.c; it now
  depends on the external libutf8proc shipped with every major
  distro (and built as a shared library in the flatpak and MinGW
  images). Distributions that ship an older libutf8proc still
  compile and run: feature probes in the package’s CMakeLists.txt
  gate optional fields and functions behind HAVE_UTF8PROC_*
  macros.
• unicode_property/2 grows width, ambiguous_width,
  bidi_mirrored, uppercase/lowercase/titlecase, and
  indic_conjunct_break properties; drops the rarely useful
  comp_exclusion and control_boundary.
• New predicates atom_graphemes/2, string_graphemes/2,
  unicode_version/1, unicode_codepoint_valid/1,
  unicode_casefold/2, and unicode_nfkc_casefold/2.
• Full PlDoc rewrite and a regression suite (80 plunit tests).

Terminal (XPCE / libedit)

• A unified Unicode-aware cell model in the xpce terminal, so
  grapheme clusters (NFD letters, emoji ZWJ sequences,
  regional-indicator flags) render and navigate as single
  user-perceived characters. Many related fixes: resize reflow,
  caret placement across combining marks and wide characters,
  DCH/ICH column arithmetic, and rendering of long NFD+emoji runs.
• libedit uses bracketed-paste mode — pasting long text no longer
  triggers TAB completion, and is considerably faster on Windows.
• On Windows, non-BMP code points (emoji, supplementary-plane CJK)
  now survive the UTF-16 round trip through ReadConsoleInput,
  el_wgetc, and el_gets unharmed.
• SIGWINCH now correctly rewraps the current input line at the
  new column count, in both xterm-style and xpce terminals.

XPCE, modernisation

• The SDL3 backend continues to replace X11 bindings. This release
  removes a large set of X-only methods (frame->hide,
  ->grab_pointer, ->open, …); adds SDL3 implementations for
  image<-rotate, image<-scale, window->flash, reading/saving
  images from/to streams, SVG loading, and more; and ships SVG
  icons for most development tools.

Assorted fixes

• Read buffer reallocation bug in <base>'.. numeric literals
  (#1483).
• GC crash when a nondeterministic foreign predicate triggered a
  constraint wakeup.
• spy/1 again enables debug mode.
• listing/2: missing declaration of the source(+Boolean) option.
• WASM class/property name conversion under Unicode.

Enjoy — Jan

SWI-Prolog Changelog since version 10.1.5

• CI: Added utf8proc as dependency

• ENHANCED: mkchangelog stops changelog entries at the first blank line
  A blank line in a commit body now terminates that commit’s changelog
  entry. Subject + first body paragraph become the entry; deeper detail
  stays visible in git log but is hidden from the generated changelog.

• TEST: write_size/4 — max_width, max_height, Unicode combining,
  wide chars New write_size unit in tests/core/test_write.pl covers:

• ADDED: write_size(+Term, -Width, -Height, +Options) write_size/4
  replaces the now-deprecated write_length/3. It returns both the
  maximum line Width and the Height (line count) the term would occupy
  when printed with write_term. Options max_width(+N) and max_height(+N)
  cap the computation — the underlying Swrite_lss aborts as soon
  as either threshold would be exceeded, bounding the runtime for
  huge terms.

• CLEANUP: pl_write_term refactored to PRED_IMPL; foreign_t → bool
  Convert pl_write_term2 / pl_write_term3 (and pl_print2) from the
  legacy foreign_t/pl_* calling convention to PRED_IMPL, and change
  pl_write_term3’s signature to return bool. Also drop its entries
  from the foreigns table in pl-ext.c now that they are registered
  via PRED_DEF in pl-write.c.

• CLEANUP: bool return types for pl-text and pl-string APIs Convert
  int → bool for PL_unify_text_range, PL_promote_text, PL_mb_text,
  PL_canonicalise_text_ex, PL_concat_text, PL_save_text, PL_text_recode,
  and the return path of several PRED_IMPL functions in pl-string.c.
  Introduce get_text_t / cmp_t enums for the tri-valued get/compare
  cases so the type tells the reader what to expect instead of relying
  on int conventions.

• ENHANCED: Make format/2 position logic aware of Unicode column logic.

• ADDED: Unicode-aware Supdatepos() using wcwidth() Supdatepos()
  now counts double-width characters (CJK, emoji) as 2 columns and
  combining marks as 0, via wcwidth(). On systems lacking a native
  wcwidth() we fall back to the embedded mk_wcwidth.c from Markus Kuhn’s
  public-domain implementation; src/CMakeLists.txt adds it to SRC_CORE
  when HAVE_WCWIDTH is false.

• ADDED: code_type/2: code_type(Code, width(W)) Provides an interface
  to wcwidth().

• UNICODE: Fixed reading in DerivedCoreProperties.txt for Unicode-17.0.
  Required to update our tables to Unicode version 17.

• FIXED: Missing declaation of listing/2 for option source(+Boolean)

• FIXED: #1483 <base>' numbers can be misread. A possible read
  buffer reallocation can cause misreading the base and result in a
  syntax error or wrong value.

• FIXED: WASM: Convert class and property names with Unicode.

• CLEANUP: Remove sigaction configuration

• FIXED: GC error If a non-deterministic foreign predicate triggers a
  constraint wakeup.

• FIXED: spy/1 to enable debug mode again

• PORT: Fix cmake SDL3 import handling for non-standard configurations
  (PGO)

Package debian

• ADDED: libutf8proc-dev as new dependency.

Package libedit

• FIXED: swipl.exe console uses ReadConsoleInputW (Unicode) for input
  libedit4pl.c’s Windows console read_char called ReadConsoleInput,
  which without UNICODE/_UNICODE defined expands to the ANSI variant
  ReadConsoleInputA. The A variant fills uChar.AsciiChar as a CP_437
  / CP_ACP byte. We then read uChar.UnicodeChar from the same union,
  which gets at best a low-byte-truncated value (the A-call leaves the
  high byte of UnicodeChar undefined; in practice it’s 0, so the byte
  value is read as a BMP code point of the same numeric value).

• FIXED: pl_getc reassembles UTF-16 surrogate pair on Windows
  editline.pl’s bracketed_paste/3 handler collects pasted bytes
  with el_getc/2 (foreign pl_getc) and then calls string_codes/2 on
  the result. On Windows wchar_t is a 16-bit UTF-16 code unit, so a
  supplementary code point (emoji like , most non-BMP CJK) lives in
  the line buffer as a surrogate pair and el_wgetc returned each half
  as its own int. The resulting code list contained lone surrogates,
  and string_codes/2 aborted with

• ENHANCED: libedit4pl.c — reassemble supplementary code points on
  read (Windows) On Windows wchar_t is a 16-bit UTF-16 code unit, but
  read_char read one UTF-8 code point at a time and stored it directly
  in a wchar_t:

• ADDED: Use “bracketed paste mode” in the terminal. This avoids
  interpreting TAB as completion and speeds up long paste in Windows.

Package pldoc

• ENHANCED: doc_latex translates common Unicode punctuation to LaTeX
  pdflatex under T1 encoding rejects non-ASCII bytes, so any PlDoc
  comment containing an em-dash, arrow, fancy quote or ellipsis produced
  hundreds of “non-ASCII character” warnings and the glyph often did
  not render.

Package protobufs

• CLEANUP: reformat with pprint:print_term (changed whitespace)

• CLEANUP: fix some typos

Package ssl

• PORT: Update for openssl 4.0

• CLEANUP: Remove dead conditional code. Now only support OpenSSL
  1.1.1 and LibreSSL.

Package utf8proc

• BUILD: Probe optional libutf8proc features and gate on HAVE_UTF8PROC_*
  Older distro libutf8proc (Ubuntu LTS) ships a version that
  predates charwidth, ambiguous_width, indic_conjunct_break,
  the EXTENDED_PICTOGRAPHIC boundclass, and single-codepoint case
  mappings. The binding now probes those at configure time via
  check_c_source_compiles / check_symbol_exists and emits HAVE_UTF8PROC_*
  macros in config.h; unicode_property/2 branches and the associated
  foreign registrations are guarded by the matching #ifdef, so the
  binding compiles against any utf8proc version from 2.4 onwards.

• TEST: Full regression suite for library(unicode) Replace the stub
  test_utf8proc.pl with a plunit suite covering the features introduced
  in the previous commit:

• ENHANCED: Review unicode_property/2, add grapheme and casefold APIs
  Rewrite unicode4pl.c around the current utf8proc property struct
  and use SWI-Prolog’s public PL_type_error / PL_domain_error /
  PL_resource_error helpers and bool return types.

• BUILD: Use external libutf8proc via pkg-config The package used to
  bundle utf8proc.c (plus its generated utf8proc_data.c), a local
  Makefile, a Ruby gem wrapper, and PostgreSQL binding code —
  all inherited from a pre-Julia upstream that no longer matches the
  maintained library. utf8proc is now maintained by the Julia project
  at GitHub - JuliaStrings/utf8proc: a clean C library for processing UTF-8 Unicode data · GitHub and ships in every major
  package manager, so there is no reason to vendor it here.

Package xpce

• FIXED: rlc_resize, rlc_shift_lines_up, rlc_clear_from_cursor +
  tests Three related fixes to the xpce ring-buffer terminal and a new
  regression-test unit that exercises them:

• FIXED: terminal paints each wide-char cluster at its column-grid
  x Pango’s natural advance for an emoji (or other wide glyph) runs
  a pixel or two wider than the 2*cw the terminal grid allocates.
  Feeding a long same-flags run like “sskll​”
  as a single s_print_utf8 call made each emoji nudge the subsequent
  glyphs rightward, so the rendered text drifted past the column-grid
  cursor that rlc_paint_text walks in integer cell units. With the
  caret walked to the end the text and the caret no longer meet, and
  a selected “see​” left the trailing “aap” shifted left.

• FIXED: s_print_utf8 reads baseline from the new text, not the stale
  layout s_print_utf8 was calling pango_layout_get_baseline BEFORE
  pango_layout_set_text, so baseline reflected whatever text was in
  the cached PangoLayout from the previous call. Consecutive calls
  with different scripts (e.g. an emoji glyph followed by plain ASCII)
  baseline-aligned each draw against the previous call’s text, so
  the second draw could land a few pixels above or below the intended
  baseline.

• FIXED: raise META_OFFSET above Unicode range META_OFFSET (used to
  tag Meta-prefixed key events in the same integer space as character
  code points) was 1<<16 = 65536. Any non-BMP code point (emoji like
  U+1F612, most supplementary-plane characters) is >= 65536 and
  collided with the Meta flag.

• TEST: terminal_non_bmp — exercise supplementary-plane code points
  Add five tests covering the non-BMP path that matters most on Windows,
  where wchar_t is 16-bit UTF-16 and a single code point (U+10000 and
  above) occupies two buffer slots as a surrogate pair. Linux stores
  the same code point in one wchar_t; these tests pin both platforms
  to the same observable behaviour:

• TEST: extend random-test model to multi-row layout Teach
  test_terminal_random’s pure-Prolog model to lay clusters out across
  rows (model_layout/8) so it can verify content that legitimately wraps.
  Rows are grouped by the row they occupy and each row is compared
  to xpce’s <-row. The per-session content cap is lifted from one
  row to 900 visual columns so random sessions routinely exercise the
  libedit+xpce wrap path.

• TEST: regression for re_refresh wide-at-cursor using visual column
  Locks in the libedit fix: long NFD content + emoji at cursor must
  not bump the computed cursor row based on code-point count.

• FIXED: rlc_insert must not clamp tl->size to b->width rlc_insert
  extended the line’s cell size only up to b->width when opening a gap
  for the inserted character. For a line whose content is NFD-heavy
  (combining marks occupy cell slots but zero visual columns) the cell
  count can legitimately exceed b->width while the visible content still
  fits, and clamping trimmed the trailing cells — most visibly the
  final cluster’s combining mark, which disappeared from the display
  after a mid-line insert.

• FIXED: ANSI column positioners must not clamp caret_x to width-1
  rlc_set_caret (CUP / HVP) and rlc_set_caret_x (CHA / HPA) clamped the
  resulting caret_x to b->width-1 after converting the requested visual
  column via rlc_vcol_to_cell. For plain narrow content that’s a no-op,
  but for rows containing combining marks or wide characters the cell
  index at visual column K legitimately exceeds K (a base + combiner
  occupies two cells for one column). The clamp silently snapped the
  caret back to the middle of the row, so libedit’s re_refresh_cursor
  —which uses CHA to position after cross-wrap moves — landed at
  the physical midpoint of an NFD-filled line instead of its right edge.

• FIXED: count wide-char clusters as 2 visual columns in DCH
  rlc_delete_chars decremented its visual-column budget by the base
  cell’s per-cell width (1), which is wrong for a wide char: the cluster
  spans base + placeholder = 2 visual columns. So a DCH with count=2
  only removed one wide cluster plus a character from the next cluster,
  instead of a single wide cluster.

• FIXED: preserve cell content when rlc_put lands a combiner mid-line
  rlc_put’s combining-mark branch used to write the combiner into the
  cell at caret_x and advance. For a re-render that inserts an NFD
  cluster with libedit’s T_IC + overwrite, the overwrite’s combining
  mark clobbered the base that T_IC had just shifted into place,
  dropping the following character from the visible row.

• TEST: add test_terminal_random/2,3 and first regression it surfaced
  Introduces a random driver that types and edits through libedit and
  after each command verifies the cursor and row text against a pure-
  Prolog model of where they should be. Single-row v1 (buffer capped
  to fit on one row). On the first divergence the driver prints the
  seed, the full session command history, and the expected/actual state,
  then throws.

• TEST: Added terminal test framework.

• ADDED: epilog/1 option object(-Ref) Get the xpce object reference of
  the new terminal.

• ENHANCED: Unicode-aware terminal cell model Teach the xpce terminal’s
  line buffer to store Unicode content by code point, with per-cell
  display width, and to move the caret by grapheme clusters or by visual
  columns as the context requires. Summary of the model:

• ADDED: terminal_image<-cursor_position and terminal_image<-row
  This allows for writing terminal handling tests.

• ADDED: text_item->placeholder Modernizing the UI

• ADDED: library(pce_backcomp): Provide better backward compatibility

• MODIDIED: Renamed ->fill_pattern to ->fill Only filling with
  colours is currently supported. Even if we re-introduce filling with
  images, the name fill_pattern is not accurate.

• MODIFIED: Use .SVG images for most of the icons of the development
  tools This patch replaces most 16x16 .PNG image files with .SVG files,
  most of which are scaled to the context, e.g., the line height.
  Furthermore

• ADDED: Support loading image objects from SVG data

• FIXED: Windows display<-selection to return canonical newlines.
  This patch also fixes a memory leak: the selection string as obtained
  from SDL3 was not freed. The leak applies to all platforms.

• FIXED: figure->shadow rendering.

• FIXED: image<-clip and image<-scale to same size Preserve
  transparency of the source.

• ADDED: Support loading images from SVG files. This uses the SDL-Image
  library support for nano-svg. This is a subset of SVG.

• MODIFIED: Deleted image<-monochrome. This is a bit outdated.
  Use <-grayscale instead.

• SDL: Implemented image<-rotate.

• ENHANCED: Allow resizing tool button images.

• SDL: Implemented image<-scale.

• ENHANCED: Several PceDraw issues.

• ENHANCED: Default arrow size.

• FIXED: ellipse->shadow drawing.

• ENHANCED: Slider look handling. Now implements gtk and motif
  Deleted old open_look and win (black-and-white) looks.

• FIXED: Pen±fill handling for box and arc/ellipse

• FIXED: Properly draw 3D elevated triangles

• ENHANCED: sizing 3D triangles to indicate popup menus based on the
  font metrics.

• FIXED: draw 3D triangle correctly.

• FIXED: Dynamic switching menu_bar->look.

• FIXED: Use 64-bit millisecond timer Now uses SDL_GetTick() for
  portability and consistently type as int64_t. This avoids 49 days
  overflow on Windows.

• SDL: Implemented @pce<-user.

• MODIFIED: Deleted console methods on @pce This deletes
  @pce->`console_label`, @pce->expose_console, @pce->iconify_consoleand@pce->show_console``.

• FIXED: Edit menu from profiler.

• ADDED: PceEmacs find_references (M-?) to reuse the window unless
  pinned.

• SDL: Implemented frame<-image.

• SDL: Implemented window->flash.

• MODIFIED: Deleted window->grab_keyboard Modern GUI frameworks no
  longer allow global grabbing of the keyboard.

• SDL: Added enable/disable screen saver.

• SDL: Implemented display->bell Was a stub. Now generates a
  sentesized beep.

• SDL: Re-implement reading images from arbitrary streams.

• SDL: Re-added support for images in saves objects.

• SDL: Support image->save Currently only for PNG format and only
  to a file.

• SDL: Implemented image<-pixel. This patch also introduces @rgba,
  a hash table that maps RGBA quadruples into existing colour objects.

• SDL: Implemented window->pointer (was a dummy) Does not work on
  all platforms. Still succeeds silently.

• FIXED: frame->background for open frames.

• FIXED: Help menu of Profiler to open correct page.

• CLEANUP: Remove RCS version control on classes.

• MODIFIED: Deleted display->open SDL has no such notion. It could
  be in headless mode, but then it still has a display that can be
  considered “open”.

• MODIFIED: Deleted unimplemented methods on class frame: These methods
  have been removed as there is no cross-platform support for them:

• MODIFIED: Deleted frame<-thread All windows are managed by the
  SDL main thread.

• DELETED: frame->grab_pointer. This is not supported by
  security-aware window systems.

• MODIFIED: Deleted frame<->border.

• MODIFIED: Deleted frame->hide. This commit removes several
  low-level bindings that are not supported by SDL and typically
  considered outdated.

• CLEANUP: Remove left overs of built-in XBM images

• CLEANUP: Remove X11 resource reference database.

• CLEANUP: Remove SDL_GRAPHICS macro (now unconditionally)

• CLEANUP: Remove WIN32_GRAPHICS configuration

• CLEANUP: Remove configuration for sigaction and union wait.

• FIXED: Use a safe way to send messages from a singal handler. This is
  used to inform xpce when the status of a child process has changed.

• CLEANUP: Remove CMake define USE_SIGINFO

• CLEANUP: Remove various X11 traces

• PORT: Support SDL < 3.2.12, which does not provide accumulative
  wheel events.

• FIXED: Windows: frame<-{open,save}_file work around for SDL3.4.0
  bug.

• PORT: Windows: get rid of remaining XBM images. Fixes
  XopenImage(@ms_left_arrow_image/image)
  message and shows correct left arrow for pullright menus.

[swi]

Pasting text broke in vi mode with the following in .swiplrc:

% vi mode in libedit
:- multifile editline:el_setup/1.
editline:el_setup(Stream) :-
   el_bind(Stream, \['-v'\]).
   el_bind(Stream, \["^I",  complete\]),
   % el_bind(Stream, \["^[?", show_completions]),
   el_bind(Stream, \["^R",  isearch_history\]).

thanks for the new release!

[jan]

@swi

Pasting text broke in vi mode with the following in .swiplrc:

Thanks. Pushed a fix. The way bracketed paste is handled is basically incompatible with vis moded operation (at least how libedit implements this), so I added some calls that allows us to make bracketed paste a mode that can be controlled and added el_get(Stream, editor(-Editor)). We now only enable this in emacs mode. Supporting it in vi mode requires implementation in the low-level libedit.