Byte ordering/conversion predicates?

Help! Predicate
1

Is there any sort of builtin SWI-Prolog functionality for doing bit representation conversion (uint32 ↔ int32, i.e. taking the two’s complement) and or byte-order fixing, like libc’s ntohl and friends? More generally, is there anything for doing binary-encoding of numeric data for interop with other systems?

2

There are some serialization operations as part of library(protobufs) in contrib-protobufs/protobufs.c at 137a9d87ccc990d29e7ae337bb4a97ad8af05982 · SWI-Prolog/contrib-protobufs · GitHub (there’s also a byte-ordering fix - it does little-endian ordering, and not the network big-endian ordering … I assume it works with a big-endian machine but have never tested)

These are currently not exported by library(protobufs), but I can easily change that (just remove some "%"s)

3

For signed/unsigned integer conversion, at least int → uint is easily done with bitwise operators (UInt16 is Int16 /\ 0xffff). The other direction is a bit more annoying (UInt16 >= 0x8000 -> Int16 is UInt16 - 0x10000 ; Int16 = UInt16, or all in one expression: Int16 is UInt16 \/ (\0xffff * ((UInt16 >> 15) /\ 1))). A proper library or builtins for this would be more convenient of course.

Regarding converting integers from/to packed binary data, there was some related discussion a few months ago:

In that thread, @peter.ludemann also suggested the internal protobuf predicates mentioned above:

FWIW, I would also be interested in having binary un-/packing operations supported in SWI, and ideally not just as undocumented internal predicates that you have to call with a module prefix :slight_smile: I haven’t done much binary data parsing/generating with SWI yet, but it’s something I want to properly try out at some point.

5

I think Int16 is (Int + 0x8000) /\ 0xffff - 0x8000 will work to truncate any integer to signed 16-bit and wrap around if the input is out of range. Could be useful if you need to simulate signed 16-bit two’s complement arithmetic for some reason. This is shorter than my original solution too, not sure why I didn’t try this earlier.

1 Like
7

Tried looking for some “prior art” related to binary un-/packing in Prolog, but there really doesn’t seem to be much.

  • I found some StackOverflow questions, but the answers all suggest manual bit twiddling with either (is)/2 or library(clpfd).
  • Logtalk has a cbor library in pure Logtalk, and it also does all un-/packing by hand with lists and bit manipulation.
  • Had a quick look over the usual collection of other Prolog systems (SICStus, Ciao, YAP, ECLiPSe, Scryer) and couldn’t find any relevant builtins or libraries.
1 Like
8

@peter.ludemann, perhaps there is a use case here to factor our those routines from protobufs into a separate library.

9

Go for it. I’m afraid I won’t be able to do much for the next month or so.

But before you do it, be sure that there’s a genuine need. :wink:

10

Just as my better half says – i am very good at suggesting ideas to others :slight_smile:

11

AFAIK there is none as comprehensive as you seek.

However for bit field see;

To see them listed as code use:

listing('$syspreds':map_bits/N).

Note: This is one of the rare cases where some of the clauses have two cuts in the same clause.

I don’t know if Jan W. would accept a PR adding more genialized such predicates in the boot level code but it does give one a general idea of what is needed to create a PR as a separate module.

Quite a while ago I did post some code that worked with streams of byte data and did some of these types of conversions.

binary.pl (9.9 KB)

12

I’m not against a library with basic serialization helper predicates. In most cases I think one should write the low-level serialization operations for implementing some protocol in an external language. In some cases there are well maintained good external libraries on top of which you can build (in others that doesn’t make life much easier, so you just as well do it from scratch). If you write the low-level operations for a specific protocol in e.g., C, there is typically not much need for a separate Prolog library. Note that there exist quite a few ways to serialize basic data types such as integers, floats and strings. Most of these require just a few lines of C and, especially when it comes to handling strings, a low level implementation performs a lot better as we can directly map from a C char* to a Prolog atom or string rather than collecting the characters one by one in a list and use string_codes/2, etc.

1 Like
13

There’s a related discussion for Python: Mailman 3 C API for converting Python integers to/from bytes sequences - Python-ideas - python.org

2 Likes