# Raw to byte or character conversion **URL:** \ **Category:** Help!\ **Created:** [December 16, 2021, 2:56pm UTC](https://swi-prolog.discourse.group/t/raw-to-byte-or-character-conversion/4777 "2021-12-16T14:56:32Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![BenEngbers](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/benengbers/32/95_2.png) [@BenEngbers](https://swi-prolog.discourse.group/u/BenEngbers)\ **Post date:** [December 16, 2021, 2:56pm UTC](https://swi-prolog.discourse.group/t/raw-to-byte-or-character-conversion/4777/1 "2021-12-16T14:56:32Z")
Hi, I just finished my work on a R-client for the BaseX XML-database (see [CRAN - Package RBaseX](https://cran.r-project.org/web/packages/RBaseX/index.html)) and am now planning to start working on a similar client in SWI-prolog. The (all) clients are based on raw non-blocking sockets. All input from the client is converted to a raw stream, the server first responds by returning a zero-terminated raw-stream and then adds either a 0x00 (=succes) or 0x01 (=error). When the input stream is binary, 0x00-bytes and 0xFF- bytes are prepended by an extra 0xFF-byte. They have to be removed from the aoutput stream as returned by the server. When the output-stream represents a character-stream, the raw data has to be converted to character data and care has to be taken of control characters which may be included. I have two questions: - Is there any interest in a Prolog-client? - As you can see, transforming and processing the stream is the hardest part of the work. Are there any prolog-modules with handy functions/predicates for transforming raw-streams (add or remove bytes, split stream into other streams etc.) Ben ---
**Author:** ![peter.ludemann](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/peter.ludemann/32/48_2.png) [@peter.ludemann](https://swi-prolog.discourse.group/u/peter.ludemann)\ **Post date:** [December 16, 2021, 8:42pm UTC](https://swi-prolog.discourse.group/t/raw-to-byte-or-character-conversion/4777/2 "2021-12-16T20:42:52Z")
You might find these functions useful: [contrib-protobufs/protobufs.c at 7fa1c0225e4a58f305f1380aff7c95eef26de7a8 · SWI-Prolog/contrib-protobufs · GitHub](https://github.com/SWI-Prolog/contrib-protobufs/blob/7fa1c0225e4a58f305f1380aff7c95eef26de7a8/protobufs.c) These are low-level conversions between some SWI-Prolog types and bytes, as needed by [library(protobufs)](https://www.swi-prolog.org/pldoc/doc_for?object=section(%27packages/protobufs.html%27)). They are currently not exported from the `protobufs` module, but that can be easily changed – you can see how they’re used in protobufs.pl. If you need these extended a bit, I can probably help you. ---
**Author:** ![jan](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/jan/32/4_2.png) [@jan](https://swi-prolog.discourse.group/u/jan)\ **Post date:** [December 17, 2021, 8:00am UTC](https://swi-prolog.discourse.group/t/raw-to-byte-or-character-conversion/4777/3 "2021-12-17T08:00:02Z")
Depending on experience, performance impact, etc., I’d consider doing most of the low-level work in C(++). That said, Prolog has all that is needed to read and write bytes. A few things have no portable solution in pure Prolog, such as converting some bytes into a float or converting bytes in unknown native byte ordering into an integer. The stuff in protobufs come handy, either for sharing or getting inspiration. Otherwise, the more interfaces the better. In the Prolog world SWI-Prolog is doing quite well considering available interfaces. It is still one of the weak points of Prolog though ☹ ---
**Author:** ![BenEngbers](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/benengbers/32/95_2.png) [@BenEngbers](https://swi-prolog.discourse.group/u/BenEngbers)\ **Post date:** [December 17, 2021, 1:37pm UTC](https://swi-prolog.discourse.group/t/raw-to-byte-or-character-conversion/4777/4 "2021-12-17T13:37:18Z")
I can read C and C++ but luckily I have always been able to avoid real programmming in one of those languages 😉. I’ll try but maybe that I will need your help. Adoption of a new interface is also promoted when a good implementation example is available. Do you have an idea for a good use case? ---
**Author:** ![EricGT](https://avatars.discourse-cdn.com/v4/letter/e/f1d935/32.png) [@EricGT](https://swi-prolog.discourse.group/u/EricGT)\ **Post date:** [December 17, 2021, 1:42pm UTC](https://swi-prolog.discourse.group/t/raw-to-byte-or-character-conversion/4777/5 "2021-12-17T13:42:30Z")
> [@BenEngbers](#): > > I can read C and C++ but luckily I have always been able to avoid real programmming in one of those languages 😉. I am in the same boat. My advise would be to get the code working correctly with Prolog, then when it works as you seek and if speed or some other checking is needed, then implement portions in C. Also if you are at the C level then consider SIMD. ([ref](https://blog.tinybird.co/2021/12/14/simd/)) ---
**Author:** ![BenEngbers](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/benengbers/32/95_2.png) [@BenEngbers](https://swi-prolog.discourse.group/u/BenEngbers)\ **Post date:** [December 17, 2021, 10:58pm UTC](https://swi-prolog.discourse.group/t/raw-to-byte-or-character-conversion/4777/6 "2021-12-17T22:58:42Z")
Hi Jan, [A C++ interface by a C++ programmer](http://www.volker-wysk.de/swiprolog-c++/index.html) by Volker Wysk [post@volker-wysk.de](mailto:post@volker-wysk.de) links to a library which dates back to 2004. Is this really the most recent information for interfacing SWI-Prolog and C++? ---
**Author:** ![peter.ludemann](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/peter.ludemann/32/48_2.png) [@peter.ludemann](https://swi-prolog.discourse.group/u/peter.ludemann)\ **Post date:** [December 17, 2021, 11:56pm UTC](https://swi-prolog.discourse.group/t/raw-to-byte-or-character-conversion/4777/7 "2021-12-17T23:56:55Z")
> [@BenEngbers](#): > > [A C++ interface by a C++ programmer](http://www.volker-wysk.de/swiprolog-c++/index.html) by Volker Wysk [post@volker-wysk.de](mailto:post@volker-wysk.de) links to a library which dates back to 2004 GCC 2.95 brings back memories … not all of them good. 😉 This seems to be newer: [https://www.swi-prolog.org/pldoc/doc\_for?object=section('packages/pl2cpp.html')](https://www.swi-prolog.org/pldoc/doc_for?object=section(%27packages/pl2cpp.html%27)) Also, for something as simple as dealing with raw bytes / characters, you might be better off (and slightly more performant) to use the C interface. The protobuf.c file that I mentioned could be a reasonable place to start. Maybe it would be best if you were to say what “raw byte/character conversion” functionality you want? ---
**Author:** ![jan](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/jan/32/4_2.png) [@jan](https://swi-prolog.discourse.group/u/jan)\ **Post date:** [December 18, 2021, 8:53am UTC](https://swi-prolog.discourse.group/t/raw-to-byte-or-character-conversion/4777/8 "2021-12-18T08:53:03Z")
Actually, the packaged SWI-cpp.h is the original interface by me. I’m a lousy C++ programmer though. Volker wrote something that is closer to how C++ programmers might want to see Prolog. I have little opinion on that 🙂 But yes, the packaged one is maintained and since recently works cleanly using `g++-11` on very high warning levels. It doesn’t do much more than a bit of basic type conversion and exception forwarding though. It also doesn’t cover the entire SWI-Prolog C API, but you can freely mix the two in you C++ code. ---
**Author:** ![peter.ludemann](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/peter.ludemann/32/48_2.png) [@peter.ludemann](https://swi-prolog.discourse.group/u/peter.ludemann)\ **Post date:** [December 18, 2021, 7:39pm UTC](https://swi-prolog.discourse.group/t/raw-to-byte-or-character-conversion/4777/9 "2021-12-18T19:39:55Z")
It’s been 10+ years since I’ve programmed in C++, so I took a quick look at [Google C++ Style Guide](https://google.github.io/styleguide/cppguide.html) to remind myself of the many problem areas in that language. There have been a lot of changes in C++ over the years; for example with copyable and movable types, smart pointers, etc. So, it would be good for someone with recent C++ experience to review the C++ interface(s). One thing I noticed in the “supported” C++ interface - it should probably should be updated to disallow copy constructors and assignment operators unless they make sense ([c++ - Why copy constructor and assignment operator are disallowed? - Stack Overflow](https://stackoverflow.com/questions/3422125/why-copy-constructor-and-assignment-operator-are-disallowed)). There’s more on this [here](https://google.github.io/styleguide/cppguide.html#Copyable_Movable_Types). ---
**Author:** ![peter.ludemann](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/peter.ludemann/32/48_2.png) [@peter.ludemann](https://swi-prolog.discourse.group/u/peter.ludemann)\ **Post date:** [July 7, 2022, 11:37pm UTC](https://swi-prolog.discourse.group/t/raw-to-byte-or-character-conversion/4777/10 "2022-07-07T23:37:39Z")
I’m starting to review `SWI-cpp.h`, by applying the Google style guide. So far, it’s found a number of subtle bugs, mainly to do with implicit casts that weren’t doing what was expected (e.g., treating a `size_t` as a `term_t`). I’ve also found some bugs in `rocksdb4.pl.cpp` that are related, and it wouldn’t surprise me if other code that depends on `SWI-cpp.h` has similar bugs. I’m also moving some of the `public` fields to `private` or `protected` – typically they can be accessed by static casts. This might break some code, but hopefully not much, and it has caught a few other subtle bugs. BTW, I think we should move to C++17 … @jan – where would I make the change? ---
**Author:** ![peter.ludemann](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/peter.ludemann/32/48_2.png) [@peter.ludemann](https://swi-prolog.discourse.group/u/peter.ludemann)\ **Post date:** [July 8, 2022, 6:17am UTC](https://swi-prolog.discourse.group/t/raw-to-byte-or-character-conversion/4777/11 "2022-07-08T06:17:45Z")
I’m thinking that the use of `operator=` for unification of terms is a bad idea. Part of the problem is that there’s a default `operator=` and this can lead to subtle bugs (I’ve already found a few places where implicit conversions caused a bug)… Instead, I propose changing the `=` operator from, e.g. `int PlTerm::operator =(const PlTerm &t2);` to `bool PlTerm::unify(const PlTerm &t2);` As far as I can tell, the only code in `github.com/SWI-Prolog` that uses `SWI-cpp.h` is [GitHub - SWI-Prolog/packages-swipl-win: Portable Qt-based console for SWI-Prolog by Carlo Capelli](https://github.com/SWI-Prolog/packages-swipl-win) … I can compile it with these changes, but I don’t know how to test it (and I don’t have a development environment for Windows). (pack(rocksdb) uses SWI-cpp.h, but I can modify that if needed) ---
**Author:** ![CapelliC](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/capellic/32/17_2.png) [@CapelliC](https://swi-prolog.discourse.group/u/CapelliC)\ **Post date:** [July 8, 2022, 6:27am UTC](https://swi-prolog.discourse.group/t/raw-to-byte-or-character-conversion/4777/12 "2022-07-08T06:27:54Z")
Hi Peter Sorry I cannot test right now, cause I’m busy with some other work, but in the weekend I could try your changes. ---
**Author:** ![jan](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/jan/32/4_2.png) [@jan](https://swi-prolog.discourse.group/u/jan)\ **Post date:** [July 8, 2022, 8:17am UTC](https://swi-prolog.discourse.group/t/raw-to-byte-or-character-conversion/4777/13 "2022-07-08T08:17:30Z")
At a conceptual level the only sensible implementation of = between two terms is unification. We notably do not want to copy the term reference itself on =. I’m all in favor of getting rid of ambiguities. We should also be careful that this may be in use in quite a few places and thus break a lot of code, as well as to make sure that the resulting semantics for = is not doing any harm. > [@peter.ludemann](#): > > I can compile it with these changes, but I don’t know how to test it (and I don’t have a development environment for Windows). On a Linux system it simply provides `packages/swipl-win/swipl-win`. Same happens on the Mac, but if I recall well the result doesn’t run out of the box. Note that the Qt version is not used on Windows. That uses a rather simple console written as my first exercise in programming on Windows ('95 🙂 ) ---
**Author:** ![jan](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/jan/32/4_2.png) [@jan](https://swi-prolog.discourse.group/u/jan)\ **Post date:** [July 8, 2022, 8:22am UTC](https://swi-prolog.discourse.group/t/raw-to-byte-or-character-conversion/4777/14 "2022-07-08T08:22:29Z")
> [@peter.ludemann](#): > > BTW, I think we should move to C++17 … @jan – where would I make the change? Only in the documentation, I guess. The overall policy of SWI-Prolog is to keep its interfaces running on all supported OSes, libraries and tool chains from the oldest to the latest. The latter is a hard requirement. Old tools may be abandoned if there is a pressing reason. If possible though we try to keep backward compatibility, possibly with loss of some new features. ---
**Author:** ![peter.ludemann](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/peter.ludemann/32/48_2.png) [@peter.ludemann](https://swi-prolog.discourse.group/u/peter.ludemann)\ **Post date:** [July 8, 2022, 6:30pm UTC](https://swi-prolog.discourse.group/t/raw-to-byte-or-character-conversion/4777/15 "2022-07-08T18:30:43Z")
In the case of rocksdb, I had to explicitly specify `CPPFLAGS=-std=c++17`, to get it to compile with the current RocksDB code. I noticed that cmake has `set(CMAKE_CXX_STANDARD 17)` but I don’t know how to check that it takes effect, nor how to propagate that to packages such as pack(rocksdb). (Presumably this directive should be added to `packages/cpp/CMakeLists.txt`?) ---
**Author:** ![peter.ludemann](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/peter.ludemann/32/48_2.png) [@peter.ludemann](https://swi-prolog.discourse.group/u/peter.ludemann)\ **Post date:** [July 8, 2022, 7:48pm UTC](https://swi-prolog.discourse.group/t/raw-to-byte-or-character-conversion/4777/16 "2022-07-08T19:48:17Z")
> [@BenEngbers](#): > > [A C++ interface by a C++ programmer](http://www.volker-wysk.de/swiprolog-c++/index.html) by Volker Wysk [post@volker-wysk.de](mailto:post@volker-wysk.de) links to a library which dates back to 2004. From what I’ve read of Volker Wysk’s documentation, his C++ interface doesn’t seem suitable for writing foreign code extensions to SWI-Prolog – it seems to be mainly targeted towards calling Prolog from C++. @jan 's SWI-cpp.h is a fairly thin wrapper on the existing SWI-Prolog.h interface. It feels pretty natural to me as an ex-C++ programmer, especially the use of exceptions (replacing `PL_raise_exception()` and checking return codes in the C code). In a few places i think it abuses some esoteric features of C++ (overloading the assignment operator) and leaves open the possibility of implicit type conversions that don’t do what the programmer wants. (I’m fairly conservative in that respect, based on bitter experience with other people’s code) I think I’ll start a new thread, discussing SWI-cpp.h and the changes I’d like to make in it. ---
**Author:** ![jan](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/jan/32/4_2.png) [@jan](https://swi-prolog.discourse.group/u/jan)\ **Post date:** [July 9, 2022, 1:47pm UTC](https://swi-prolog.discourse.group/t/raw-to-byte-or-character-conversion/4777/17 "2022-07-09T13:47:29Z")
> [@peter.ludemann](#): > > It feels pretty natural You are too kind for me 🙂 I’m a lousy C++ programmer (and intend to keep it that way). B.t.w. I know a real C++ programmer has been working on the C++ interface. You may want to consider his work. Contact me offline if you are interested and I’ll check with him whether he is willing (and able) to share this work. ---
**Author:** ![peter.ludemann](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/peter.ludemann/32/48_2.png) [@peter.ludemann](https://swi-prolog.discourse.group/u/peter.ludemann)\ **Post date:** [July 9, 2022, 3:51pm UTC](https://swi-prolog.discourse.group/t/raw-to-byte-or-character-conversion/4777/18 "2022-07-09T15:51:34Z")
> [@jan](#): > > You are too kind for me 🙂 I’m a lousy C++ programmer (and intend to keep it that way). My biggest criticism of SWI-cpp.h is that it uses some features of C++ that experienced C++ programmers avoid because they’re error-prone (that’s one of the reasons Google has a C++ style guide – it’s to prevent too much cleverness … especially as the C++ template language is Turing complete). And there are a few C-isms in the code (such as using `int` instead of `bool`) and subtleties about how constructor member initialization work. ---
**Author:** ![peter.ludemann](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/peter.ludemann/32/48_2.png) [@peter.ludemann](https://swi-prolog.discourse.group/u/peter.ludemann)\ **Post date:** [July 11, 2022, 7:45pm UTC](https://swi-prolog.discourse.group/t/raw-to-byte-or-character-conversion/4777/19 "2022-07-11T19:45:09Z")
> [@CapelliC](#): > > but in the weekend I could try your changes. If you’re referring to [GitHub - SWI-Prolog/packages-swipl-win: Portable Qt-based console for SWI-Prolog by Carlo Capelli](https://github.com/SWI-Prolog/packages-swipl-win) , I’m already making sure that my changes to SWI-cpp.h work with it (with a few small changes to swipl-win, of course). Do you have other code that uses SWI-cpp.h? ---
**Author:** ![BenEngbers](https://yyz2.discourse-cdn.com/free1/user_avatar/swi-prolog.discourse.group/benengbers/32/95_2.png) [@BenEngbers](https://swi-prolog.discourse.group/u/BenEngbers)\ **Post date:** [July 11, 2022, 8:28pm UTC](https://swi-prolog.discourse.group/t/raw-to-byte-or-character-conversion/4777/20 "2022-07-11T20:28:50Z")
Hi Peter and Jan, My progress in my Prolog programming work is slow. This is partly because there are many things - particularly in the area of C++ programming - that I need to understand before I can really come up with results. However, I am still following this thread with interest. Ben [Next page](https://swi-prolog.discourse.group/t/raw-to-byte-or-character-conversion/4777.md?page=2)