Current status of SWI-Prolog with RocksDB packages installed using MSYS2

Thanks.

Correct any of this that is wrong or needs to be updated.

  1. SWI-Prolog for Windows can be installed using an install exe which can be downloaded from here - This is the option I take for Windows unless something does not work or I am trying something out.
  2. SWI-Prolog for Windows can be installed using MSYS2. I don’t know where the most up to date and accurate instructions are. I know you have been making changes to this over the last few months but am not sure if the work is complete so do not use.
  3. I know that RocksDB has been built using MSYS2 but have not tried to install it.
  4. To use RocksDB with SWI-Prolog to allow predicates to be used seamlessly from disk Jan W. created two code repositories on GitHub
  • SWI-Prolog interface for RocksDB (GitHub) - This is a SWI-Prolog pack that provides library(rocksdb), a binding to RocksDB.
  • Store predicates in RocksDB (GitHub) - This library builds on top of the rocksdb add-on. Right now it requires the GIT version of SWI-Prolog and recompiling the rocksdb pack from source.
    Each has to be installed as a package and compiled.
  1. As noted here I have used SWI-Prolog on Ubuntu on WSL on Windows with RocksDB and the two packages to load up 34.1 GB of facts. The facts stored on RocksDB can be accessed from Prolog code just by using them in a goal.

I have not tired all of this with MSYS2 because I was not sure it was all working.


My end goal is to be able to build SWI-Prolog, RocksDB, the two RocksDB packages all on Windows natively, (not using MSYS2) and see the performance numbers. However I am aware that because all of this is so new such numbers for now are meaningless until the two SWI-Prolog RocksDB packages are brought up to production standards.


If the OP desires that these RocksDB questions be moved from this topic to a new topic, just let us know.

1 Like

I see…

(msys2)

ROCKSDB_DISABLE_JEMALLOC=1 ROCKSDB_DISABLE_TCMALLOC=1 make -j6 -C rocksdb static_lib EXTRA_CXXFLAGS=-fPIC EXTRA_CFLAGS=-fPIC USE_RTTI=1 DEBUG_LEVEL=0
make[1]: Entering directory '/c/Users/c7201178/AppData/Local/swi-prolog/pack/rocksdb/rocksdb'
$DEBUG_LEVEL is 0
Unknown platform!
  CC       cache/cache.o
  CC       cache/cache_reservation_manager.o
  CC       cache/charged_cache.o
  CC       cache/clock_cache.o
  CC       cache/fast_lru_cache.o
  CC       cache/lru_cache.o
In file included from ./cache/fast_lru_cache.h:16,
                 from cache/fast_lru_cache.cc:10:
./cache/sharded_cache.h:125:11: error: 'port' does not name a type
  125 |   mutable port::Mutex capacity_mutex_;
      |           ^~~~

it give some reasons:

  • ‘port’ is not defined (as member function or procedure:
    • function: void foo(void) { },
    • procedure: int foo(int a=4, int b=2) { return a + b; }
    • the class or struct ure is not known
  • it is redefined in a header file → see: #undef (eventuelly ioports.header)
  • it lay in a namespace

I am getting there. I have recompiled rocksdb with RTTI and PIC on msys (RTTI was already set to on, PIC needed an extra run), so the main challenge is to remove it from the prolog pack, see here for more information

1 Like

FWIW, this is how I do the build and run the tests on Linux (which sets -fPIC USE_RTTI=1, etc):

# touch $SRC/rocksdb/cpp/rocksdb4pl.cpp
make -C $SRC/rocksdb SUBMODULE_UPDATE= SWIPL=$BUILD_DIR/src/swipl PACKSODIR=/tmp/rocksdb-so SOEXT=so CFLAGS="-fPIC -pthread -I$SRC/swipl-devel/src -I$SRC/swipl-devel/src/os -I$SRC/swipl-devel/packages/cpp" plugin check

Which commit are you sync-ed to? Are you using the latest commit (Changes for SWI-cpp2.h PlPredicate, PlAtom, etc. · JanWielemaker/rocksdb@92ea372 · GitHub), which uses SWI-cpp2.h?

Also, do you have the version of the submodule for rocksdb (I’ve only tested with V7.7.3, and I’ve found problems in the past when sync-ing to rocksdb head, which is why I haven’t updated the submodule in a while).

I was planning on some modifications to rocksdb, but got sidetracked with fixing some problems in SWI-cpp2.h (which I’m still working on – it’s turned out to be trickier than I had anticipated).

1 Like

Hello,
maybe not the exact Topic On, but I have started a new PCE Project on my GitHub.
It does nothing super dubba - only display 2 message boxes.
But it is compiled under Windows 11 with MingW64 g++.
All sources so far are included.
You have to make some changes into the Makefile.
But you should skilled enough to see what is to do (path, and toolchain).

Here is the Link:
AlphaTest for Zwap

This topic seems to have died :thinking:

In doing some leg work to get RocksDB working with SWI-Prolog via MSYS2 using Jan’s two repos

did this as an exercise to learn more

As many of you know but I just discovered, SWI-Prolog interface for RocksDB is built as a static library and not a DLL or shared object. (ref) The norm seems to be that a DLL or shared object is preferred and a static library is the expectation. (ref)

As Jan W. noted here

  • The static lib is not compiled with -fPIC and can thus not be used to create a shared object.

but then @mgondan1 notes here

I have recompiled rocksdb with RTTI and PIC on msys (RTTI was already set to on, PIC needed an extra run), so the main challenge is to remove it from the prolog pack

Any further updates? Care to share the MSYS2 build steps? Why the need to remove it from the prolog pack?

@peter.ludemann does note the make for Linux which which sets -fPIC USE_RTTI=

make -C $SRC/rocksdb SUBMODULE_UPDATE= SWIPL=$BUILD_DIR/src/swipl PACKSODIR=/tmp/rocksdb-so SOEXT=so CFLAGS=“-fPIC -pthread -I$SRC/swipl-devel/src -I$SRC/swipl-devel/src/os -I$SRC/swipl-devel/packages/cpp” plugin check

which is helpful.

@paule32 has provided a GItHub repo that I need to look at in more detail.

Thanks to all involved.


Side note

For me this is just a stepping stone to

  1. Build SWI-Prolog using Microsoft Visual Studio (ref)
  2. Add the RocskDB functionality for the Windows build using Microsoft Visual Studio

have make a Update: Editor Link

1 Like

Thanks for the files. However your C++ and builds skills are beyond mine so most of what is there I can not use or understand how it can help me. I do appreciate the effort. :slightly_smiling_face:

1 Like

in the release binary zip, you can find Windows pre-compiled test files.
there is a Editor - which I would use for my Prolog implementation.
In the same directory, you can find the “prolog.exe”, it is a sub project of
my Prolog-Implementation (a start up).
in the Editor.exe, you can write Text, and in a near future, parsed by the
Prolog.exe.
The rest show, how you can use DLL files in SWIPL.
There must be a test prolog file, which can be called, to see the example
implementation of DLL files.
All comes with sources.
If you have any Question’s, let me know.

update: now, with setup.exe => Setup IDE 32-Bit

If I recall correctly, I got this “make” command by looking at what pack_install/2 does (I wanted to build RocksDB outside of a pack_install).

1 Like

successfully compiled for Windows 64-Bit

OK

Here are my questions.

I took a look and see lots of exe files. Are these for use with MSYS2 or native Windows?

If this is for MSYS2, I take it you aware of the RocksDB package that is kept up to date.
Package: mingw-w64-x86_64-rocksdb - MSYS2 Packages
That is what I use to get a working version of RocksDB on MSYS2.

More importantly, if these were built for Windows (not MSYS2) and did no cross compiling then what are the build steps used? At this point don’t want a makefile or cmake but the build steps that can be done manually to understand how it is done.

MSYS2 is optimized for Windows 32- and 64-Bit Version’s.
So, MSYS2 will run native in no SandBox or Virtual ByteCode.
It comes with a rich amount of DLL Files:

  • beginning by the (Win)DOS libc
  • crosses at Win32API
  • and end with very much Framework Project DLL
    (can be, that these Frameworks have to be installed seperatly
    e.g.: Qt 5 - The Free Alternative for BORLAND Delphi VCL)

Other than Cygwin - which you can also use.
The only thing is, that the plain Console Application follows a internal structure of Directory-Layout.
So, MSys2 helps you, to port Linux ISO-C/C++ Projects to Windows - by using the Win32API.
Cygwin is optimized for X-Server Application’s, but has its own Directory-Layout.
So, if you run native MSYS2 Applications on Console, the Path-Layout is:

/HardDriveLetter/path/to/… e.g.: # cd /E/Projects/RocksDB

the last one change the Directory to HardDrive E:\Projects\RocksDB.

In Cygwin, you have to add a Prefix: # cd /cygdrive/E/…
Both Systems have cons, and pros.


To Compile RockDB on Windows 10 MinGW-64, you need the sources: Source Link

  • download
  • depack it on huge (disk space Drive)
  • make sure, you have install g++, gcc, …
  • go to the MSYS2 folder, and click the Mingw64.exe Application Icon in the Explorer
    (a Shell Window should be open)
  • change the Directory: e.g.: # cd /E/Projects/RocksDB
    (make sure, you are in the Root-Folder
  • create a new Directory called “build” with: # mkdir build
  • navigate to build through: # cd build
  • make sure, you have install CMAKE (maybe you need ninja, too)
  • to start the Compile Process, type in: # cmake …
    (important: two Points !)
    This folder will then use for Compiler Output, so if you fail, you don’t need to re-install (zip file)
    because cmake create a lot of temporary stuff
  • after some seconds, you can see Output Text, take care there is no Warning or Error at last
    step/last line.
  • if no error, dont take care about “not found” stuff - the most Compiler things will work.
  • now, type in ./build the following Command: # ninja
  • press Enter/Return Key and don’t forget to make the Cafe working, from now on it will take
    a while … if you have luck, there should be compiled about 707 no. of Files
    … depend on your Computer Speed, and RAM
    … I recommend 16 GB of RAM, 200 GB Hard Disk Space, 8 Core CPU (< 10 Minutes)
  • if ninja Job ends, and no Warning or Error is displayed on the Screen, then you have
    fresh new Installation Files (.dll, .a, and .exe Files for/and Test purposes)

If you plan to contribute/publish this Files, you need two .dll that comes from MinGW-64
or MSYS2. What the Name of this Files are, can you determine, if you remove the PATH
Environment Variable (I assume you are in a DOS-Shell like Command.com or cmd.com)
by type in:

1. E:\echo %PATH%    rem: to store/output the old state of PATH for restore
2. E:\set PATH=      rem: this clear the PATH Variable

Then navigate to your RocksDB Directory (E:\Projects\RocksDB), and try to start a .EXE File
In common way, you will be noticed by the Windows System, which .DLL is need to start the
Application.
But since this are Global .DLL Files, you can use it with other Software of your Desire.

Note:
Make sure, you use the same .DLL Files like you have compile the RocksDB File’s.

  • So, you can not use 64-Bit .DLL in 32-Bit Application’s.
  • But you can use 64-Bit Application’s with 32-Bit DLL Files.

An other Point of View is the Fact, that the Local (char set) is important.
The new Windows Versions at XP upwards, use UTF-8, UTF-16.
So, you can stuckle in Trouble about the Encoding’s of File Layout.

UTF-8 is a Mixture of ANSI + UTF-8 (ANSI: 0x20 … 0x1F) (UTF-8: 0x1f … 0xff)
UTF-16 is widely used. There, one Character consists of 2 Byte’s !

The Space Char. in UTF-16 is: 0x20 0x00
The the new line in UTF-16 is: 0x0A 0x00

Please let me know, when I have forget something.

Thank’s for reading
paule32

1 Like

here a WebSite, you can get inspired, for install the GCC ToolChain’s:
GNU GCC C/C++ Compiler ToolChain’s

Today, I start Documentation my new Zwapel Project, which should be a SWIPL conform, and compatible Port - but with Remote Assemblies, to make the Execution of Code much faster.
I used the official Logo of SWIPL, and I hope to “do not” strand up with troubles of your Original Software Project.

The Link to Online Resources …

Sorry for the delayed response, very simple:

MSYS2 shell for MinGW-W64-x86_64:

$ pacman -S mingw-w64-x86_64-rocksdb mingw-w64-x86_64-swi-prolog
$ swipl
?- pack_install(rocksdb).

Then, respond “no” when it asks to run the post-installation scripts.

?- halt.
$ cd /c/Users/matth/AppData/Local/swi-prolog/pack/rocksdb

[or whereever the rocksdb sources are located]

$ g++ -shared -D__SWI_PROLOG__ -I/mingw64/lib/swipl/include cpp/rocksdb4pl.cpp /mingw64/lib/librocksdb.a -L/mingw64/lib/swipl/lib/x64-win64 -lswipl -lsnappy -lzstd -lz -llz4 -lbz2 -lshlwapi -lrpcrt4 -o lib/x64-win64/rocksdb4pl.dll
$ swipl
?- use_module(library(rocksdb)).
true.
?- rocks_open(".", R, []), rocks_put(R, aap, noot), rocks_get(R, aap, Noot), rocks_close(R).
R = <rocksdb>(000001d01ada5c50),
Noot = noot.

Thanks,

will give them a run and let you know what happens.

Please keep me in the loop with any changes. I hope to get back to RocksDB; but my work on SWI-cpp2.h (which RocksDB uses) has turned out to use much more time than I had anticipated.