Not sure this is intended to work. This option exists such that libswipl.so is installed in the globally searched libraries and can be found using the general OS shared library search as opposed to the default where libswipl.so is installed in the Prolog tree and available to swipl using the so called rpath mechanism. I think this should run when called using
LD_LIBRARY_PATH=$HOME/swipl/lib ~/swipl/bin/swipl
But that doesn’t make much sense. This option was added for Linux distribution packagers who typically want it that way.
Assuming that I have the “normal” swi-prolog installation from apt install swi-prolog, libswipl.so seems to be found in /usr/lib/libswipl.so (not /usr/lib/swi-prolog/lib/x86_64/libswipl.so). Assuming that I want to load the so-file from a program, how do I find the correct path? I guess it is /usr/lib in 99% of the systems, but is there an algorithm that is 100% correct?
The guess is that swipl is in <prefix>/bin and libswipl.so in <prefix>/lib. It is no rule though, sometimes the OS places an architecture directory in between. I don’t remember whether dlopen() can do the normal search for you. If it can, it should fine the right library. And, if you need to link against libswipl you can simply use -lswipl without -L flags.
I don’t know about apt and the distros using it; I install swi via homebrew. I know that the offcial homebrew package for swi gets the pkg-config support wrong, and this was one of the reasons why I made my own package. (I discussed it here a couple of months ago.) But I would hope Debian & friends get pkg-config support right, so you should try using it.