Problem loading python module

Help!
,
1

I’m using: SWI-Prolog version 9.1.22-26-g4eb8604da

I want the code to: load the python module numpy

But what I’m getting is:

(liftcover) [friguzzi@login02 ~]$ $HOME/bin/swipl
Welcome to SWI-Prolog (threaded, 64 bits, version 9.1.22-26-g4eb8604da)
SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software.
Please run ?- license. for legal details.

For online help and background, visit https://www.swi-prolog.org
For built-in help, use ?- help(Topic). or ?- apropos(Word).

?- py_version.
% Interactive session; added `.` to Python `sys.path`
Deprecated: instead of py_call(Obj:Attr = Value), use py_setattr(Obj,Attr,Value)
% Janus 1.1.3 embeds Python 3.10.8 (main, May 10 2023, 02:53:14) [GCC 11.3.0]
% Janus: using venv from '/leonardo_work/IscrC_NSGP/liftcover'
true.

?- py_module_exists(numpy).
false.

?-
% halt
(liftcover) [friguzzi@login02 ~]$ /leonardo_work/IscrC_NSGP/liftcover/bin/python
Python 3.10.8 (main, May 10 2023, 02:53:14) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>>
2

So, that is still there. Pushed a fix for that. Should be innocent.

But, this does not reproduce. Created a Python virtualenv, activated it, ran python -m pip install numpy, started Prolog in the venv and got

81 ?- py_version.
% Interactive session; added `.` to Python `sys.path`
% Janus 1.1.3 embeds Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
% Janus: using venv from '/home/janw/venv'
true.

82 ?- py_module_exists(numpy).
true.

I’m afraid you need to do a little debugging. py_module_exists/1 is written in Prolog, calling some Python stuff.

1 Like
3

I will try to do some debugging. But I also tried

(liftcover) [friguzzi@login02 ~]$ $HOME/bin/swipl
Welcome to SWI-Prolog (threaded, 64 bits, version 9.1.22-26-g4eb8604da)
SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software.
Please run ?- license. for legal details.

For online help and background, visit https://www.swi-prolog.org
For built-in help, use ?- help(Topic). or ?- apropos(Word).

?- py_version.
% Interactive session; added `.` to Python `sys.path`
Deprecated: instead of py_call(Obj:Attr = Value), use py_setattr(Obj,Attr,Value)
% Janus 1.1.3 embeds Python 3.10.8 (main, May 10 2023, 02:53:14) [GCC 11.3.0]
% Janus: using venv from '/leonardo_work/IscrC_NSGP/liftcover'
true.

?- py_shell.
Python 3.10.8 (main, May 10 2023, 02:53:14) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import numpy
Traceback (most recent call last):
  File "<console>", line 1, in <module>
ModuleNotFoundError: No module named 'numpy'
>>>
4

So, it is not py_module_exists/1 that is to blame. Probably something with setting up the Python virtualenv in Janus. See janus.pl. That is what I could find on the internet on how to setup a venv by hand. Works for me (as I showed). Possibly you can find out why it does not for you. Guess you need to compare some possible relevant settings in Python when run on the commandline and using the embedded version.

5

The problem was that numpy was not installed in the venv but in site-packages:
it was in /leonardo/home/userexternal/friguzzi/.local/lib/python3.10/site-packages

From
https://www.swi-prolog.org/pldoc/man?section=janus-venv
I see that

Remove all directories with base name site-packages or dist-packages from sys.path

So exiting from the venv, uninstalling numpy, reentering in the venv, installing numpy solved the problem.

6

Good. That is resolved. Now the question is whether removing the site-packages is correct? I recall I observed that behavior from Python itself as well, but possibly I was wrong? The idea is that the embedded Python should behave as the normal Python executable inside the virtual environment.

Well, the embedded python is linked against some specific version of libpython, so it may not execute the same version of Python if the venv points to some specific Python. That cannot be avoided AFAIK.

7

I have no idea, I’m no big expert in python

Yes, I compiled swipl with the venv activated.

I have another question regarding command line options and python:

(liftcover) [friguzzi@login02 fbk237]$ cat testgpu.pl
main :-
  py_version,
  py_module_exists(numpy),
  py_module_exists(cupy).
(liftcover) [friguzzi@login02 fbk237]$ $HOME/bin/swipl -s testgpu.pl -g "main"
ERROR: -g main: false
(liftcover) [friguzzi@login02 fbk237]$ $HOME/bin/swipl -s testgpu.pl -t "main"
Welcome to SWI-Prolog (threaded, 64 bits, version 9.1.22-26-g4eb8604da)
SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software.
Please run ?- license. for legal details.

For online help and background, visit https://www.swi-prolog.org
For built-in help, use ?- help(Topic). or ?- apropos(Word).

% Interactive session; added `.` to Python `sys.path`
Deprecated: instead of py_call(Obj:Attr = Value), use py_setattr(Obj,Attr,Value)
% Janus 1.1.3 embeds Python 3.10.8 (main, May 10 2023, 02:53:14) [GCC 11.3.0]
% Janus: using venv from '/leonardo_work/IscrC_NSGP/liftcover'
% halt

Why the different behavior of -g and -t?

8

Possibly this?

Janus, like Python, only adds ‘.’ to the module search path for an interactive session. Else, use py_add_lib_dir/1,2

These days, I normally use initialization/2.

:- initialization(main, main).
9

but the modules numpy and cupy are not in .

10

Then I wouldn’t know. Put a call to gtrace/0 at the beginning of your main/0?

11

Further information:

(liftcover) [friguzzi@login02 fbk237]$ cat testgpu.pl
main :-
  trace,
  py_module_exists(numpy),
  py_module_exists(cupy).

:- initialization(main).

(liftcover) [friguzzi@login02 fbk237]$ $HOME/bin/swipl testgpu.pl
   Call: (25) janus:py_module_exists(numpy) ? creep
   Call: (26) error:must_be(atom, numpy) ? skip
   Exit: (26) error:must_be(atom, numpy) ? creep
   Call: (26) janus:py_call(sys:modules:'__contains__'(numpy), @(true)) ? creep
   Call: (28) janus:py_initialize ? creep
   Call: (29) getenv('VIRTUAL_ENV', _4266) ? skip
   Exit: (29) getenv('VIRTUAL_ENV', '/leonardo_work/IscrC_NSGP/liftcover') ? creep
   Call: (29) prolog_to_os_filename(_5888, '/leonardo_work/IscrC_NSGP/liftcover') ? skip
   Exit: (29) prolog_to_os_filename('/leonardo_work/IscrC_NSGP/liftcover', '/leonardo_work/IscrC_NSGP/liftcover') ? creep
   Call: (29) atom_concat('/leonardo_work/IscrC_NSGP/liftcover', '/pyvenv.cfg', _7510) ? skip
   Exit: (29) atom_concat('/leonardo_work/IscrC_NSGP/liftcover', '/pyvenv.cfg', '/leonardo_work/IscrC_NSGP/liftcover/pyvenv.cfg') ? creep
   Call: (29) access_file('/leonardo_work/IscrC_NSGP/liftcover/pyvenv.cfg', read) ? skip
   Exit: (29) access_file('/leonardo_work/IscrC_NSGP/liftcover/pyvenv.cfg', read) ? creep
   Call: (29) current_prolog_flag(executable, _10760) ? skip
   Exit: (29) current_prolog_flag(executable, '/leonardo/home/userexternal/friguzzi/lib/swipl/bin/x86_64-linux/swipl') ? creep
   Call: (29) current_prolog_flag(py_argv, _12382) ? skip
   Exit: (29) current_prolog_flag(py_argv, []) ? creep
   Call: (29) janus:py_initialize('/leonardo/home/userexternal/friguzzi/lib/swipl/bin/x86_64-linux/swipl', ['-I'], []) ? creep
   Call: (30) janus:py_initialize_('/leonardo/home/userexternal/friguzzi/lib/swipl/bin/x86_64-linux/swipl', ['-I'], []) ? creep
   Exit: (30) janus:py_initialize_('/leonardo/home/userexternal/friguzzi/lib/swipl/bin/x86_64-linux/swipl', ['-I'], []) ? creep
   Call: (30) absolute_file_name(library('python/janus.py'), _16492, [access(read)]) ? skip
   Exit: (30) absolute_file_name(library('python/janus.py'), '/leonardo/home/userexternal/friguzzi/lib/swipl/library/ext/swipy/python/janus.py', [access(read)]) ? creep
   Call: (30) file_directory_name('/leonardo/home/userexternal/friguzzi/lib/swipl/library/ext/swipy/python/janus.py', _18672) ? skip
   Exit: (30) file_directory_name('/leonardo/home/userexternal/friguzzi/lib/swipl/library/ext/swipy/python/janus.py', '/leonardo/home/userexternal/friguzzi/lib/swipl/library/ext/swipy/python') ? creep
   Call: (30) janus:py_add_lib_dir('/leonardo/home/userexternal/friguzzi/lib/swipl/library/ext/swipy/python', first) ? skip
   Exit: (30) janus:py_add_lib_dir('/leonardo/home/userexternal/friguzzi/lib/swipl/library/ext/swipy/python', first) ? creep
   Call: (30) janus:py_connect_io ? skip
   Exit: (30) janus:py_connect_io ? creep
   Call: (30) janus:repl_add_cwd ? creep
   Call: (31) current_prolog_flag(break_level, _27424) ? creep
   Fail: (31) current_prolog_flag(break_level, _27424) ? creep
   Fail: (30) janus:repl_add_cwd ? creep
   Fail: (29) janus:py_initialize('/leonardo/home/userexternal/friguzzi/lib/swipl/bin/x86_64-linux/swipl', ['-I'], []) ? creep
   Fail: (28) janus:py_initialize ? creep
   Fail: (26) janus:py_call(sys:modules:'__contains__'(numpy), @(true)) ? creep
   Redo: (25) janus:py_module_exists(numpy) ? creep
   Call: (26) janus:py_call(importlib:util:find_spec(numpy), _33148) ? creep
   Exit: (26) janus:py_call(importlib:util:find_spec(numpy), @(none)) ? creep
   Call: (26) @(none)\== @(none) ? creep
   Fail: (26) @(none)\== @(none) ? creep
   Fail: (25) janus:py_module_exists(numpy) ? creep
^  Redo: (13) setup_call_cleanup(system:with_mutex('$load_file', '$mt_start_load'('/leonardo_work/IscrC_NSGP/kg/fbk237/testgpu.pl', <clause>(0xd19a90), [expand(false)])), system:'$mt_do_load'(<clause>(0xd19a90), 'testgpu.pl', '/leonardo_work/IscrC_NSGP/kg/fbk237/testgpu.pl', user, [expand(false)]), system:'$mt_end_load'(<clause>(0xd19a90))) ? creep
   Call: (26) push_msg(initialization_failure(user:main, '/leonardo_work/IscrC_NSGP/kg/fbk237/testgpu.pl':7), _38076) ? skip
main :-
   Exit: (26) push_msg(initialization_failure(user:main, '/leonardo_work/IscrC_NSGP/kg/fbk237/testgpu.pl':7), []) ? creep
   Call: (33) prolog:message_prefix_hook(thread, _39834) ? skip
   Fail: (33) prolog:message_prefix_hook(thread, _39834) ? creep
Warning: /leonardo_work/IscrC_NSGP/kg/fbk237/testgpu.pl:7: Initialization goal failed
   Call: (25) pop_msg([]) ? creep
   Exit: (25) pop_msg([]) ? creep
^  Exit: (18) setup_call_cleanup(system:'$start_run_initialization'([expand(false)], false), system:'$run_initialization_2'('/leonardo_work/IscrC_NSGP/kg/fbk237/testgpu.pl'), system:'$end_run_initialization'(false)) ? creep
^  Exit: (13) setup_call_cleanup(system:with_mutex('$load_file', '$mt_start_load'('/leonardo_work/IscrC_NSGP/kg/fbk237/testgpu.pl', <clause>(0xd19a90), [expand(false)])), system:'$mt_do_load'(<clause>(0xd19a90), 'testgpu.pl', '/leonardo_work/IscrC_NSGP/kg/fbk237/testgpu.pl', user, [expand(false)]), system:'$mt_end_load'(<clause>(0xd19a90))) ? creep
   Call: (14) push_msg(load_file_errors('testgpu.pl', 0, 1), _45226) ? skip
   Exit: (14) push_msg(load_file_errors('testgpu.pl', 0, 1), []) ? creep
   Call: (13) pop_msg([]) ? creep
   Exit: (13) pop_msg([]) ? creep
   Call: (9) push_msg(welcome, _48536) ? skip
   Exit: (9) push_msg(welcome, []) ? creep
   Call: (19) prolog:version_msg(_50330) ? skip
   Fail: (19) prolog:version_msg(_50330) ? creep
Welcome to SWI-Prolog (threaded, 64 bits, version 9.1.22-26-g4eb8604da)
SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software.
Please run ?- license. for legal details.

For online help and background, visit https://www.swi-prolog.org
For built-in help, use ?- help(Topic). or ?- apropos(Word).

   Call: (8) pop_msg([]) ? creep
   Exit: (8) pop_msg([]) ? creep
?-
% halt
12

I’m confused also by the -g and -t options:

(liftcover) [friguzzi@login02 fbk237]$ cat testgpu.pl
main :-
	%  trace,
  py_module_exists(numpy),
  py_module_exists(cupy).

%:- initialization(main).
(liftcover) [friguzzi@login02 fbk237]$ $HOME/bin/swipl testgpu.pl -g main
Welcome to SWI-Prolog (threaded, 64 bits, version 9.1.22-26-g4eb8604da)
SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software.
Please run ?- license. for legal details.

For online help and background, visit https://www.swi-prolog.org
For built-in help, use ?- help(Topic). or ?- apropos(Word).

?-
% halt
(liftcover) [friguzzi@login02 fbk237]$ $HOME/bin/swipl testgpu.pl -t main
Welcome to SWI-Prolog (threaded, 64 bits, version 9.1.22-26-g4eb8604da)
SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software.
Please run ?- license. for legal details.

For online help and background, visit https://www.swi-prolog.org
For built-in help, use ?- help(Topic). or ?- apropos(Word).

?-
% halt
(liftcover) [friguzzi@login02 fbk237]$ $HOME/bin/swipl testgpu.pl
Welcome to SWI-Prolog (threaded, 64 bits, version 9.1.22-26-g4eb8604da)
SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software.
Please run ?- license. for legal details.

For online help and background, visit https://www.swi-prolog.org
For built-in help, use ?- help(Topic). or ?- apropos(Word).

?- main.
% Interactive session; added `.` to Python `sys.path`
Deprecated: instead of py_call(Obj:Attr = Value), use py_setattr(Obj,Attr,Value)
true.

?-
13

That is not good. Pushed a fix. With this failure, the virtual env is not properly setup.

-g is executed before becoming interactive. -t defines the interactive toplevel. That is the difference: on -t, the repl_add_cwd/0 call succeeds and the initialization of Python is properly completed.

P.s. Do not use initialization/1 to run an application. Although it is executed after loading the code that belongs to a file, it still executes in a locked environment as it needs to complete before the call that caused this file to be loaded can complete. That is where initialization(Goal, main) comes in. This goal is executed after all -g goal and prevents the REPL to start unless Goal explicitly re-enables the REPL loop.

14

I thought about it: I think the best would be to have Janus behave as command line Python. So in this case, not removing site-packages.

Another thing that would be useful is to provide a predicate to find out to which Python executable swipl was linked.

15

I agree, but I think it does. At least, here (Ubuntu 22.04). The first run is before entering the venv, the other two are pyhon and Prolog after entering the venv. That all looks pretty consistent to me. Prolog only add the directory holding janus.py to the path.

janw (~) 4_> python
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/home/janw/.local/lib/python3.10/site-packages', '/home/janw/.local/lib/python3.10/cmeel.prefix/lib/python3.10/site-packages', '/usr/local/lib/python3.10/dist-packages', '/usr/lib/python3/dist-packages']
>>> 
janw (~) 5_> source venv/bin/activate
(venv) janw (~) 6_> python
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/home/janw/venv/lib/python3.10/site-packages']
>>> 
(venv) janw (~) 7_> swipl
Welcome to SWI-Prolog (threaded, 64 bits, version 9.1.22-35-gdd909bac0-DIRTY)
SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software.
Please run ?- license. for legal details.

    CMake built from "/home/janw/src/swipl-devel/build.pgo"

For online help and background, visit https://www.swi-prolog.org
For built-in help, use ?- help(Topic). or ?- apropos(Word).

77 ?- py_call(sys:path, L).
% Interactive session; added `.` to Python `sys.path`
L = ['', '/home/janw/src/swipl-devel/build.pgo/home/library/ext/swipy/python', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/home/janw/venv/lib/python3.10/site-packages'].

py_version/0 tells the version, but not where it is installed. That is a little hard. The only way to find out is to get the absolute path of libpython that is linked in, but there is no nice portable way to get that. On Linux, you can run ldd on janus.so in the Prolog tree. On MacOS you do the same using otool -L. On Windows you probably best call win_process_modules/1. In all cases, make sure to have the same environment settings as all dynamic linkers react on the environment.

But, I think the location doesn’t matter too much. The version matters, as well as the Python module search path, which is accessible as sys.path. I could be wrong though; I’m not a Python expert (although I learned a few tricks in the past months :slight_smile: ).

16

The problem is you can create an environment with

python -m venv liftcover --system-site-packages

so in this case the site-packages should not be removed. See https://docs.python.org/3/library/venv.html#module-venv

17

Thanks. Pushed a patch to take care of this. I now get the same sys.path as Python in the venv. For this scenario it is a bit of a pity that one has to take care of setting up the environment yourself when embedding Python :frowning: That is what I found on SO, but no comprehensive description of what needs to be done to mimic Python’s default behavior. We’ll have to get there incrementally :frowning: