Janus: loading a Python module from a new Python module search path raises ModuleNotFoundError

General
1

I’m trying to load a Python module from a directory added to the Python module search path with py_add_lib_dir/1 and failing. The module search path seems to be recognised but the file is still not found.

The below code and output shows how I’m trying to add the new Python module search path and then use it, and the resulting error:

% Directive adding module_path to Python module search path.
:- py_add_lib_dir(model_data).

% Part of program calling the Python module under the new search path. 
% This is in the same Prolog module as the directive adding the search path.
environment_init(E,[Sim|Fs],q0,O0,[Sim|Gs]):-
    ...
    ,py_call(basic_sim:init_sim(agent_start=[Xpv,Ypv]),Sim)
    ...


% Checking that the module search path is correctly added; it's the first one:
?- py_lib_dirs(_Ds), maplist(writeln,_Ds).
c:/users/yegoblynqueenne/documents/prolog/ilp_systems/louise/data/bath/model/data/model_data
c:/program files/swipl/library/ext/swipy/python

c:/program files/python310/python310.zip
c:/Program Files/Python310/Lib
c:/Program Files/Python310/DLLs
c:/Program Files/swipl/bin
c:/Users/YeGoblynQueenne/AppData/Roaming/Python/Python310/site-packages
c:/Program Files/Python310
c:/Program Files/Python310/lib/site-packages
c:/Program Files/Python310/lib/site-packages/vboxapi-1.0-py3.10.egg
true.

% Call to environment_init/5 that raises an error:
?- basic_sim_grid:environment_init('Episode0',Fs,Q0,O0,Gs).
ERROR: Python 'ModuleNotFoundError':
ERROR:   No module named 'basic_sim'
ERROR: In:
ERROR:   [13] janus:py_call(basic_sim:init_sim(...),_17572)
ERROR:   [12] basic_sim_grid:environment_init('Episode0',[_17624,...|...],q0,_17618,[_17636,...|...]) at c:/users/yegoblynqueenne/documents/prolog/ilp_systems/louise/data/bath/model/data/basic_sim_grid_environment.pl:32
ERROR:   [11] toplevel_call(user:basic_sim_grid: ...) at c:/program files/swipl/boot/toplevel.pl:1318
^  Exception: (4) setup_call_cleanup('$toplevel':notrace(call_repl_loop_hook(begin, 0)), '$toplevel':'$query_loop'(0), '$toplevel':notrace(call_repl_loop_hook(end, 0))) ? abort
% Execution Aborted

If I move the Python file I’m trying to load (basic_sim.py) to the same directory as the running Prolog process, it is loaded without error.

So it looks like the new Python module search path is ignored, even though it is reported by py_lib_dirs/1.

I’m on SWI-Prolog 9.3.8-27-g80f956be2 64 bits.

2

I’m asking too many Janus questions ain’t I? :slight_smile:

Yesterday Jan pointed out a different mechanism to add Python code to a Prolog project using py_module/2, but it seems like I’ll have to write a bit of Python and I figured it’s better to have it in separate file(s) after all. That will make it easier to add comments etc. to the Python code.

3

I don’t know. Could be some Windows \/ issue as development is on Linux as is most of the testing. You can run

?- py_shell.

To get an interactive Python shell for the embedded Python system. That allows for some more comfortable looking around.

I wouldn’t say so. I consider Janus an important component of the system and something that should run as smoothly as we can get it.

1 Like
4

Hi Jan,

I tried the same queries on Windows Subsystem for Linux (Ubuntu 24.04) and I thought I might have better luck this time because I saw this message when I loaded my project, which I hadn’t seen on windows:

% Interactive session; added `.` to Python `sys.path`

But it doesn’t seem that’s got anything to do with it because I still get the same behaviour:

?- py_lib_dirs(_Ds), maplist(writeln,_Ds).
/mnt/c/Users/YeGoblynQueenne/Documents/Prolog/ILP_systems/louise/data/bath/model/data/model_data

/home/yegoblynqueenne/swipl-devel/build/home/library/ext/swipy/python
/opt/ros/humble/lib/python3.10/site-packages
/opt/ros/humble/local/lib/python3.10/dist-packages
/usr/lib/python310.zip
/usr/lib/python3.10
/usr/lib/python3.10/lib-dynload
/home/yegoblynqueenne/.local/lib/python3.10/site-packages
/usr/local/lib/python3.10/dist-packages
/usr/lib/python3/dist-packages
true.

?- environment_init('Episode6',_Fs,_Q0,_O0,_Gs),executor(_Fs,_Q0,_O0,_Gs,_As,[XY,_Ms]), print_map(tiles,_Ms).
ERROR: Python 'ModuleNotFoundError':
ERROR:   No module named 'basic_sim'
ERROR: In:
ERROR:   [14] janus:py_call(basic_sim:init_sim(...,...),_20706)
ERROR:   [13] basic_sim_grid:environment_init('Episode6','<garbage_collected>','<garbage_collected>',_20754,'<garbage_collected>') at /mnt/c/Users/YeGoblynQueenne/Documents/Prolog/ILP_systems/louise/data/bath/model/data/basic_sim_grid_environment.pl:33
ERROR:   [12] '<meta-call>'('<garbage_collected>') <foreign>
ERROR:   [11] toplevel_call('<garbage_collected>') at /home/yegoblynqueenne/swipl-devel/build/home/boot/toplevel.pl:1318

I can try on a native Linux installation (not WSL) later if that will make a difference.

Yikes. I don’t know what to do with that. My Python is really not that good. But I did notice something. When I list the Python path I see there’s an empty string in it:

?- py_shell.
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import sys
>>> print(sys.path)
['/mnt/c/Users/YeGoblynQueenne/Documents/Prolog/ILP_systems/louise/data/bath/model/data/model_data', '', '/home/yegoblynqueenne/swipl-devel/build/home/library/ext/swipy/python', '/opt/ros/humble/lib/python3.10/site-packages', '/opt/ros/humble/local/lib/python3.10/dist-packages', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/home/yegoblynqueenne/.local/lib/python3.10/site-packages', '/usr/local/lib/python3.10/dist-packages', '/usr/lib/python3/dist-packages']

See the second path, after my added path. Any chance that this is causing some trouble? I don’t think this is coming from my code.

Alternatively, is there a mechanism in Janus to point to a Python module by an absolute path so that I don’t rely on the module search path? I considered py_import/2 but that doesn’t seem to be the way to use it.

5

That’s great then but please let me know if my style of describing issues is confusing :slight_smile:

6

Something interesting. I tried loading my Python module explicitly, with an absolute path, and here’s what happened. This is in WSL and the file paths are very long so below I’m showing the errors with shortened paths that’s easier to read. I’m showing only the relevant part of the errors for brevity, I hope that’s OK. I also show how I’m calling py_import/2:

% Importing Python module with:
% ,expand_file_search_path(model_data('basic_sim.py'),BS)
% ,py_import(BS,[as(basic_sim)])

ERROR: Python 'ModuleNotFoundError':
ERROR:   No module named '/mnt/c/.../model/data/basic_sim'
ERROR: In:
ERROR:   [15] janus:py_import_('/mnt/c/.../model/data/basic_sim.py',[as(basic_sim)])

Note that the system is trying to find a module named basic_sim, instead of basic_sim.py. I tried removing the .py extension from the file and the py_import/2 call, and I got the same error:

% Importing Python module with:
% ,expand_file_search_path(model_data('basic_sim'),BS)
% ,py_import(BS,[as(basic_sim)])

ERROR: Python 'ModuleNotFoundError':
ERROR:   No module named '/mnt/c/...model/data/basic_sim'
ERROR: In:
ERROR:   [15] janus:py_import_('/mnt/c/...model/data/basic_sim',[as(basic_sim)])

I also tried calling py_import/2 with an empty list of options and I noticed it adds as(py) automatically:

% Importing Python module with:
% ,expand_file_search_path(model_data('basic_sim.py'),BS)
% ,py_import(BS,[])

ERROR: Python 'ModuleNotFoundError':
ERROR:   No module named '/mnt/c/...model/data/basic_sim'
ERROR: In:
ERROR:   [15] janus:py_import_('/mnt/c/...model/data/basic_sim.py',[as(py)])

But if I don’t add the py extension in the file search path, then it gets really weird and verbose:

% Importing Python module with:
% ,expand_file_search_path(model_data('basic_sim'),BS)
% ,py_import(BS,[])

ERROR: Python 'ModuleNotFoundError':
ERROR:   No module named '/mnt/c/.../model/data/basic_sim'
ERROR: In:
ERROR:   [15] janus:py_import_('/mnt/c/Users/YeGoblynQueenne/Documents/Prolog/ILP_systems/louise/data/bath/model/data/basic_sim',[as('/mnt/c/Users/YeGoblynQueenne/Documents/Prolog/ILP_systems/louise/data/bath/model/data/basic_sim')])

Note the whole path is now added to the as/1 option. This looks like there’s some error in the logic that separates file paths into basenames and extensions when looking for a python module.

Is there something I can run int he python shell to confirm this?

7

IIRC, Windows has some limits on path lengths that are shorter than the usual Linux limits - I speculate that this could also apply to WSL because the underlying file system uses Windows. (I’ve been bitten by this in the past, but forget the details … I think it was the case that I could list a directory (folder) but couldn’t access a file inside it because the combined path length was too long.)

1 Like
8

I seem to remember something like that too, but here it seems the Python module search path I declare is searched correctly. I’ll have a poke around on my native Linux too though just to be sure.

9

py_import/2 is not intended to load absolute file names. it is intended to deal with Python submodules, but using the normal Python search mechanism.

I’m a bit worried about this '/mnt/. On Windows we also need to be careful about absolute file names longer than 128 characters. Windows can deal with that, but it requires some precautions. Do you exceed this limit?

10

No, the WSL path is 78 characters, including the slashes. This is the full path on WSL:

/mnt/c/Users/YeGoblynQueenne/Documents/Prolog/ILP_systems/louise/basic_sim.py
11

Make sure you don’t have another visible directory named basic_sim/ since if you do it will look for basic_sim/__init__.py as if this was your module impl code (which will be missing!)

1 Like
12

The other possiblity is that your basic_sim.py is having an error (like with its dependancies or whatever) while its loading and so the module doesnt get defined in the end

You can try to wrap it in yet another module:

Here i am making hyperon_module import the hyperon so if hyperon is not found I find out while loading hyperon_module.

1 Like
13

Thanks. Unfortunately that’s not it, I have no such directory.

I don’t think it’s that either. If I copy my Python file to the directory from which I launch my Prolog process it is loaded correctly. I think that Janus sets Python’s ‘.’ directory to the Prolog process’ current directory so it’s included in Python’s search path (I’m guessing).

I’ll try to make a simpler example just in case there’s something in my (large) project that is confusing things, and see if I can reproduce the error with that simpler example. In fact, I should remember to do that every time I report a problem here.

1 Like
14

I think I’ve figured it out. I’ve made a minimal example and sent it to @jan. I’m copying it here for the benefit of the community.

In short, I think that the problem is in py_add_lib_dir/1 after all. When the argument to that predicate is an alias for a directory nested at least two layers deep, it is not expanded correctly.

Here’s what I mean. Suppose I define these file search paths:

user:file_search_path(data, project_root(data)).
user:file_search_path(patata, data(patata)).

Now, if I try to add project_root/data/patata to the Python module search path, this is how the module search path is expanded:

:- py_add_lib_dir(patata).

?- py_lib_dirs(_Ds), maplist(writeln,_Ds).
c:/python_modules/errors/patata

Where c:/python_modules/errors/ is the project_root directory. As you see, the “data” component of the path is missing. If I add project_root/data to the Python module search path instead then the path will be correct, but that’s only by accident:

:- py_add_lib_dir(data).

?- py_lib_dirs(_Ds), maplist(writeln,_Ds).
c:/python_modules/no_errors/data

That is, I think the file expansion in py_add_lib_dir/1 only keeps the tip of the directory branch when it expands a path.

Here’s a more complete walkthrough on my minimal example. I have two directories as follows:

├───errors
│   │   load_project.pl
│   │
│   └───data
│       └───patata
│               here.py
│
└───no_errors
    │   load_project.pl
    │
    └───data
        │   here.py
        │
        └───patata

Both instances of the project_load.pl file are the same, except for the path they add to the Python module search path.

The file under errors adds data(patata) to the Python module search path:

% Contents of errors/load_project.pl:

:-prolog_load_context(directory, Dir)
,asserta(user:file_search_path(project_root, Dir)).

user:file_search_path(data, project_root(data)).
user:file_search_path(patata, data(patata)).

:- edit(project_root(load_project)).

:- py_add_lib_dir(patata).

where_are_you:-
        py_call(here:where_am_i()).

Now if I call where_are_you/0, there’s the familiar error:

?- where_are_you.
ERROR: Python 'ModuleNotFoundError':
ERROR:   No module named 'here'
ERROR: In:
ERROR:   [13] janus:py_call(here:where_am_i())
ERROR:   [12] where_are_you at c:/python_modules/errors/load_project.pl:15
ERROR:   [11] toplevel_call(user:user:where_are_you) at c:/program files/swipl/boot/toplevel.pl:1318

The file under no_errors adds the directory data to the Python module search path:

% Contents of no_errors/load_project.pl:

:-prolog_load_context(directory, Dir)
,asserta(user:file_search_path(project_root, Dir)).

user:file_search_path(data, project_root(data)).
user:file_search_path(patata, data(patata)).

:- edit(project_root(load_project)).

:- py_add_lib_dir(data).

where_are_you:-
        py_call(here:where_am_i()).

With this file loaded, I can call where_are_you/0 without error:

?- where_are_you.
I am here
true.

Note that I’m now on SWI-Prolog 9.3.10, 64 bits. Still on WIN 11. I can try on linux too if necessary, but the concern about OS’s was with path lengths and I did all this just under windows’ C: so I don’t think that’s an issue after all.

Cheers,
Stassa

15

Answer from @jan copied with his permission:

Hi Stassa,

Nope :slight_smile: The argument you give is patata, which is not interpreted as
using the patata alias, but simply as a relative file name. As it
should. py_add_lib_dir/1,2 did not properly handle Alias(Dir). Pushed
a fix for that. The way to wrire it though is as

:- py_add_lib_dir(patata(.)).

That works after updating to the current git version.

    Cheers --- Jan

As usual I only checked my mailbox today figuring Jan would take the afternoon off instead of reading my bug report, only to find that he had sent that half an hour after my email and pushed a fix for the error he found (which wasn’t the reason for my problem btw).

Jan, Dude. Weekends. You gotta rest :slight_smile:

16

Ahah, I had stopped using py_add_lib_dir/1 for whatever reason. I didn’t realize there was an issue, thank you both for taking to the time to catch and fix it!

For those stuck on older versions who need to add directories to their Python paths, I’ve manually been setting the Python environment variable. (I’ve done this before with setenv/2 before initalizing Janus)

But probably saner before starting:

export PYTHONPATH=/path/to/your/python/dir:$PYTHONPATH

This ensures that the Python interpreter includes the necessary directories when searching for modules.

It’s a straightforward workaround, especially useful if you’re on an older version of SWI or not building from source.

2 Likes