I’m observing some curious module loading behavior.
I have the following directory structure:
subdir
foo.pl
bar.pl
contents of foo.pl:
:- module(foo, []).
:- use_module(bar).
contents of bar.pl:
:- module(bar, []).
:- format("hello!~n").
Now, two different things happen when I load foo.pl, depending on whether I’m inside the subdir or not.
inside the subdir:
$ swipl -f foo.pl -g halt
ERROR: /tmp/example/subdir/foo.pl:2:
ERROR: source_sink `bar' does not exist
Warning: /tmp/example/subdir/foo.pl:2:
Warning: Goal (directive) failed: foo:use_module(bar)
in the root dir:
$ swipl -f subdir/foo.pl -g halt
hello!
That is very surprising to me. I would not expect the module system to care what my working directory is. In fact, the docs for absolute_file_path/3, under relative_to indicate that absolute_file_name should use the source file rather than the current working directory as a reference when loading from an atomic spec. And yet, in this case it very clearly does not.
Note that if I place foo.pl and bar.pl in the same directory, then no matter what my working dir is, the import will be correct. No mysterious loading from the current working dir in that case. It looks like the source file relative load always takes precedence. But if an atomic path cannot be resolved relative to the loading file, it seems to try the current working dir as a fallback.
Is there a reason for this behavior? And is there a way to turn it off? I don’t like module loads working or not working depending on what directory I’m in.
History I fear Might indeed be a good idea to drop that. Should we then add a search path cwd or (maybe better) working_directory to explicitly allow loading files from the working directory, regardless of the current location?
Explicit behavior is better than implicit behavior, so if it has to be kept around in some way, that sounds like the way to go. That said, I don’t fully understand what the use case for this behavior is. Why would you want to have a search path that moves around with the cwd?
There is a bit of a problem. Directives are normally executed with the same resolution for absolute_file_name/3, i.e., taking the directory of the source first and the current directory second. Working relative to the source is typically what you want for normal directives that are part of loading the program. Directives executed as part of running the application however should work relative to the working directory (I think). Now, SWI-Prolog has initialization/2, in particular :- initialization(Goal, main). to start the application, but ISO does not and many people start the application from an initialization/1 directive in the main file.
Now, it is bad that in the current situation the system will first look in the source directory and than in the working directory for the application main goal.
We can easily define different rules for load_files/2 (subsuming all source loading predicates) than for absolute_file_name/3 in a directive.
Still, what is the proper behaviour, where we would like to avoid major compatibility issues …
Ah, it is not as bad as I thought. initialization/1 as and the main initialization neither provide a source location and these are thus executed relative to the working directory. That is a relief.
I can confirm that the file-search branch doesn’t show the behavior I described in my opening post. I can also make use of the working_directory file_search_path.
Hey, I wonder if that was the change that broke my older code that assumed explicit paths were always relative to the directory of the SWI-Prolog process (see here). For what it’s worth I feel that is the intuitive behaviour but maybe that’s just my Stockholm Syndrome after working like that for a few years now.
Yes. This is the relevant change. I think it is sound. As I’ve seen from your load_project.pl, your setup file search paths. That is ok. Any internal reference to other source files in the project should either use these file search path aliases or a relative path to the file itself. Using the program’s working directory will break the program when started from another directory.
I guess I was hasty to say “an easy fix”. This change keeps breaking my code unexpectedly because it is only raised when I try to run the code that uses an explicit path. Just now I was trying to run some code from a repository that contains some experiments from an older paper and it broke, which means it will break for anyone trying to reproduce the experiments.
I don’t want to add more work to you Jan, but was there no way to fix this without making a breaking change that would raise an error in older code?
Ultimately, what is the real problem with having a less rigid way to access a file from another file, keeping in mind that SWI-Prolog does not define a default directory structure for projects?
That’s an interesting way of doing prolog project setup.
I guess the intention here was that every module path is relative to a project root, which happened to work out like that cause that’s where swipl was launched?
I do see the logic in wanting project-root-relative paths. But that wasn’t what it was doing before. A more consistent way of getting that kind of behavior is by defining a file_search_path.
% load_paths.pl in your project root
:- prolog_load_context(directory, ProjectRoot),
asserta(user:file_search_path(project, ProjectRoot)).
Then you can just consule that file wherever you need a load path setup like so:
For projects that have a natural starting point, like a main file, it makes sense to just do file search path setup from there before loading anything else, and then you don’t have to consult anything in individual files.
The cwd-relative behavior is still there, it’s just explicitly opt in now.
use_module(working_directory(path/to/foo)).
This will do module imports relative to the working directory, which may or may not line up with your project root depending on how you load your code.
IMO it’s not a huge problem, as long as file-relative takes precedence. It’s just weird and inconsistent. You might get a load error (or worse, load an entirely different file) if you start your program from anywhere but the project root. This will bite you if you ever need to install this as a system-wide application, or if you want to reuse your code from another project, or if you are going to ship it to someone who will want to run it differently from what you intended. But probably you’re not doing any of those things.
There may not be a default directory structure for projects, but file-relative loading lends itself well for setting up any project structure, and it is how most every programming language out there does file loads.
Personally I very much don’t want SWI-Prolog to even try to magically load files in my current working directory if it doesn’t find them where they should be (file relative). I just want an error in that case.
If the old behavior is to be reinstated I’d like it if there was a prolog flag controlling the behavior.
Thanks for the comprehensive overview. I fully agree with this change. I don’t like adding a flag to restore the old behaviour and surely the default would be the new behaviour. Today I did have to spend about 15 min on an old project as it failed to load for these reasons.
In general, backward compatibility is not perfect and thus old projects either need to stick with the Prolog version they were build for or they may need a little updating. That is hard to avoid
I’m interested in who is having how much trouble with the behaviour. If there are enough people, I guess we should introduce a flag
I was thinking of a warning rather than a flag. For example, in an even older project I get this warning because of the way I set the stack limit:
Warning: set_prolog_stack/2: limit(Size) sets the combined limit.
Warning: See https://www.swi-prolog.org/changes/stack-limit.html
And I just noticed that this older project also fails because of the change to the file paths. For me it’s probably affecting every SWI-Prolog project I ever worked on because I like to spread things out over multiple files and directories
To clarify, I’m starting all my projects from a project-load file that sets file search paths. I linked to an example of such a project load file in an earlier comment.
The problem is when I have a file A that refers to a file B, where both A and B are in the same directory (or B may be in a directory under A’s directory). In that case, I will sometimes use the file search path defined in the project load file, which is certainly good practice, but sometimes I will just refer to file B with an explicit path that is relative to file A. That’s where I get all the errors now.
That’s what I mean by “less rigid” way to refer to files. That way, which is the way it worked until now, leaves it up to the user and does not try to anticipate specific use cases, which is always a bad idea. See all the SO messages that begin "why would you eve want to… " etc.
I’m very surprised to learn this doesn’t work for you. This is how I do most of my loads.
without an alias, predicates like use_module/1, but also just consulting, should be file relative.
As in, if this is in a file foo.pl that is deeply nested in some project structure, like maybe prolog/submodule/foo/foo.pl, this will load prolog/submodule/foo/sibling.pl, no matter what the path is to swipl.
Example:
> cat main.pl
:- [prolog/submodule/foo/foo].
> cat prolog/submodule/foo/foo.pl
:- [sibling].
> cat prolog/submodule/foo/sibling.pl
:- format("hi hello I am sibling I am being consulted.~n").
> swipl -f main.pl -q -g halt
hi hello I am sibling I am being consulted.
Yeah, sorry, I got that wrong. That’s not when I get errors or at least that’s not the errors I’ve noticed recently. I explained my problem in another post linked from this comment:
Indeed, reusing my code from other projects is a major use case for me. My projects are composed of libraries composed of libraries, etc. That’s one way that the new change is “biting me” as you say.
The problem, at least as I am experiencing it, arises when a file is looking for a path relative to the cwd not for a file in the cwd. It’s relative paths that are now breaking. See my earlier post for examples.
Btw, I was curious about your previous comment:
I don’t know “most every programming language”, for sure, but I’ve tried this with R and it sure enough takes the cwd as the root for relative paths. For example, below I started the R console in its default path and tried to source a script in another path, and the script fails because it’s trying to source from a file in the path ./data/data.r relative to the script, which is not found relative to the cwd:
> getwd()
[1] "C:/Users/YeGoblynQueenne/Documents"
> source("C:\\Users\\YeGoblynQueenne\\Documents\\Prolog\\blood_pressure\\scripts\\bp_averaged.r")
Error in file(filename, "r", encoding = encoding) :
cannot open the connection
In addition: Warning message:
In file(filename, "r", encoding = encoding) :
cannot open file '../data/data.r': No such file or directory
After I cd’d to the script directory the script was executed without trouble:
> setwd("C:\\Users\\YeGoblynQueenne\\Documents\\Prolog\\blood_pressure\\scripts\\")
> source("C:\\Users\\YeGoblynQueenne\\Documents\\Prolog\\blood_pressure\\scripts\\bp_averaged.r")
Loading required package: lattice
Loading required package: survival
Loading required package: Formula
Loading required package: ggplot2
Attaching package: ‘Hmisc’
The following objects are masked from ‘package:base’:
format.pval, units
# ... etc
I also tried the same thing with Python. This is me trying to run a Python script from my Documens directory. The script fails because it can’t find a file params.txt that it’s looking for in python’s cwd:
> C:\Users\YeGoblynQueenne\Documents
> python .\Prolog\ILP_systems\louise\data\bath\python\0000_exec_fsc_Episode0.py
Traceback (most recent call last):
File "C:\Users\YeGoblynQueenne\Documents\Prolog\ILP_systems\louise\data\bath\python\0000_exec_fsc_Episode0.py", line 2, in <module>
ss = survey_simulation.SurveySimulation(mode='test',save_loc='../episodes/playback/Episode0',map_path='./maps/Map1.png',agent_start=[60.0,190.0])
File "C:\Users\YeGoblynQueenne\AppData\Roaming\Python\Python310\site-packages\survey_simulation\simple_survey_sim.py", line 36, in __init__
params = self.load_params(params_filepath)
File "C:\Users\YeGoblynQueenne\AppData\Roaming\Python\Python310\site-packages\survey_simulation\simple_survey_sim.py", line 150, in load_params
with open(param_file) as fh:
FileNotFoundError: [Errno 2] No such file or directory: 'params.txt'
If I change path to where the params.txt is, the script runs without error:
> python 0000_exec_fsc_Episode0.py
Overwriting the following parameters:
save_loc = ../episodes/playback/Episode0
map_path = ./maps/Map1.png
agent_start = [60.0, 190.0]
Saving output to Episode 0
Overwriting the following parameters:
save_loc = ../episodes/playback/Episode0
map_path = ./maps/Map1.png
agent_start = [60.0, 190.0]
# ... etc
Now, the R project is mine, but the Python project is not. I’m guessing that if the Python project is trying to find a file in a directory relative to the cwd, i.e. the path of the currently running python process, then that’s not by chance and that it’s the standard way to do things in Python. But, of course, I couldn’t really prove that unless I looked at a few thousand Python projects.
Anyway, @jan, I won’t kvetch about this no more, because it seems you have your reasons to make the change you made even though I don’t understand it, after all. It’s a breaking change and you say that it caused you too some PITA, and for what? Why break something that worked and didn’t bother anyone? However, at the end of the day, SWI-Prolog is your project, not mine, I just use it for free, so you do whatever you like with it. Just, please, next time when there’s a possibly breaking change would it be possible to warn about it beforehand, so that I (and I bet others) don’t wake up to find their projects suddenly raise strange new errors out of nowhere?
That is not the way it works. Yes, in the end I take a decision, but this should be as a reasonable one. I (still) think @maren had a good point. And yes, I had to make changes as well, but IMO that made the code better.
Many programming languages seem to resolve a relative path to a set of directories being searched. I think that is also the case for Python using sys.path? If Python is used interactively, it also searches relative to ., but in non-interactive mode is does not (AFAIK). Anyway, SWI-Prolog, inherited from Quintus, has Search(File) notation for that. Plain file names are not subject to this search mechanism. Possibly they should? That would allow users to define their own rules, but it requires adding two reserved search path: one to which plain file names are mapped and one that allows specifying search relative to the current source. So, we could do
Where any non-absolute file is mapped to plain(File) and source is a reserved search path referring to the directory of the file currently being loaded. Would that make sense?
Alternative, I think we should decide this change was too drastic (that is why we have the development series) and it should be changed to by default print a warning with a flag to disable using the working directory completely (and get the 9.3.9 behaviour).
Might indeed be a good idea to drop that. Should we then add a search path 