Janus: calling python exec fails

General
1

I am wondering why the following code fails, is there anything I am doing wrong?

:- use_module(library(strings)).
main(R) :-
       S={|string||
   import nltk
   from nltk.tokenize import sent_tokenize, word_tokenize

   nltk.download('punkt')  # Download the punkt tokenizer models if you haven't already.

   def truecase_text(text):
       # Tokenize the text into sentences and then words.
       sentences = sent_tokenize(text)
       truecased_sentences = []

       for sentence in sentences:
           words = word_tokenize(sentence)
           truecased_words = []

           for word in words:
               # Use NLTK's built-in `title()` function to truecase each word.
               truecased_word = word.title()
               truecased_words.append(truecased_word)

           # Reconstruct the truecased sentence from the truecased words.
           truecased_sentence = ' '.join(truecased_words)
           truecased_sentences.append(truecased_sentence)

       # Reconstruct the truecased text from the truecased sentences.
       truecased_text = ' '.join(truecased_sentences)

       return truecased_text

   # Example usage:
   text = "this is an example. it demonstrates truecasing."
   truecased_text = truecase_text(text)
   print(truecased_text)
   |},
   py_call(exec(S),R).

when I run it I get a frame does not exist error:

2 ?- main(R).
ERROR: Python 'SystemError':
ERROR:   frame does not exist
ERROR: In:
ERROR:   [11] janus:py_call(exec("import nltk\nfrom nltk.tokenize import sent_tokenize, word_tokenize\n\nnltk.download('punkt')  # Download the punkt tokenizer models if you haven't already.\n\ndef truecase_text(text):\n    # Tokenize the text into sentences and then words.\n    sentences = sent_tokenize(text)\n    truecased_sentences = []\n\n    for sentence in sentences:\n        words = word_tokenize(sentence)\n        truecased_words = []\n\n        for word in words:\n            # Use NLTK's built-in `title()` function to truecase each word.\n            truecased_word = word.title()\n            truecased_words.append(truecased_word)\n\n        # Reconstruct the truecased sentence from the truecased words.\n        truecased_sentence = ' '.join(truecased_words)\n        truecased_sentences.append(truecased_sentence)\n\n    # Reconstruct the truecased text from the truecased sentences.\n    truecased_text = ' '.join(truecased_sentences)\n\n    return truecased_text\n\n# Example usage:\ntext = \"this is an example. it demonstrates truecasing.\"\ntruecased_text = truecase_text(text)\nprint(truecased_text)\n"),_412)
ERROR:    [9] toplevel_call(user:user: ...) at /swi/home/boot/toplevel.pl:1235
ERROR: 
ERROR: Note: some frames are missing due to last-call optimization.
ERROR: Re-run your program in debug mode (:- debug.) to get more detail.
3 ?-
2

Good question. Probably there are more informed answers in a Python forum :slight_smile: This applies for anything you want to exec():

101 ?- py_call(exec("import sys\n"), V).
ERROR: Python 'SystemError':
ERROR:   frame does not exist

That could be the reason why e.g. Py_Run() and similar interfaces exist? This does work:

104 ?- py_run("import sys\n", py{}, py{}, R, []).
Correct to: "janus:py_run(\"import sys\\n\",py{},py{},R,[])"? yes
R = @(none).

Possibly we should make py_run/5 public? Iā€™m afraid I donā€™t really understand this interface though. Alternatively we could decide that this stuff must be in a Python module. That was surely the original intend of the Janus developers. Does that make sense?

3

Seems to be a prolog problem, not a python one, it works fine in the py_shell:

25 ?- py_shell.
Python 3.11.5 (main, Sep  2 2023, 14:16:33) [GCC 13.2.1 20230801] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> exec("print('hello')")
hello
>>>

yes, I think this is very useful, to define python function within prolog by passing a string.

EDIT: hmm, how do you call a function defined with py_run/5?

37 ?- py_run("def myfun(): print('myfun')",py{},py{},R,[]).
Correct to: "janus:py_run(\"def myfun(): print('myfun')\",py{},py{},R,[])"? yes
R = @(none).

38 ?- py_call(myfun(),R).
ERROR: python_builtin `myfun' does not exist
ERROR: In:
ERROR:   [10] janus:py_call(myfun(),_13326)
ERROR:    [9] toplevel_call(user:user: ...) at /swi/home/boot/toplevel.pl:1235
39 ?-
4

There seems to be something special when making calls to Python directly from C. It probably has to do with ā€œPython framesā€, but do not ask ā€¦ Iā€™m a Prolog expert slowly understanding the Python environment.

The question is how you get this defined in some module and if you canā€™t/donā€™t specify the module, where it ends up. That too is more a Python question. An unqualified function call by py_call/2 resolves the function as a built-in.

5

It seems to be stored in module ā€˜Noneā€™, so I tried this:

42 ?- py_run("def myfun(): \n   print('myfun')\nprint('module: ',myfun.__module__)",py{},py{},R,[]).
Correct to: "janus:py_run(\"def myfun(): \\n   print('myfun')\\nprint('module: ',myfun.__module__)\",py{},py{},R,[])"? yes
module:  None
R = @(none).

43 ?- py_call('None':myfun(),R).
ERROR: Python 'TypeError':
ERROR:   'NoneType' object is not callable
ERROR: In:
ERROR:   [10] janus:py_call('None':myfun(),_12586)
ERROR:    [9] toplevel_call(user:user: ...) at /swi/home/boot/toplevel.pl:1235

You canā€™t add a function to a module (python) without creating a file.

6

This is from Prolog not ā€˜Noneā€™, but the Python object None represented as @none in Prolog (at least, that is what I suspect). I think it trying to tell us that myfun() has no module. Next question is you get access to a function object that is not part of a module? Normally, functions are simply attributes of their module.

7

I can call it with py_run/5, but inside the same string:

29 ?- py_run("def myfun(): \n   print('hello I am myfun')\nprint('module: ',myfun.__module__)\nmyfun()",py{},py{},R,[]).
Correct to: "janus:py_run(\"def myfun(): \\n   print('hello I am myfun')\\nprint('module: ',myfun.__module__)\\nmyfun()\",py{},py{},R,[])"? yes
module:  None
hello I am myfun
R = @(none).

Notice the python code inside py_run:

def myfun():
   print('hello I am myfun')
print('module: ',myfun.__module__)
myfun()

The call to myfun() inside the py_run string works fine, because it prints:

module:  None
hello I am myfun
R = @(none).

however, it does not work with py_call/1:

30 ?- py_call(@(none):myfun()).
ERROR: python_builtin `@' does not exist
ERROR: In:
ERROR:   [10] janus:py_call(@(none):myfun())
ERROR:    [9] toplevel_call(user:user: ...) at /swi/home/boot/toplevel.pl:1235

EDIT: the use case is one prolog source file (script, for example) that includes python source code in a string. The simple solution is to write the string to a file and import that, but this does not work in a read-only filesystem.

8

That will never work as, even if @none maps to the Python None, this has no attributes. I think you should read this as ā€œmyfunc has no moduleā€.

I understand. You have to figure out how to create a Python module from a string. If you know how to do that it might already be possible using the current interface. If not, we can think about making it possible.

9

Figured out how to do it:

:- use_module(library(strings)).
pyfunc :-
    S={|string||
       def pyhello(num):
          print("hello ",num+1)
       print("module",pyhello.__module__)
    |},
    py_call(globals(),Globals,[py_object(true)]),
    py_call(exec(S,Globals),R).

1 ?- pyfunc.
module __main__
true.

2 ?- py_call('__main__':pyhello(5)).
hello  6
true.

What was missing was the globals() object for the exec call, which is why it complained about missing a frame and also the reason why it worked in py_shell (because the globals was already there). See python - How to load a module from code in a string? - Stack Overflow to make a module with a different name.

EDIT: the documentation for py_call/3 specifies:

Call Python and return the result of the called function. Call has the shapeā€˜[Target][:Action]*`, where Target is either a Python module name or a Python object reference. Each Action is either an atom to get the denoted attribute from current Target or it is a compound term where the first argument is the function or method name and the arguments provide the parameters to the Python function.

could you give an example on how to pass several Actions to py_call?

10

Note that you can do

py_call(exec(S, eval(globals()), R).

Iā€™m still unsure whether we want the eval() there.

This implements chaining. There is an example in the library to add a dir to sys:path as

py_call(sys:path:append(Dir)).

Typically, you want as few as possible intermediate explicit Python objects references in Prolog. To create such as reference we must create an atom. This will be garbage collected later, which implies work and later reclaim of the Python object than needed. It is also relatively complex as the atom garbage collector cannot grab the Python GIL needed to decrement the reference count (would deadlock too easily). So, the Python objects referenced is added to a queue and this queue is emptied whenever Prolog gets the GIL for some other reason.

That makes chaining and eval() attractive: more compact, faster and fewer objects known to Prolog. But, even the explicit route is pretty fast and there is typically no reason to not let code clarity prevail.

Thanks for the link. That allows us to define py_load(Module, Source). Would that be good? Better name? py_module? py_create_module? Longer names are not very attractive as it pushes the Python code further to the right with normal layout.

1 Like
11

Added to the current git. Note that this is just a POC. Suggestions for a better name and better handling when reloading the Prolog file are welcome.

12

This is great! Most especially replacing the module if the string hash changes. This is very useful for reloading. I think a better name is py_module(...), since the primary effect is to create/modify a module.

13

Agree. Pushed a change.

1 Like