Using library(persistency) inside pengines

Help!
1

I am attempting to set up a server using pengines in prolog which loads facts from a file using the persistency library and then use a javascript client to make queries of the prolog server and am having a problem accessing the persisted facts from the pengine client.

My repository is linked here.

The server is run from prolog like so:

:- use_module(library(pengines)).
:- use_module(library(http/http_server)).

% Load your Prolog files
:- consult('ais_system.pl').
attach_db('facts.pl').

% Start the HTTP server on port 5000
server(Port) :-
    http_server(http_dispatch, [port(Port)]),
    format('Pengines server running at http://localhost:~d/~n', [Port]).

:- initialization(server(5000)).

And the client makes a request db:get_all_ais_ping(_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,"Under way using engine",_,_,_,_,_,_,_,Matches) using the javascript library.

I get the following response:

An error occurred: {
  code: 'permission_error',
  data: "No permission to call sandboxed `with_mutex(_3490,_3492)'\n" +
    'Reachable from:\n' +
    '\t  db:get_all_ais_ping(A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z,A1)',
  event: 'error',
  id: 'bb84b48d-0095-4d40-8ea0-910c18a14e55',
  ...
 }

I also noticed that in the swipl session if I run the command in the swipl session I get no Matches:

?- db:get_all_ais_ping(_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,"Under w
ay using engine",_,_,_,_,_,_,_,Matches).
Matches = [].

?- findall(Match,
        db:get_all_ais_ping(_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,"Under way using engine",_,_,_,_,_,_,_,Match),
        [MatchesList]),
length(MatchesList, Count).
MatchesList = [],
Count = 0.

However when I run them in swipl normally I get the Matches I expect:

$ swipl
Welcome to SWI-Prolog (threaded, 64 bits, version 9.3.12)
SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software.
Please run ?- license. for legal details.

For online help and background, visit https://www.swi-prolog.org
For built-in help, use ?- help(Topic). or ?- apropos(Word).

?- ['ais_system.pl'].
true.

?- attach_db('facts.pl').
true.

?- db:get_all_ais_ping(_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,"Under way using engine",_,_,_,_,_,_,_,Matches).
Matches = [ais_ping(_, _, _, "Unknown", _, "99.0", _, "AIS", "Unknown", _, _, "101", "Unknown", 55.6534, _, 10.4751, 538005558, _, "Under way using engine", "0.0", "Undefined", "12.6", "06/07/2024 00:00:00", "Class A", "Undefined", _), ais_ping(_, _, _, "Unknown", _, _, _, "AIS", "Unknown", _, _, _, "Unknown", 55.94244, _, 11.866282, 219027804, _, "Under way using engine", _, "Undefined", "0.0", "06/07/2024 00:00:00", "Class A", "Undefined", _), ais_ping(_, _, _, "Unknown", _, "120.0", _, "AIS", "Unknown", _, _, "240", "Unknown", 55.001667, _, 8.341, 219573000, _, "Under way using engine", "0.0", "Undefined", "0.1", "06/07/2024 00:00:00", "Class A", "Undefined", _), ais_ping(_, _, _, "Unknown", _, "237.6", _, "AIS", "Unknown", _, _, "238", "Unknown", 55.614987, _, 15.477585, 423519100, _, "Under way using engine", "0.0", "Undefined", "11.1", "06/07/2024 00:00:00", "Class A", "Undefined", _), ais_ping(_, _, _, "Unknown", _, "219.2", _, "AIS", "Unknown", _, _, "219", "Unknown", 55.314413, _, 14.44054, 230351000, _, "Under way using engine", "0.0", "Undefined", "13.3", "06/07/2024 00:00:00", "Class A", "Undefined", _), ais_ping(_, _, _, "Unknown", _, "154.0", _, "AIS", "Unknown", _, _, "157", "Unknown", 55.322848, _, 13.156212, 209869000, _, "Under way using engine", "0.0", "Undefined", "13.5", "06/07/2024 00:00:00", "Class A", "Undefined", _), ais_ping(_, _, _, "Unknown", _, "56.9", _, "AIS", "Unknown", _, _, "56", "Unknown", 54.655555, _, 11.201777, 219472000, _, "Under way using engine", "0.0", "Undefined", "5.7", "06/07/2024 00:00:00", "Class A", "Undefined", _), ais_ping(_, _, _, "Unknown", _, "65.0", _, "AIS", "Unknown", _, _, "65", "Unknown", 55.45345, _, 14.8722, 566060000, _, "Under way using engine", "0.0", "Undefined", "18.1", "06/07/2024 00:00:00", "Class A", "Undefined", _), ais_ping(..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ...)|...].

?- findall(Match,
            db:get_all_ais_ping(_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,"Under way using engine",_,_,_,_,_,_,_,Match),
            [MatchesList]),
    length(MatchesList, Count).
MatchesList = [ais_ping(_, _, _, "Unknown", _, "99.0", _, "AIS", "Unknown", _, _, "101", "Unknown", 55.6534, _, 10.4751, 538005558, _, "Under way using engine", "0.0", "Undefined", "12.6", "06/07/2024 00:00:00", "Class A", "Undefined", _), ais_ping(_, _, _, "Unknown", _, _, _, "AIS", "Unknown", _, _, _, "Unknown", 55.94244, _, 11.866282, 219027804, _, "Under way using engine", _, "Undefined", "0.0", "06/07/2024 00:00:00", "Class A", "Undefined", _), ais_ping(_, _, _, "Unknown", _, "120.0", _, "AIS", "Unknown", _, _, "240", "Unknown", 55.001667, _, 8.341, 219573000, _, "Under way using engine", "0.0", "Undefined", "0.1", "06/07/2024 00:00:00", "Class A", "Undefined", _), ais_ping(_, _, _, "Unknown", _, "237.6", _, "AIS", "Unknown", _, _, "238", "Unknown", 55.614987, _, 15.477585, 423519100, _, "Under way using engine", "0.0", "Undefined", "11.1", "06/07/2024 00:00:00", "Class A", "Undefined", _), ais_ping(_, _, _, "Unknown", _, "219.2", _, "AIS", "Unknown", _, _, "219", "Unknown", 55.314413, _, 14.44054, 230351000, _, "Under way using engine", "0.0", "Undefined", "13.3", "06/07/2024 00:00:00", "Class A", "Undefined", _), ais_ping(_, _, _, "Unknown", _, "154.0", _, "AIS", "Unknown", _, _, "157", "Unknown", 55.322848, _, 13.156212, 209869000, _, "Under way using engine", "0.0", "Undefined", "13.5", "06/07/2024 00:00:00", "Class A", "Undefined", _), ais_ping(_, _, _, "Unknown", _, "56.9", _, "AIS", "Unknown", _, _, "56", "Unknown", 54.655555, _, 11.201777, 219472000, _, "Under way using engine", "0.0", "Undefined", "5.7", "06/07/2024 00:00:00", "Class A", "Undefined", _), ais_ping(_, _, _, "Unknown", _, "65.0", _, "AIS", "Unknown", _, _, "65", "Unknown", 55.45345, _, 14.8722, 566060000, _, "Under way using engine", "0.0", "Undefined", "18.1", "06/07/2024 00:00:00", "Class A", "Undefined", _), ais_ping(..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ..., ...)|...],
Count = 5826.

?-

@jan replied to my github issue about this with:

It says what it should say: with_mutex/2 is not allowed by the sandbox. That is correct as you can easily block things you should not be able or deadlock the system by misusing locks.

So, if you want to use this code, you’ll have to make sure it is safe and next declare the interface using the primitives of library(sandbox).

Could I ask for some tips about how I might go about making sure it is safe and declaring the interface using the privates of library(sandbox)?

Also, do you know why when I run the findall queries in the swipl session running the server I get no matches? Is this expected?

2

I have attempted to mark the predicates as safe to perform as described below and updated it on github.

I created a file which imports pengines and sandbox and defines the safe predicates and with_mutex as a safe meta_predicate like so:

safe_predicates.pl

:- module(safe_predicates, []).

:- use_module(library(pengines)).
:- use_module(library(sandbox)).

% Define safe predicates through the pengines_sandbox:safe_primitive/1 hook
:- multifile pengines_sandbox:safe_primitive/1.
:- multifile pengines_sandbox:safe_meta/2.

% Declare with_mutex/2 as a safe meta-predicate
pengines_sandbox:safe_meta(db:with_mutex(G,_), [G]).

% Declare other database predicates as safe
pengines_sandbox:safe_primitive(db:get_all_ais_ping(_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_)).
pengines_sandbox:safe_primitive(db:add_ais_ping(_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_)).
pengines_sandbox:safe_primitive(db:remove_ais_ping(_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_)).

And I imported it into my prolog server-side code like so:

:- use_module(library(pengines)).
:- use_module(library(http/http_server)).
:- use_module(library(sandbox)).
:- use_module(safe_predicates).  % Load sandbox permissions

% Load your Prolog files
:- consult('ais_system.pl').
attach_db('facts.pl').

% Start the HTTP server on port 5000
server(Port) :-
    http_server(http_dispatch, [port(Port)]),
    format('Pengines server running at http://localhost:~d/~n', [Port]).

:- initialization(server(5000)).

However I still get the same permission error on the client.

3

I don’t know where you got pengine_sandbox from. Sandbox declarations go into the module sandbox, e.g.

:- multifile sandbox:safe_primitive/1.

sandbox:safe_primitive(db:get_all_ais_ping(_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_)).
4

Thanks for getting back to me, I’ve altered my prolog safe_predicates.pl file as you suggested, replacing pengine_sandbox with sandbox so it now reads as follows:

:- module(safe_predicates, []).

:- use_module(library(pengines)).
:- use_module(library(sandbox)).

% Define safe predicates through the sandbox:safe_primitive/1 hook
:- multifile sandbox:safe_primitive/1.
:- multifile sandbox:safe_meta/2.

% Declare with_mutex/2 as a safe meta-predicate
sandbox:safe_meta(db:with_mutex(G,_), [G]).
sandbox:safe_meta(findall(G,_), [G]).

% Declare other database predicates as safe
sandbox:safe_primitive(db:get_all_ais_ping(_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_)).
sandbox:safe_primitive(db:add_ais_ping(_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_)).
sandbox:safe_primitive(db:remove_ais_ping(_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_)).

But I get the following error for my safe_predicates when I attempt to run swipl server.pl:

swipl server.pl
ERROR: /home/ash/Documents/pengines_server_client_example/pengines_server/safe_predicates.pl:15:
ERROR:    No permission to declare safe_goal `db:get_all_ais_ping(_26944,_26946,_26948,_26950,_26952,_26954,_26956,_26958,_26960,_26962,_26964,_26966,_26968,_26970,_26972,_26974,_26976,_26978,_26980,_26982,_26984,_26986,_26988,_26990,_26992)'
ERROR: /home/ash/Documents/pengines_server_client_example/pengines_server/safe_predicates.pl:16:
ERROR:    No permission to declare safe_goal `db:add_ais_ping(_28584,_28586,_28588,_28590,_28592,_28594,_28596,_28598,_28600,_28602,_28604,_28606,_28608,_28610,_28612,_28614,_28616,_28618,_28620,_28622,_28624,_28626,_28628,_28630)'
ERROR: /home/ash/Documents/pengines_server_client_example/pengines_server/safe_predicates.pl:17:
ERROR:    No permission to declare safe_goal `db:remove_ais_ping(_30222,_30224,_30226,_30228,_30230,_30232,_30234,_30236,_30238,_30240,_30242,_30244,_30246,_30248,_30250,_30252,_30254,_30256,_30258,_30260,_30262,_30264,_30266,_30268)'

I am unsure why I don’t have permission to declare safe_goal from the server-side code.

And still get the same permission error on my client-side script:

No permission to call sandboxed `with_mutex(_3490,_3492)'\n" +
    'Reachable from:\n' +
    '\t  db:get_all_ais_ping(A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z,A1)'
5

In don;t think you want this. The idea is to define verify that the API your library is offering is safe and define the API to be safe, not internal predicates.

To be able to declare something a safe primitive it must

  • be defined at the moment you make the declaration
  • be static (so it doesn’t change)
  • not be a meta-predicate.

I guess one of these is not true. Note that there is a lot of this stuff in various libraries to give you some inspiration.

6

Could it be because the predicates are defined in the ais_system.pl file but then when you attach the persistency database of dynamic predicates which contains a list of facts the functions are used from this scope?

e.g.

get _ais_ping is defined like so:

:- persistent
  db:ais_ping(_A, _B, _C, _Callsign, _CargoType, _Cog, _D, _DataSourceType, _Destination, _Draught, _Eta, _Heading, _Imo, _Latitude, _Length, _Longitude, _Mmsi, _Name, _NavigationalStatus, _Rot, _ShipType, _Sog, _Timestamp, _TypeOfMobile, _TypeOfPositionFixingDevice, _Width).

get_ais_ping(A, B, C, Callsign, CargoType, Cog, D, DataSourceType, Destination, Draught, Eta, Heading, Imo, Latitude, Length, Longitude, Mmsi, Name, NavigationalStatus, Rot, ShipType, Sog, Timestamp, TypeOfMobile, TypeOfPositionFixingDevice, Width) :-
  with_mutex(db, ais_ping(A, B, C, Callsign, CargoType, Cog, D, DataSourceType, Destination, Draught, Eta, Heading, Imo, Latitude, Length, Longitude, Mmsi, Name, NavigationalStatus, Rot, ShipType, Sog, Timestamp, TypeOfMobile, TypeOfPositionFixingDevice, Width)).

But then used like so:

['ais_system.pl'].
attach_db('facts.pl').
db:get_ais_ping(_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,"Under way using engine",_,_,_,_,_,_,_).

How do I mark such a predicate as safe?

I’ve tried both

sandbox:safe_primitive(db:get_ais_ping(_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_)).

and

sandbox:safe_primitive(get_ais_ping(_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_)).

And I’ve tried putting the line:

:- use_module(safe_predicates).  % Load sandbox permissions

before, after and between the following lines:

:- consult('ais_system.pl').
attach_db('facts.pl').

However it always results in No permission to declare safe_goal .

Are you aware of any examples of prolog code using persitency inside a pengines or sandbox?

7

I’ve resolved the issue with permissioning, by marking the predicate methods as safe_meta it permits me to call the methods from the client on the database.

:- module(safe_predicates, []).

:- use_module(library(pengines)).
:- use_module(library(sandbox)).

% Define safe predicates through the sandbox:safe_primitive/1 hook
:- multifile sandbox:safe_primitive/1.
:- multifile sandbox:safe_meta/2.

% Declare with_mutex/2 as a safe meta-predicate
sandbox:safe_meta(with_mutex(G,_), [G]).
sandbox:safe_meta(findall(G,_), [G]).

% Declare other database predicates as safe
sandbox:safe_meta(get_ais_ping(_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_)).
sandbox:safe_meta(get_all_ais_ping(_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_)).
sandbox:safe_meta(add_ais_ping(_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_)).
sandbox:safe_meta(remove_ais_ping(_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_)).

However, I now get the following error: procedure db:db’ does not exist` when I attempt to call any methods on the database, I presume there is an issue with the way I am attaching the database in the pengines server?

Query error: db:get_ais_ping(_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,"Under way using engine",_,_,_,_,_,_,_) {
  arg1: 'procedure',
  arg2: 'db:db',
  code: 'existence_error',
  data: "procedure `db:db' does not exist\n" +
    'Reachable from:\n' +
    '\t  with_mutex(db,ais_ping(_3412,_3414,_3416,_3418,_3420,_3422,_3424,_3426,_3428,_3430,_3432,_3434,_3436,_3438,_3440,_3442,_3444,_3446,_3448,_3450,_3452,_3454,_3456,_3458,_3460,_3462))\n' +
    '\t  db:get_ais_ping(A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z)',
  event: 'error',
  id: '84010363-a89c-49f9-a5e4-2c311562a4dd',
  pengine: Pengine {
    options: {
      server: 'http://localhost:5000/pengine',
      format: 'json',
      ask: 'db:get_ais_ping(_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,_,"Under way using engine",_,_,_,_,_,_,_)',
      oncreate: [Function: oncreate],
      onsuccess: [Function: onsuccess],
      onfailure: [Function: onfailure],
      onerror: [Function: onerror],
      ondestroy: [Function: ondestroy]
    },
    id: '84010363-a89c-49f9-a5e4-2c311562a4dd'
  }
}

Also note, I have to allow the with_mutuex as safe_meta or the db:get_ais_ping method is not marked as safe. I will be communicating with this pengine server from other server-side processes only and not exposing the api for clients to use directly so I’m not too worried about locking down the pengines server itself.

8

As before, the idea is to create a module, put everything you need in there, export the API and declare the API as safe. Attached is a small example. You can run this as

swipl db.pl
?- safe_goal(user(A,B)).
true.

db.pl (290 Bytes)

9

I took the db.pl you attached and attempted to use that in my server, I added the attach_db predicate so that I could attach a database and assert facts.

Like so:

:- module(db,
          [ user/2,
            attach_db/1 % +File  - attach's database file
          ]).
:- use_module(library(persistency)).

% define predicate rule to attach database file used to store persisted facts (data)
attach_db(File) :-
db_attach(File, []).

:- persistent
    user_db(name:string,
            age:nonneg).

user(Name, Age) :-
    with_mutex(db, user_db(Name, Age)).

:- multifile
    sandbox:safe_primitive/1.

sandbox:safe_meta(with_mutex(G,_), [G]). % I added this because when I attempted to assert a user from the client I got No permission to call sandboxed `with_mutex()`
sandbox:safe_primitive(db:user(_,_)).

I then loaded the module in the server and attempted to attach the database in the server like so:

:- use_module(library(pengines)).
:- use_module(library(http/http_server)).
:- use_module(library(sandbox)).

:- consult('db.pl').
attach_db('users.pl').

% Start the HTTP server on port 5000
server(Port) :-
    http_server(http_dispatch, [port(Port)]),
    format('Pengines server running at http://localhost:~d/~n', [Port]).

:- initialization(server(5000)).

I do get the following warning, so I suspect attach_db isn’t working as expected

Warning: /home/pengines_server_client_example/pengines_server/server.pl:7:
Warning:    Local definition of user:attach_db/1 overrides weak import from db

If I query from the client with user(A, B) it returns the false. result (showing the database isn’t attached, as I have 1 user fact asserted). I can make the not(user(A, B)) query and it returns true.. When I attempt to add a user db:assert_user_db("Bob", 51) I get the following error:

Query error: db:assert_user_db("Bob", 51) {
  arg1: 'procedure',
  arg2: "persistency:'$persistency'",
  code: 'existence_error',
  data: "procedure `persistency:'$persistency'' does not exist\n" +
    'Reachable from:\n' +
    "\t  with_mutex('$persistency',db_assert_sync(db:user_db(_3082,_3084)))\n" +
    '\t  persistency:db_assert(db:user_db(A,B))\n' +
    '\t  db:assert_user_db(A,B)',
  event: 'error',
  id: '16ec0913-5cb6-40e7-9459-beb4f6eaab61',
  ...
}
10

You must declare the API safe. Library persistency is not safe in the sense of the sandbox. It has side-effects (not allowed) and does things with locks and files, etc. Do not declare attaching safe as it is not: the user may attach /etc/passwd. That will result in a lot of syntax errors, but it will expose users of your system in the error messages.

Here is a version that allows asserting users. Just keep working this way: add something you’d like the pengine user to do to this module, export it and declare it safe.

db.pl (507 Bytes)

The sandbox denies every call Module:Goal if Goal is not exported from Module. It must, as the private predicates may provide access to information or allow modifying things the user should not be allowed to. For example, it could store privacy sensitive data in some predicate just holding facts. In itself, the predicate is safe (no side effects), but still Pengine users should not have unlimited access to data about other users.

1 Like
11

Ahh, it’s making sense to me now, thanks for all the help.

The last piece I was missing after your alterations was attaching the database inside the server predicate definition like so:

:- use_module(library(pengines)).
:- use_module(library(http/http_server)).
:- use_module(library(sandbox)).

:- consult('db.pl').

% Start the HTTP server on port 5000
server(Port) :-
    http_server(http_dispatch, [port(Port)]),
    attach_db("users.pl"), % attach database in pengines server
    format('Pengines server running at http://localhost:~d/~n', [Port]).

:- initialization(server(5000)).

I can then call the persitency predicates I have marked as safe as expected to get data from my database or assert new entries.