Help getting started with multithreaded dynamic databases

Help!
1

I’m new to prolog. I think I’ve got the basics down but I’d like a bit
of help to get me started on what I’d like to accomplish.

I’d like a dynamic database over which I execute some goal query
every second. The goal itself has side effects and may assert and
retract over the database. I’d like the goal query to be running as
one process and in another process modify the database.

As a simple example imagine 3 tasks a, b, and c. c requires b to
complete, b requires a, and a has no prerequisites. I’d like to
accomplish task c. Notionally then:

:- dynamic
       aDone/0,
       bDone/0,
       cDone/0.

doC :- bDone, writeln("c is done!"), asserta(cDone), !.   % do c if b is done
doC :- aDone, writeln("b is done"), asserta(bDone), !.   % do c if b is done
doC :- writeln("a is done"), asserta(aDone), !.                % do a

retractall :-                                                                       % create fresh database
    retractall(aDone), retractall(bDone), retractall(cDone).

start :-
    retractall,   % start with no task complete
    repeat,
    doC,
    sleep(1.0),  writeln("done iteration..."),   % poll every 1s 
    fail.

Now, in a separate process, I’d like to intervene on the database,
for example as retract(bDone), retract(cDone), and have doC
automatically redo the b task in order to accomplish c.

I have no experience with threading; is that what I need here, or do I
need pengines? Could someone offer some suggested starting code?

Any help is much appreciated!

1 Like
2

As much as I want to give an answer, the one you really want to wait to see an answer from is Jan W.

Problem is it is the middle of the night there. :frowning_face:

I do recall a similar question

Solving two consecutive dependent goals from command line

but if you read into it you might find that either it is similar to what you seek, different or will give you some idea that Prolog has some ways of doing things that are not quite obvious.

I really like the work you do; nice seeing you here.

3

Thanks for the welcome Eric! I’ll take a look at that thread. While waiting for the expert, I’d love your thoughts…

1 Like
4

My thoughts can be dangerous and lately have been more guess than actually knowing, but they were good enough for some.

Your need for threading has wondering why? I am not saying that is wrong, but I am wondering if Prolog’s ability to backtrack will not do what you need. On the other hand concurrent_forall/2 (example) might fill the bill.

You also note the need to assert/1 and retract/1 which has me thinking and will also have many hear asking why? It is not that it can not be used, but it is one of those flags that suggest there is a concept you might not know. The concepts that pop to mind in this regards are the library(persistency) as noted in the other question, tabling, or ideas similar to using SQL database. ( Wiki Discussion: Prolog in the mind set of SQL).

Also the need to pass the data along to consecutive processes has me again wondering if Prolog’s backtracking ability and tabling might work.

As I have never used tabling, I can only make reference to it. Also note that Jan W. added tabling in the last year or so, so if you do use it, don’t take for granted all of the bugs are even known.

As far the library(persistence), that I have practical knowlege on but only in the cases I needed it for. There is an option db_sync/1 with option update

As reload , but use incremental loading if possible. This allows for two processes to examine the same database file, where one writes the database and the other periodically calls db_sync(update) to follow the modified data.

but I never was able to get it to work.

Also for library(persistency) the automatic setting of locks were changed a few months ago for the better, but locks are still needed for anything more than simple updates.

So these are my thoughts, but as often happens, Jan W. will answer with much more detail or answer with something I had no to little knowledge of that adds more to my experience.

HTH

EDIT

Check the packages for added code. These are one of the most often overlooked parts of Prolog and loaded with hidden gems.

EDIT

Oops, almost forgot this.

See: TerminusDB - The splash page only hints at what is possible, however one of the primary creators, Gavin, is on this site or on their forum

5

Thanks for the thoughts Eric, it’s likely that I’m misconceptualizing the proper solution. Here’s an expansion of my use case.

Ultimately, I’d like to use prolog to control the behaviour of a robot in a dynamic environment, but I’m starting with a toy model of the environment described using a dynamic database (hence my perceived need for asserting and retracting). A top-level controller runs every second and determines what task the robot should be doing at each time slice. The permitted logic for task switching is determined by a bunch of predicates (e.g. my original example of “a then b then c” task structure).

To demo the viability of this I’d like the controller running as a background process and polling the database every second. Then, in some other prolog session, I’d dynamically alter the environment and watch the controller adapt the robot’s task switching appropriately to accomplish its goal.

1 Like
6

Would RoboCup be similar?

7

Are we talking processes or threads? Given the robotics domain I’d guess threads. Surely you do not need Pengines. They might be a good idea in some designs, but it is not my first association.

There is no problem with multiple threads reading and writing the dynamic database. The elementary actions are thread-safe. One thing to keep in mind is that any query on a predicate keeps a consistent view on the state of the predicate as it finds it when the query is started: facts added or removed after the query started do not affect that query, including backtracking on that query. The single threaded example of the logical update view is below. Although the first solution removes a(2), this is still reported as a second solution. This consistency is also ensured if some other thread adds or removes a/1 clauses.

:- dynamic a/1.
a(1).
a(2).

run(X) :- a(X), (X == 1 -> retract(a(2)) ; true).

If multiple primitive actions are requires to go from one state to another or you want to reason about the state of the world while it is changing you should probably look into transaction/1 and snapshot/1.

As a rough design I would assume one or more threads that listen to sensors. How depends a lot on where the data comes from. It might be a Prolog thread calling some C function to get the data from the environment or there could be an internal network, so you can have these Prolog threads reading from a socket. The sensor threads may use transaction/1 to make the world view jump atomically between consistent states.

The reasoning threads will probably also need to use transaction/1 or snapshot/1 to get a consistent view.

You could also consider using a Redis database as shared blackboard between various processes. See Redis – a SWI-Prolog client for redis

In both cases use the latest development version as transactions as well as the Redis interface are new.

1 Like
8

Thanks for the suggestions Jan.

My initial goals are modest; for now, I’d like the simplest solution that would allow me to have one thread running the control loop and another where I can modify the shared environment database. The purpose is to convince myself (and others in the company) that prolog will be a flexible and intuitive way to orchestrate task control. Later, if we decide to proceed with a prolog solution, I’ll contract software help to develop a real design/implementation along the lines you suggest as that is beyond my expertise.

I can see that the logical update view may complicate things but is there a simple way of meeting my initial goals that a prolog newbie could realize? Are there tutorials on multithreading a shared database? I have no experience with concurrent programming (in any language) so I’m starting from scratch.

I really appreciate any advice!

9

Yes, that’s a good example Eric. However, I’m more focused on settings where the robot has to reason about causality and the complexities of a more general human-populated environment.

1 Like
10

If you search this group for “transaction”, you’ll find the initial discussion about transaction/1, snapshot/1, etc. and when transaction control is and isn’t needed.

FWIW, when transaction control was added on top of Google Bigtable (which has row-level atomicity), the overhead was something in the order of 10x (this might have been improved since) – there’s a reason why noSQL databases are popular (and SWI-prolog’s assert/1, retract/1 can be thought of as a kind of noSQL database).

1 Like
11

It is fairly clear that this is a multithreaded application. What isn’t so clear to me is when and why a dynamic database is the preferred mechanism for thread communication and synchronization. SWIP offers alternatives (see
Multithreaded Applications in the reference manual) which may be better suited depending on the details.

Maybe just my view/experience, but “self-modifying programs” are usually on the bottom of my list when looking for solutions, but that’s another discussion.

12

Well, your code works :slight_smile: The question is how the changing world gets into the picture. There are many ways to do that which range from simply reading updates to the world as part of the reasoning loop to concurrent updates. Which is to be used depends on things like update frequency, need for real time responses, relations between updates, affect of partial updates to the reasoning, etc. Just about anything that is used to get these problems solved is around in current SWI-Prolog.

The overhead of SWI-Prolog transactions is about a meta-call. Transactions in a single shared memory model are easy and cheap when compared to distributed systems. Prolog already requires a notion of database generations to deal with the ISO demand for the logical update view. Transactions are little more than some fiddling with the generation logic.

Theoretically asserting and retracting facts is of course a self-modifying program. I wouldn’t really call it that way though. As long as only dynamic facts are modified a Prolog program is pretty close to any other program that manages data. Yes, you can also modify rules :slight_smile: That gets you in a quite different world.

13

There is some difference between relational databases and Prolog in this scenario, from the point of view of the developer (ok, from my point of view :slight_smile: I know I am just a data point).

On the one hand, there is the transactions thing; SWI-Prolog has all the tools but the mechanics of using them is slightly different. In other words, you need to pay attention.

Then, there is the “consistent state” issue. The mindset of designing a relational database is that you put effort in figuring out the constraints on the data, and codifying them early, when you define the schema. This is again perfectly doable with Prolog, but you need to figure out some mechanism to implement the constraints.

14

When I read what you quoted by Jan

As long as only dynamic facts are modified a Prolog program is pretty close to any other program that manages data.

I was not thinking SQL databases but the heap.

The other thing I was thinking was that the dynamic facts can represent state and that instead of passing the state around as two variables for each predicate, e.g. a planner system (FYI I am Guy Coder on StackOverflow), this is taking the option to use dynamic facts for the state.

15

I guess this is all true. Dynamic predicates consisting of facts are of course quite close to relational database models. Probably quite a bit of the design rationale for RDBMS can be reused. At the same time there are differences. For one thing, Prolog values can be arbitrary terms, including variables. In some cases variables may play a similar role than NULL, but they are still quite different. Indexing and query planning works quite different as well.

Constraints can (I think) quite easily be expressed as logic programs. The transaction/1 and transaction/3 provide different ways to ensure they remain respected. This is a unique feature of SWI-Prolog.

1 Like
17

Yes, I was just hoping to summarize very briefly what would be different for someone with experience with a relational database.

What I meant with constraints is that you can write them, forget about them, then be reminded when you try to manipulate the database into an inconsistent state. This is definitely doable with Prolog but you need to implement a DML layer that intercepts any data manipulation and decides which constraints must be checked. Isn’t that so?

18

Yes. Given a goal is_consistent/0 though the thread-safe way is one of the update predicates below. The semantics is the same. The first version is good if Goal is cheap while the second is interesting if Goal is expensive and inconsistencies are rare as you have to retry on failure. In both cases, is_consistent/0 is protected by a mutex and thus serialized.

update_1(Goal) :-
    with_mutex(db, transaction((Goal, is_consistent))).

update_2(Goal) :-
    transaction(Goal, is_consistent, db).

is_consistent/0 can be a simple constraint, you can reason about Goal to generate a dedicated constraint that only checks things that may be violated and is_consistent/0 can ask the transaction in which it appears about the pending changes to check in a smarter way.

You don’t get this stuff for free :frowning: You can make it fit your application perfectly though. We also see lots of projects moving to NoSQL databases that also give you little for free.

1 Like