Wiki Discussion: SWI-Prolog in the browser using WASM

[EricGT]

@jan

Eventually it should be able to verify the core Prolog engine using ctest under Node.js.

Thanks.

I will use that as a destination and see what becomes of that journey.

[rla]

@jan

Eventually it should be able to verify the core Prolog engine using ctest under Node.js.

Running tests under Node.js brings decision whether we should build both browser and node.js variants of the WebAssembly version. The main difference is:

• Browser variant uses virtual filesystem and you need to inject files before reading them from Prolog;
• Node.js variant uses Node.js filesystem APIs and reads native (host) files.

I think so far we built both variants.

I would lean a bit to the side where we only have the browser variant because:

• If you can run Node.js then you can probably run SWI in easier ways than through WebAssembly;
• Having only the browser variant would bring less confusion.

Edit: running browser variant in Node.js would just use virtual filesystem.

Of course, the best case would be a configurable option whether you get the virtual filesystem or Node.js API-based IO.

[rla]

@jan

One issue is how to call JavaScript from Prolog, so we can manipulate the DOM. I see:

• emscripten_run_script() can call anything, but we get no return value and I’m unsure we can handle exceptions.
• EM_JS() allows calling a fixed JavaScript function. Possibly we can make this work by first placing the function (name) and arguments somewhere, use this and fetch the results from some other well known place?
• We could use js_yield(Request, Response)

There is also addFunction which gives a raw function pointer. This allows to register a JavaScript function, obtain a pointer, pass this pointer to WebAssembly and have WebAssembly/C call it as a function. While this is also likely too low level, it’s very likely an important building block to provide a working interface. Interacting with code — Emscripten 3.1.49-git (dev) documentation

I dislike emscripten_run_script because it leads to code passed around as strings which is incredibly unreadable.

js_yield is interesting but needs some way to pass around more structured data.

[jan]

@rla

I think so far we built both variants.

As is, we probably need that. The Node version is used for the Prolog built steps (rather than using native Prolog using the NATIVE_FRIEND mechanism) and is useful for running ctest. I guess eventually we want to make this available as an npm module where we might only include the browser version?

@rla

There is also addFunction which gives a raw function pointer.

I had seen that, but forgot about it That looks better With some auto-generated wrappers we should be able to just call a JavaScript function from Prolog.

[rla]

@j4n_bur53

If you completely asyncify your Prolog system integration with the browser,
things will look completely differently. At the moment the SWIPL WASM provides
some callback hell. This is useful for example to provide the buttons next and
abort, by some inverted programming style.

It’s difficult to understand what “complete asyncify” means in this case. It does not seem to be asyncify as meant by Emscripten? Asyncify — Emscripten 3.1.49-git (dev) documentation

From what I understand, Emscripten asyncify comes with heavy overhead in both code size and runtime performance.

I would not say that js_yield is a callback hell. Many apps just calling Prolog predicates would likely never have to deal with it and calling js_yield from Prolog itself appears completely sync, there is no need to wrap it in an engine or use Prolog callbacks.

[EricGT]

@Jan
@jamesnvc

FYI

One feature this Discourse site is lacking is the ability to run code in place in a post.

This might be possible now that the SWI-Prolog WASM (#wasm_wiki #wasm_demo) is making progress.

As we (Discourse admins) know we can not add anything we want to this Discourse site but we can add theme components as we did with

GitHub - jamesnvc/discourse-linkify-prolog-predicates: theme to auto linkify...

Based on

Discourse theme components now support Wasm 🎊

all of the stars seem to be aligning for this to happen.

[rla]

@jan

The WASM version clearly wants the HTML, JavaScript and JSON support, but not the HTTP server libraries.

How feasible it would be extracting JSON-related code from there? HTML is big one and probably troublesome since so many guides and documentation refers to HTML support as being from http package.

General JSON representation was mentioned here:

Blocking operations in the WASM version (JavaScript expertise needed)

Sounds good. For now my emphasis is on the low level. I definitely need support to get the high level right. I have had discussions with @ericzinda on a new general JSON representation for Prolog terms. The idea is to create something where the terms you normally like to transfer (lists, strings, numbers, objects) come pretty natural, but the representation has escapes that allow for faithful round trip of arbitrary Prolog terms. One thing at a time Auto yield is in the…

Was it about this representation? https://www.swi-prolog.org/pldoc/doc_for?object=term_to_json/3

[rla]

@jan

I guess eventually we want to make this available as an npm module where we might only include the browser version?

I think ideally it should be both and on Node.js you should be able to select either of them. The actual WebAssembly binary should be exactly the same. The difference is in the wrapping JavaScript provided by Emscripten. I’m working slowly on the package. My main goal so far was actually TypeScript support and automated build through docker to make current code easier to use by others. Improving API, term representation etc. was secondary.

[peter.ludemann]

Besides Javascript/Typescript, there’s also Dart (used to implement Flutter, I think), which runs in the browser (Chrome, Edge, Firefox, Safari). Some years ago, I attended a talk that claimed Dart had more parallelism than Javascript, but I don’t remember the details; and Javascript might have adopted some or all of Dart’s parallelism features in the interim.

Anyway, I found a few articles, in case they’re useful:

Dart asynchronous programming: Isolates and event loops

Concurrency in Dart

[jan]

@rla

Was it about this representation? term_to_json/3

That was the starting point. I have been discussing a new JSON format with @ericzinda at Consider adding an option to use a different JSON Format · Issue #4 · SWI-Prolog/packages-mqi · GitHub

@rla

How feasible it would be extracting JSON-related code from there? HTML is big one and probably troublesome since so many guides and documentation refers to HTML support as being from http package.

It is not very hard of course. The main issue is what you hint at: a lot of stuff will move and that has implications for the build process, users and documentation. We can minimize some of that by adding a library at the old location that loads and reexports the new library (and prints a deprecated warning). That is also what happened to the DCG support that started in the HTTP package as well. I’m still in doubt. The other option would be to simply minimize the HTTP package for the WASM version. That is far easier but doesn’t solve the long term issue.

One of the next steps is to be able to add the foreign libraries in the WASM version.

[jan]

@jan

@rla

Was it about this representation? term_to_json/3

That was the starting point. I have been discussing a new JSON format with @ericzinda at Consider adding an option to use a different JSON Format · Issue #4 · SWI-Prolog/packages-mqi · GitHub

I have now added support for @ericzinda’s proposal for JSON/Object representation to the prolog.js module as Prolog.toJSON() and Prolog.toProlog() that is connected to the WASM version. I’m also working towards a better high level interface to call Prolog from JavaScript. You can see it at work at https://dev.swi-prolog.org/wasm/test. The source is in the src/wasm directory as test.html and test.pl. Roughly, this adds:

• A Prolog.consult(url1, url2, ...).then(function) that downloads the urls from the server, puts them in /tmp and loads them into Prolog.
• Prolog.query(...) which has several signatures and returns an iterable object that represents the query. So, we can do call a goal from a string and get the bindings as an object that holds the JavaScript translation of the results, using the variables as keys. Variables starting with an _ are not included in this object.

  Prolog.with_frame(() =>
  { let n = 0;

    for(const r of Prolog.query("p(X)"))
    { println("stdout", r.X);
    }
  });

Or, provide input and do a once goal. The code below computes the sum of all numbers in a list in Prolog. It takes an object with input bindings that is translated to Prolog data. The output object does not include the input variables.

function sum_list(list)
{ return Prolog.with_frame(() =>
  { return Prolog.query("sum_list(List, Sum)", {List:list}).once().Sum;
  });
}

There are still several issues with this stuff.

• I’d rather get rid of the Prolog.with_frame() that scopes the Prolog terms references.
• Breaking out of the query due to an exception (not yet done) or a break from the for…of construct does not close it. AFAIK JavaScript has no destructor like C++ when an object gets out of scope.
• Errors are practically nowhere handled. I’m still doubting between return codes and JavaScript exceptions. The lack of destructors might make cleanup complicated
• It all looks pretty ugly and is not documented.

Still, feedback on the overall direction is welcome

[rla]

@jan

• I’d rather get rid of the Prolog.with_frame() that scopes the Prolog terms references.
• Breaking out of the query due to an exception (not yet done) or a break from the for…of construct does not close it. AFAIK JavaScript has no destructor like C++ when an object gets out of scope.
• Errors are practically nowhere handled. I’m still doubting between return codes and JavaScript exceptions. The lack of destructors might make cleanup complicated
• It all looks pretty ugly and is not documented.

In my opinion it looks pretty good. It is much more usable than it was a week ago. We could just require query to be closed and throw an error if a new query is opened without closing the previous one (another option is to queue them).

Your JavaScript is very good and pretty much self-documenting. Only the indentation is a bit odd but it matches the C code style in SWI

I would like to provide Typescript annotations. For JavaScript programmers they would be very helpful since modern IDE-s will display and autocomplete code by them. Things are moving very fast but I hope to find time to put into this.

[jan]

I have pushed an update to #wasm and updated the wiki page. There are two big developments:

• Call between Prolog and JavaScript and JavaScript and Prolog are getting close to their final version.
• Calls from Prolog to JavaScript can represent DOM elements as Prolog blobs. The DOM elements known to Prolog are subject to Prolog garbage collection

For example, we can how define a Prolog predicate to add a paragraph to the shell output window:

add_par(Text) :-
  Out  := document.getElementById("output"),
  More := document.getElementById("more"),
  Par  := document.createElement("p"),
  Par.textContent := Text,
  _ := Out.insertBefore(Par, More).

After which we can call

?- add_par("Hello world!").

We can also call the other way around nicely, for example writing all the results of p/1:

for(const r of Prolog.query("p(X)")) {
  console.log(r.X);
}

[jan]

It is indeed inspired by @nicos work on R/real. I don’t know whether this approach has a name. This can’t deal with functions and other complex syntax. We notably need a way to add event listeners.

@j4n_bur53

So basically you would write
add_par() in JavaScript and register it to Prolog.

There is no need to register anything. Just define the function and call _ := add_par("Hello world!") Of course you can define a wrapper predicate or we can add something that creates these wrappers.

[j4n_bur53]

(Nothing of Importance here)

[josderoo]

Aside from the above, in EYE we have to do

:- catch(use_module(library(sha)), _, true).
:- catch(use_module(library(pcre)), _, true).
:- catch(use_module(library(http/http_open)), _, true).
:- catch(use_module(library(semweb/rdf_turtle)), _, true).

because those libraries are not yet available in WASM.
This is not a burning issue, I just wondered if there is a plan to support those libraries in WASM?

[jan]

@j4n_bur53

The result is, so one is free to do whatever one wants with (:=)/2?!

Yes. What is wrong with that? You can do whatever JavaScript can do except for stuff that requires more complex syntax such as functions. In other words you can access methods, setters and getters and chain these using a.b.c… In fact, internally they are chained as a[b][c]… because . is a function in SWI-Prolog. goal_expansion on a.b.c… translates this to a[b][c]… to avoid the function evaluation. Next step :=/2 translates this into a Prolog term that expresses the actions that need to be done as a list of primitive actions. This is passed through ‘$js_call’/2 which is defined using EM_ASM_INT(), passing the raw Prolog terms to JavaScript. JavaScript calls Prolog.toJSON() (should that be Prolog.toObject()? as it is not really JSON). JavaScript executes the (now) array of actions resulting in a return value. This is handed to Prolog.toProlog() and made available as the result. If the arguments are sufficiently instantiated we could do most of the translation at compile time and simply pass the shared JavaScript object.

@josderoo

because those libraries are not yet available in WASM.
This is not a burning issue, I just wondered if there is a plan to support those libraries in WASM?

Most of them depend on foreign components. I have not yet looked into how easy or hard it is to support these. It seems emscripten has a notion of shared libraries, so it should be possible. Opening a URL as a stream as done by library(http/http_open) may be a real problem. A quick search suggests that emscripten has sockets, but they are a wrapper around websockets, not raw TCP/IP sockets. The obvious solution is to download in JavaScript, similar to Prolog.consult().then(), which downloads a file asynchronously using fetch(), compiles the file and runs the then(). This would not allow streaming parsing of e.g. Turtle input.

[jan]

@j4n_bur53

Wasn’t SWISH a client sandbox? Or mainly a server sandbox?

SWISH is just a server side sandbox. And yes, doing eval() on strings is dangerous if you do not control where the string comes from. Otherwise I don’t see the difference with <script> elements.

[jan]

@josderoo

:- catch(use_module(library(sha)), _, true).
:- catch(use_module(library(pcre)), _, true).
:- catch(use_module(library(http/http_open)), _, true).
:- catch(use_module(library(semweb/rdf_turtle)), _, true).

I pushed a starting point to deal with this. After reading all the warning on emscripten against using dynamic linking I decided for another route I had in mind to resolve this issue: allow adding the foreign extensions to the core system. This is now a new option -DSTATIC_EXTENSIONS to cmake which is enabled by default for Emscripten. This allows building fully static SWI-Prolog executables with extensions, which surely has value on its own

Currently includes some of the clib libraries, the sgml library and some of the http libraries.

An overall worry is the size of the whole thing

Pushed to #wasm_demo

[josderoo]

@josderoo

use_module(library(sha))

now works fine, thanks @jan