WASM Shell and non-standard file extensions

General
1

I want to use the WASM shell to load something via a HTTP URL.
Running into problems as follows. My server doesn’t allow retrieval of

*.pl files, because it thinks they are pearl files. And renaming everything
to *.p doesn’t work from within WASM shell:

?- ['https://www.rubycap.ch/hotlink/julian.p'].
ERROR: https://www.rubycap.ch/hotlink/julian.p:10:
ERROR:    url `'https://www.rubycap.ch/hotlink/julian/calendar/gregorian.p.pl'' does not exist

It works from within the ordinary SWI-Prolog top-level, I got
other errors, but basically the SWI-Prolog top-level did the following:

?- ['<path>/julian.p'].
.... some other errors ...
%  julian/calendar/gregorian.p compiled into julian_calendar_gregorian 0.14 sec, 17 clauses

P.S.: I used this use_module directive inside julian.p:

:- use_module('julian/calendar/gregorian.p', [gregorian/3, month_number/2]).

Any fix or workaround especially for the WASM Shell?

2

According to swipl-devel/wasm.pl at 7a546d6e9e3df6d15343894a71405d5ff1bd712d · SWI-Prolog/swipl-devel · GitHub, it adds .pl if the given extension is not a known Prolog extension. So, I think using ‘*.p’ as file name should work if you first add

user:prolog_file_type(p, prolog).

We could consider using the first extension returned by user:prolog_file_type(Ext, prolog). rather than hard coded .pl. That would allow adding the above using asserta/1 and get what you want?

Of course, you could also get yourself a better web server :slight_smile: Just using a GitHub repository is pretty simple and cool. The only twist seems to be that the raw content server of GitHub doesn’t add a modification stamp, which makes either caching or updating a bit hard :frowning:

5

It is, but it doesn’t work for HTTP[S] locations. It only works for the preloaded library.

Running code directly from the web is not hard to implement. The hook implemented in library(wasm) can also be used with the normal version (possibly needs some tweaks, haven’t checked). I don’t know how useful it is. To be efficient it surely needs caching and ways to let the user control the cache updates.

Many languages can do this, but it doesn’t seem widespread. There is probably a reason for that :slight_smile:

6

How do you measure widespread?
And what are you exactly refering to?

It is what the new import() can do, from JavaScript. But it doesn’t
seem to use a search path, it works like HTML links, just relative
to some base URL. Not sure which base URL it is, since I use

it in Dogelog player, available since release 1.0.2 for native libraries,
only with absolute URLs, and do some relative URLs computation before
hand with other means, and therefore have more control and design

options for the resultion. Search path should be also possible, when one
does the resolution on its own. Caching of the resolution don’t know
yet what the solution would be, if some search path would also arrive

in Dogelog player. I take resolution caching different from ensure loaded
caching, the later I see rather working with a resolved path. But it should
also work for HTTPS and not restricted to any preloaded stuff,

which would make it less dynamic:

JavaScript - import()
The import() call, commonly called dynamic import , is a function-like
expression that allows loading an ECMAScript module asynchronously and
dynamically into a potentially non-module environment.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/import

And since the synchronous require() from nodeJS could already load
JSON basically via the analogue of the Prolog ensure loaded route, i.e.
keeping track what is already loaded and avoiding duplicate loads, this

is now also in preparation for its asynchronous counterpart import():

import() - JSON
Importing JSON using ES modules was submitted as feature to TC39 in mid 2020, and is (at the time of this edit) in stage 3, which is the last stage before being accepted in to the spec (see https://github.com/tc39/proposal-json-modules for more details).
https://stackoverflow.com/a/39855320/17524790

So shuffling data to the client becomes more and more easier. My vision
is putting Prolog texts on par to JSON, end-users should give means to
dynamically shuffle data to the client in the form of Prolog texts, currently

I can shuffle data in the form of Prolog text to the client via import(), if I first
transpile it to JavaScript. It is then amenable to the import() statement.

7

In web context, loading code from the web is quite normal. In desktop context most environments have some pack/bundle/… mechanism that allows for installing it locally from some repository.

That is pretty much what the WASM version does, no? The load appears as synchronous to Prolog. Actually Prolog yields control back to JavaScript every 20ms and thus your browser remains responsive.

Ideally we would bring engines to the game. That would allow for multiple active threads of control in Prolog. As claimed before, this is possible but not trivial as, while engines do not need threads, the current implementation of engines is bootstrapped from threads. These aspects would need to be untangled.

9

It is pretty clear what it does. It does a JavaScript fetch() to get the content as a string. That should run asynchronously. After the fetch completes Prolog is asked to compile the string. That runs inside Prolog while Prolog yields every 20 ms, restarting after a zero timeout. The latter works fine as shown by one of the tests. So, from Prolog it all looks synchronous while the Prolog work is done in slices of about 20ms, allowing other tasks in the browser to proceed.

There is significant copying of the input string going on :frowning: It probably makes at least two copies. That can probably be avoided by linking the Prolog stream directly to the JavaScript ArrayBuffer rather than copying it to a Prolog string and opening that. All the copying works synchronously as well. Yet, copying a couple of MBs of memory should not be noticeable on modern hardware.

10

It only allows other tasks to run, which are put on the task
queue by the browser, like mouse evens and screen refresh.
But not some Prolog tasks. You have one task who waits

the fetch to be finish. What are the other Prolog tasks. So I
wondern how serious these claims are, namely this here:

But import() on the other hand can do multi-tasking concerning
loading. I didn’t try yet, and I also don’t know how to invoke
it via the functional form. Did some Australian guy demonstrate

it recently for Eyebrow? The script tag has various options:

For module scripts, if the async
attribute is present then the scripts and all their dependencies
will be executed in the defer queue, therefore they will get
fetched in parallel to parsing and evaluated as soon
as they are available.
https://developer.mozilla.org/en-US/docs/Web/HTML/Element/script

So basically without engines, one could possibly have,
these Prolog texts foo.p, bar.p and baz.p consultet in parallel.
Similar like Isabelle/HOL has a parallel loader:

/* Multi-Consult Can it be made Parallel */
?- ['foo.p', 'bar.p', 'baz.p'].

Loaded in parallel, when delegating to JavaScript import(). So
parallel loading is adressed by JavaScript without invoking ideas
of a multi-tasking environment or engine for the application.

Maybe some key idea is chunking. Not sure. It could be also that
different browser engines or browser engine versions offer different
solution. Probably only needed when you have slow but nevertheless

parallel internet connection. i.e. enough bandwidth to yeet multiple requests.

Edit 06.10.2022
Possibly this works also with fetch, if you combine multiple promisses
via Promise.all() or some such. But fetch() is only one half of consult, it
only fetches the text data. It doesn’t add the data to the knowledge base.

On the other hand a JavaScript code, when the import() does also some
evaluate, can possibly do both. Namely fetch the text data, and add the
data to the knowledge base. Bringing data to the client in a parallel fashion

ist most likely what import() allows. Didn’t try yet. Maybe use this syntax
for parallel import. A list wrapped into a singleton list. Or use some flag
to control it, since a default parallel behaviour could be too dangerous?

/* Parallel Variant Syntax of Multi-Consult ? */
?- [['foo.p', 'bar.p', 'baz.p']].

Of course engines or what Tau Prolog did with tasks and promises, could
also lead to a parallel load. Maybe can get away with a cheaper solution,
not bring tasks to Dogelog player, only juggle with import() ?

11

So what was rummaging in my brain. Why did I get repelled
by the idea of engines. I don’t know, or maybe I know?

Does SWI-Prolog have only this here, namely:

await(+Promise, -Result)
execution of await/2 completes when the Promise resolves and Result is unified with the value passed to the Promise.then() method.
https://www.swi-prolog.org/pldoc/doc_for?object=await/2

How about a new await_all/2, that takes a promise list?

What could be done with await_all/2? BTW: JavaScript has more
in stock than only await_all/2 which would correspond to

Promise.all(), there are like 1-2 other multi-waiters.

Edit 06.10.2022
And if your WASM SWI-Prolog also runs from within nodejs,
you can also test it outside a browser, even headless with
command line. And just a side remark, was testing nodejs

12.x it didn’t have import() but for example nodejs 18.x
does also have import(), just in case. Sorry for this imprecise
release data I am providing, maybe the internet provides a

better feature list somewhere.

12

As is, none of this is going to work in the current WASM version. As far as the WASM version of SWI-Prolog is concerned, the world is single threaded and synchronous. It only yields control to allow other tasks to proceed. If we get Prolog engines into the picture this can change.

As is, the WASM Wiki and discussion threads raised quite a few reactions. It seems the idea is appreciated. Now we have to see whether people want to use this stuff for anything serious. If so, we’ll see what they need. Actual asynchronous Prolog tasks? Get the size down? Something else?

We’ll see. For now my plan is to maintain the system at the current level, probably with minor (contributed) improvements. It is usable as it is for substantial tasks. It provides the normal core Prolog, constraints, tabling with well formed semantics, manipulation of the browser DOM, Browser event handling and much more.

If someone needs more, please contact me so we can make a plan how to make that happen. That does require making resources available.

13

Well, well, you use it already:

/**
 * Download one or more files concurrently and consult them.  Note that
 * the consult happens in arbitrary order.
 */
return Promise.all(args.map((url) => consult_one(url)));

https://github.com/SWI-Prolog/swipl-devel/blob/7f90f6d1fab3a85e33b3a2e8a230907077740909/src/wasm/prolog.js#L529

Or is this some dead code? Or some experimental code?
Code written by Raivo Laanemets and not by Jan Wielemaker?

But if you inspect the code, you will see consult_one() is based
on fetch(). My point is only, you could also do something based
on import(). Both fetch() and import() return a promise.

Its the same (engineless) game.

If you dig deeper, you will possibly find that he above concurrent
download and consult, will only interleave the download due to the
courtesy of how the browser interleaves fetch,

but the consult will probably not be interleaved. But if the consult
in comparison to download, does ony take a fraction of time, there
is still a benefit. You could draw a timing diagram.

With import() you get possibly more benefit, because the browser
will also interleave the evaluation, but I am not 100% sure. Have
to try it, when I am less busy.

14

This is an old version. The current version does this, loading the files in order as possibly needed by Prolog. The downside is that the downloads are also ordered. That could be improved by downloading all the files using a Promise.all() call and then passing all strings to Prolog.

  consult(...args)
  { return this.forEach("load_files(Files)", {Files:args});
  }

It seems there is something weird with Promise.all(). Trying the code below in the WASM REPL, one/1 runs fine. test/2 nicely creates the three promises, but trying to continue after the fetches it somehow corrupts the WASM stack and crashes :frowning: I have little clue why … So far both the fetch() promises and sleep implemented as a promise worked fine. What could be special about the .all() promise?

fetch_all(URLs, Strings) :-
   maplist(fetch_promise, URLs, Promises),
writeln(Promises),
   AllPromise := 'Promise'.all(#Promises),
writeln(AllPromise),
   await(AllPromise, Strings).

fetch_promise(URL, Promise) :-
   Promise := prolog.fetch(#URL, _{cache: 'no-cache'}, #text).

test(Strings) :-
   fetch_all([ 'index.html',
               'test.html'
             ], Strings).

one(S) :-
   fetch_promise('index.html', Promise),
   await(Promise, S).
15

My guess: You need to be careful to not put the promise on
the browser queue activating it and at the same time place it as
an argument to Promise.all(). The arguments array of Promise.all()

are supposed to be not yet activated. Promise.all() will activate them.

Yes or no?

Edit 07.10.2022
Also the old multi-consult used a then() to connect the fetch()
with the Prolog consult of the obtained download. And did the
Promise.all() on this combol. This gives a slightly different dynamics.

Means some Prolog consult will start earlier, and depending how
the browser implements the fetch(), the first Prolog consult might
start when there are still pending not yet completed downloads,

you might see a little bit more concurrency, or maybe less? Don’t
know. For example internet packets might arrive etc… at the network
connectors from the other downloads while the Prolog consult of

the first arrived download is going on. Also I am not sure how
browser really implement fetch. Some old rumors were, that they
use ServiceWorkers under the hood. So the browser is going

multi-threading for the fetches?

16

I think that is not the issue. Prolog.fetch() does this:

  fetch(url, opts, type)
  { return fetch(url, opts).then((response) => response[type]());
  }

That indeed starts the fetch() promise (if I understand all this correctly), but returns a fresh Promise, no? Anyway, if I replace the Prolog.fetch() with just calling fetch() the one/1 returns a Response instance and the other still crashes. Seems there is either something special with this promise or the “special” is it returning an array and something is wrong handling that. Unfortunately debugging WASM is no fun. The Firefox debugger has little more to say that it concerns a wasm call.

18

A Promise object has 3 states: pending, fulfilled, and rejected. Promise.all does not care in which state of promises you use in its arguments. There is no activated state. Promise.all itself returns a promise.

If something goes wrong, then it could be because something still happens despite one of the promises being rejected. A rejected promise (a remote endpoint did not exist in a fetch call for example) will NOT cancel other promises in Promise.all arguments array. There is no promise cancellation mechanism other than ad-hoc abort you have implement for every use case individually. Their results are just ignored and that should have no issues if they have no side effects.

20

I believe that import() provides no benefits over fetch() for loading Prolog code or anything besides JavaScript because concurrent/parallel parsing is implemented by browser specifically for JavaScript and happens behind the scenes. As far as I know it cannot be used for non-JavaScript resources at all.

To get around memory pressure of holding whole fetch result in memory, we can use Readable: ReadableStream - Web APIs | MDN (mozilla.org) and “pipe” response to Prolog for loading.

21

Thats not completely true. But my picture with const was wrong.
I wanted to say something else. You need to get out of your
main script to activate a promise. You can try yourself:

const promise = new Promise((resolve) =>
     {setTimeout(() => {console.log('Done'); resolve}, 1000});

while (true) {
}

Nothing will happen. If you uncomment while (True) { } the
promise gets activated. But where? During new or somewhere else?
The SWI-Prolog WASM has also some main script.

There can be harmfull side effects to a promise or dependencies
that a promise has, occuring and maybe changing until you have
left the main script. That could be also a problem, not enough

encapsulation of the promise. So the engineless approach
has some pitfalls. But maybe “activate” is not right word here.
Need to check JavaScript documentation, this is also explained.

I used it with transpiled Prolog, as an alternative to quick load format.
Works fine, but I didn’t do yet some measurements about parallel loading.
Will do when I am less busy.

22

The reason proved simpler: a missing break that cause the translation of arrays and some other objects to pass a null term reference to Prolog, which Prolog doesn’t like much :frowning: We can now indeed concurrently fetch multiple URLs to strings in Prolog using this code. Works on #wasm_demo :slight_smile:

fetch_all(URLs, Strings) :-
   maplist(fetch_promise, URLs, Promises),
   AllPromise := 'Promise'.all(Promises),
   await(AllPromise, Strings).

fetch_promise(URL, Promise) :-
   Promise := prolog.fetch(#URL, _{cache: 'no-cache'}, #text).

test(Strings) :-
   fetch_all([ 'index.html',
               'test.html'
             ], Strings).

To test promise results I’ve added a function promise_any(data) to shell.html, so we can do e.g.

?- A := promise_any(42), await(A, V).
A = <js_Promise>(1),
V = 42.
1 Like
23

Are you saying the promise is not pending? It is right after instantiation with new. You can put console.log(promise) right before the infinite loop.

With that while loop you just block the browser event loop. setTimeout will have no chance to run and the promise will not be fulfilled.

24

I went to debug the code and got no errors, you fixed it already :smiley:

1 Like
25

Thanks. Added some more return code checking which would have avoided this … More should follow. Or, should we throw JavaScript exceptions?