Work on SWISH

SWI-Prolog web site and services
1

I have revived an old project to make SWISH more robust using horizontal scaling. That is still work in progress. Steps:

  • Use Redis to allow SWISH instances to realize a shared view on the world and send each other messages. That was done some years ago, but taking the project to the next level stalled.
  • Create a High availability with Redis Sentinel network and extend SWI-Prolog’s Redis interface to deal to TLS and Redis Sentinels.
  • Migrate SWISH to use this infrastructure and run SWISH in a Docker container for easier deployment. That is now done with the primary public SWISH instance.
  • Launched the instance swish2.swi-prolog.org. You can use it already, but be aware that it is restarted much more frequently than swish.swi-prolog.org during development. Note that the instances share saved documents, profiles, messages, etc.

That is where we are right now. This implies swish2.swi-prolog.org is available as backup and a possible fatal failure of one of the servers no longer results in data loss.
Next steps:

  • Allow a SWISH client to switch to a different backend. There is already a menu item File/Backends … You can already manually select a different backend. Unfortunately only a small part of the functionality actually supports this now. Eventually, the system should automatically choose the best backend and switch the current backend fails or maybe even if it becomes too crowded.
  • Have a failover mechanism that ensures we can always get the initial page.
  • Reduce load on Redis. Some caching or smarter data structures are required to reduce the number of requests.

Status

Main system is running again. There are some stability issues. Initially the health checking malfunctioned. That is hopefully fixed.

6 Likes
2

Status update

SWISH seems to be running happily again. It now uses a Redis “high availability cluster” for storing most of its dynamic data. In addition, each server maintains the gitty file store, a GIT-inspired store for versioned files. Files are automatically replicated over SWISH servers connected to the Redis cluster.

In addition to https://swish.swi-prolog.org, there is now https://swish2.swi-prolog.org with runs on a different location. This second server now runs two SWISH instances behind an nginx load balancer.

I don’t have time to continue this for now. The overall plan would be to allow a SWISH client to switch backends, i.e., you load the initial page from any of the backends through a load balancer and the downloaded client will then use the most appropriate backend and optionally switches if the current backend fails.

If anyone wishes to pick this up, please contact me. The main expertise required now is on the JavaScript side.

In the meanwhile, please start using https://swish2.swi-prolog.org, so we can test that the load balancing as well as multiple backends work as intended. Please report issues.

2 Likes