Starting subprocesses, reading their output? (for a build system)

General
1

Hi,
I’m new to Prolog, and just started reading Bratko’s book. On another project I’ve been working on a custom build system for compiling a C++ project. Currently that uses Python. It seems like Prolog might help for that kind of thing, but I’d need some replacement for Python’s imperative functions that do the following. Does SWI Prolog have them?

  1. Start a subprocess and read it’s stdout and stderr, as it comes back
  2. Split a thread or somehow allow parallel calls to #1. The python code throttles the amount of parallelism there with a semaphore.
  3. Parse json (clang-scan-deps reports dependencies between C++ files this way)

I’m so new to Prolog I’m not sure it will help me here, but I wanted to play around with it as I read the book. I can see I need to re-wire my brain a bit. :slight_smile: There are number of things that make a build system confusing. You specify a lot of rules for how to handle different files, or different build conditions. Then you need rules for how to combine the rules. There’s also the DAG (directed acyclic graph) and dependency analysis - figuring out which files need to be compiled (or “precompiled” now, if using C++20 modules) before others.

Anyway, as I was writing that build-system code I got the feeling I was creating a poor man’s logical inference engine, so now I’m here.

Rob

2

Welcome. If there is nothing what does what you want then Prolog is probably a good tool for this :slight_smile:

That is done using process_create/3. This is a very flexible, but also somewhat confusing predicate. There are a lot of examples in library(git) (that is, git.pl in the library directory).

There are several ways to do that. The simplest is probably using thread_create/3. It is however also possible to make process_create/3 return after creating the process and monitor I/O and status change of the created processes. That scales better, but is harder to implement. Prolog threads are pretty flexible. There are a lot of ways to make them communicate and synchronize. Possibly thread_wait/2 is an option here. Using message queues (thread_send_message/2 and many related predicates) is also an option.

see json_read/2

Note that if you already have the Python stuff to control processes, handle the JSON, etc., you could also try the new Janus package to reuse this. The package comes with the recent development versions and allows both embedding Python in Prolog and the other way around.

Success :slight_smile:

1 Like
3

Ah, another build system amateur ^^

I have been very interested in build system myself, so I’ll try to reference what already exists in the swi-prolog world about build systems.

First, there is the biomake pack that implement a make clone to execute makefiles. It also brings improvements by allowing using prolog inside makefiles.
I believe the use-case is more about data processing than in program compilation.

I would also like to self-promote a very simple pack I wrote called ninja which helps generating a ninja build system file. The idea is to provide basic DCG goals to help produce rules and build edges in the ninja format.
Once the ninja build file is generated, you use ninja to execute it.

So if your main interest in build systems is to work on the generation of the DAG in a specific domain using expert knowledge, for example the compilation of a C++ program with all the knowledge of how to invoke compilers, e.g. a CMake clone, you could skip all the DAG parallel execution part by using my pack.

On the other hand, if your main interest is to efficiently execute the DAG in parallel using prolog, e.g. a ninja or make clone, you will probably need a graph manipulation library. For example, I’m sure that the ugraph library will be of use to you, especially the top_sort/2 predicate ^^

2 Likes
4

Thanks! That gives me much to chew on for now. I’ll probably be back with newbie questions.

5

Thanks!

Yes, haha. I was using CMake, but rolled my own about 8 months ago. It probably does 0.2% of what CMake does; but it was been doing 100% of what I needed and was easier to understand. Now I want to add support for C++20 modules which requires some new dependency analysis. I may end up just going back to CMake, but since I want to learn Prolog anyway, I’m experimenting.

6

I just wanted to give you an example of how you would compile a c++ program with my ninja pack:

Given the following c++ files hello.h, hello.cpp and main.cpp:

c++ files 
// hello.h
void hello();

// hello.cpp
#include <iostream>

void hello()
{
  std::cout << "Hello World!" << std::endl;
}

// main.cpp
#include "hello.h"

int main(int argc, char** argv)
{
  hello();
}

You could generate a build.ninja file with the following prolog program and even header dependencies work using the depfile feature of ninja:

% build.pl
:- use_module(library(dcg/basics)).
:- use_module(library(dcg/high_order)).
:- use_module(library(ninja)).

% this is a top level directive so that calling `swipl build.pl` will
% result in directly calling the predicate `graph/0`
:- initialization(write_build(graph), main).

% first, we declare all filenames used in our project
stem(main).
stem(hello).

stem(Stem) -->
   { stem(Stem) },
   atom(Stem).
source(Stem) -->
   stem(Stem), ".cpp".
header(Stem) -->
   stem(Stem), ".h".
object(Stem) -->
   stem(Stem), ".o".
executable(Exe) -->
   atom(Exe).
depfile(Stem) -->
   stem(Stem), ".d".

% then we specify our build graph with ninja `rule` and `build` instructions
graph -->
   rule(compile, "g++ -MD -MF $depfile -c -o $out $in"),
   rule(link, "g++ -o $out $in"),
   target(main, [main, hello]).
target(Exe, Stems) -->
   sequence(compile, Stems),
   build([executable(Exe)], link, [sequence(object, " ", Stems)]).
compile(Stem) -->
   build([object(Stem)], compile, [source(Stem)], [variables([depfile-depfile(Stem)])]).

After defining build.pl, you just need to generate the build.ninja file once with:

$ swipl build.pl

Now, you should have a build.ninja file containing the following graph:

build.ninja 
rule generate
  command = swipl $in
  generator = 1
build build.ninja: generate /home/kwon-young/prog/music/various/build.pl | /home/kwon-young/prog/music/various/build.pl /usr/lib64/swipl-9.0.4/library/dcg/basics.pl /usr/lib64/swipl-9.0.4/library/dcg/high_order.pl /home/kwon-young/.local/share/swi-prolog/pack/ninja/prolog/ninja.pl
rule compile
  command = g++ -MD -MF $depfile -c -o $out $in
rule link
  command = g++ -o $out $in
build main.o: compile main.cpp
  depfile = main.d
build hello.o: compile hello.cpp
  depfile = hello.d
build main: link main.o hello.o

Now calling ninja will compile your program:

$ ninja          
[3/3] g++ -o main main.o hello.o
$ ./main
Hello World!

Changes in any sources, headers or build.pl will be picked up by ninja and will correctly rebuild your program.