Windows MIDI interface

The idea was, assuming this script is called midi_input.py, to load this into Prolog. E.g., you use py_add_lib_dir/1 to add the directory holding this script to the search path and than you can load the script using

 ?- py_call(midi_input:'__file__', F).

That might actually work, but the py_call/2 will not return because the script never finishes. Now, I don’t know how this is handled in Python. Possibly you can remove the while-loop, which will cause the script to return and it will just keep working. Alternatively, if you don’t use Python for anything else from Prolog, you can do

?- thread_create(py_call(midi_input:'__file__', F), TID).

That should give control back to Prolog and I think will nicely print the messages to the Prolog console. As the thread holds the Python GIL, you can’t do make other calls to Python from Prolog.

Handling asynchronous input is something that should work with Janus, but there could be problems. Now I don’t have physical midi devices, so I guess I cannot test this? Is anyone aware of another Python example that I can use without a MIDI device and which is likely to require a similar solution?

You don’t need physical MIDI devices. You can install MIDI Tools | Mountain Utilities and use the virtual keyboard to send MIDI messages.

I’m a Linux guy, so I quickly found vmpk, Virtual MIDI Piano Keyboard that can do the same. Turns out that mido.open_input() creates a Python thread and this calls Prolog without a Prolog engine attached to this thread, so … crash. I added stuff to avoid that. Now I can load the script in Prolog and get the messages, but only running the script in the foreground thread and easily creating a deadlock. Seems some more work needs to be done to make Python threads and Prolog properly cooperate. I’ll have a look.

Ok. Progress. For this to work you need the latest version of SWI-Prolog, which you can download from Download daily builds for Windows tomorrow (should be dated Oct 3).

It turns out that mido.open_input() creates a Python thread and there were several issues dealing with Python threads. At least what is needed for this demo is now resolved :slight_smile: Now, I use this Python code, saved as input.py:

import janus_swi as janus
import mido

engine = None

def on_message(message):  # callback function
    global engine
    if not engine:
        engine = janus.attach_engine()
        print(f"Created Prolog engine {engine}")
    janus.once("midi_message(Msg)",
               {'Msg':{'type':message.type,
                       'time':message.time,
                       'channel':message.channel,
                       'note':message.note,
                       'velocity':message.velocity
                       }})

def open(port):
    mido.open_input(port, callback=on_message)

And this Prolog code, saved as inputs.pl:

:- use_module(library(janus)).
:- py_add_lib_dir(.).

go :-
    py_call(input:open('MIDI Out:out 129:0')).

:- public
       midi_message/1.

midi_message(Msg) :-
    writeln(Msg).

Now, we load input.pl into Prolog and call

?- go.

This calls input:open() with the MIDI port. This load input.py and runs the open() function that opens the MIDI input and creates a Python thread that calls on_message on each callback. First thing we do is to associate a Prolog engine with this thread. We can also omit that, but then each callback will create and destroy a Prolog engine, which could be a bit slow.

Next, we can use the keyboard and we see

10 ?- go.
true.

11 ?- Created Prolog engine 3
py{channel:0,note:44,time:0,type:note_on,velocity:100}
py{channel:0,note:44,time:0,type:note_off,velocity:0}
py{channel:0,note:36,time:0,type:note_on,velocity:100}
py{channel:0,note:36,time:0,type:note_off,velocity:0}
py{channel:0,note:46,time:0,type:note_on,velocity:100}
py{channel:0,note:46,time:0,type:note_off,velocity:0}
py{channel:0,note:40,time:0,type:note_on,velocity:100}
py{channel:0,note:40,time:0,type:note_off,velocity:0}
py{channel:0,note:55,time:0,type:note_on,velocity:100}
py{channel:0,note:55,time:0,type:note_off,velocity:0}

Note that I create a Python dict from the message with the information I want. This gets mapped to a Prolog dict rather than a Prolog reference to the Python object. You can also extract the information at the Prolog site, but this is probably simpler and faster.

Note that the callbacks happen in a different Prolog thread (thread 3). You can probably make a second open() call, which will make another thread listening to a different MIDI port. You now need to link the stuff together. There are many ways to do that. Assert the messages in the database, using e.g.

midi_message(py{channel:Channel,note:Note,time:Time,type:Type,velocity:Velocity}) :-
     assertz(midi_message(Channel, Note, Time, Type, Velocity)).

Then use thread_wait/2 to wait for events and act on them? That is just one of the many ways. Curious to hear your story :slight_smile:

2 Likes

If he’s willing to tell it of course…maybe he just needed technical support :blush: and he found an excellent one for sure

Wow ! This is a work of art ! :slight_smile: Many thanks for this code because this is so advanced that I would never have been able to design let alone program this myself.

I’ll test this approach on my Windows PC as soon as I’ll be home (late) tonight. I downloaded the new SWI prolog version.

Usually I use several MIDI ports simultaneously. Will it cause a lot of overhead?

I’ll use Prolog’s pattern matching in order to filter MIDI messages, just to select those that are of interest at any time.

I suppose I first need to uninstall the (wrong) janus python package I installed a few days ago because all the py_… predicates I tried crash the prolog engine. I found a Report.wer file which seems to contain some system data when one of the crashes happened. Could this be of any help for you?

1 Like

Note that I didn’t test on Windows. I think it should work, but one never knows :frowning:

I don’t know. That mainly depends on how well Python mido handles this. I don’t think Python is well know for its real time multi-threaded behavior and neither is Windows. Hopefully it is good enough. Quite likely you’ll get way better throughput connecting Prolog directly to the C++ library used by mido.

Probably not. I wonder why the other janus package interferes, but I probably can sort that out on Linux just as well.

Warning: I forgot to commit a fix for the main Prolog engine that is required for the latest Janus version. Without, it simply crashes. Pushed and manually triggered the daily build. So, you need swipl-w64-2023-10-04.exe as uploaded Wed Oct 4 08:55:56 2023, sized 13,586,124 bytes.

I replaced your MIDI port by mine in both scripts, but I get no MIDI messages. The Python script is called and the MIDI port is opened. However, the on_message callback function is never called, because I inserted a print(“on_message called”) as the first instruction of that function.
To verify my MIDI circuitry I tested the old python script I published 2 weeks ago. It still works and with the same port.

?- go.
port opened with callback
true.

Here is the slightly adapted script I use :

import janus_swi as janus
import mido

engine = None

def on_message(message):  # callback function
    print("on_message called")	# added
    global engine
    if not engine:
        engine = janus.attach_engine()
        printf("Created Prolog engine {engine}")
    janus.once("midi_message(Msg)",
               {'Msg':{'type':message.type,
                       'time':message.time,
                       'channel':message.channel,
                       'note':message.note,
                       'velocity':message.velocity
                       }})

def open(port):
    mido.open_input('loopMIDI Port 0', callback=on_message)	# used own MIDI port
    print("port opened with callback")	# added

Strange. Installed

on Windows (11). This allows using the keyboard on the MIDI device “Microsoft MIDI Mapper”, but neither MIDI tools, nor mido see any MIDI input channels. Any clue how to work around that?

If you tried this using swipl-win.exe (the gui version), you may try using swipl.exe (the commandline version) because redirecting I/O Python I/O may be a problem.

I use the free software loopMIDI | Tobias Erichsen to create virtual MIDIports.

Thanks. Using that, my original worked (after changing the port). Of course, the port should be mentioned only once and

should have been

def open(port):
    mido.open_input(port, callback=on_message)

It turns out that, when using swipl-win.exe, the print() in Python after opening the port causes the thing to lock up. This works fine in swipl.exe. There seems to be an issue with capturing and forwarding the console output from Python to the Prolog output that is used in swipl-win.exe to make the output visible. swipl.exe runs in a Windows terminal, so we can simply have Python do its own I/O.

It doesn’t “lock up”. We get the same issue after calling this in the Linux version

?- py_call(janus:connect_io()).

before ?- go.. The debugger says that the Python thread that processes the input exits as a side effect of the print() call. This happens if you print before the first MIDI message is received. After the first MIDI message is received you can print fine!?

My Python experience is too limited to debug that in a reasonable time.

Now for a version without using any Python code. With more work on threads in Janus, this requires the GIT version (or tomorrows daily build).

:- use_module(library(janus)).

inputs :-
    py_call(mido:get_input_names(), Names),
    format('MIDI Inputs: ~p~n', [Names]).

capture(Dev) :-
    py_call(mido:open_input(Dev), Port),
    thread_create(input_from(Port), _).

:- dynamic
       midi_message/5.		% Channel, Type, Note, Velocity, Time

input_from(Port) :-
    forall(repeat, receive_one(Port)).

receive_one(Port) :-
    py_call(Port:receive(), Msg),
%   py_pp(Msg),
    get_time(Now),
    py_call(Msg:type, Type),
    py_call(Msg:channel, Channel),
    py_call(Msg:note, Note),
    py_call(Msg:velocity, Velocity),
    assertz(midi_message(Channel, Type, Note, Velocity, Now)),
    py_free(Msg).   % may also be left to GC, but this probably requires less resources.

I also did a little performance analysis. On Linux it seems the time to send and receive a MIDI message
is about 50 us, i.e., about 20,000 messages a second. I don’t know how this is spread over sending, receiving and the Linux MIDI infrastructure.

1 Like

Thanks a lot for this effort. I’m very impressed. The latter solution looks nicer because only one (Prolog) script is needed. I’ll need to study it in more detail as I’m not familiar with Prolog’s threads handling. I’ll let you know more after I’ve tested it this weekend.

I’m sure 20,000 messages a second will be more than enough, but it’ll depend on the hardware used. MIDI was designed for transmitting at a maximum rate of 31250 bps (baud), due to the hardware limits at the time. That means approximately 1000 short messages per second. If you don’t use old MIDI gear, but USB MIDI interfaces or a software only solution, this old limit no longer matters.

Good. Before starting, get the latest daily :slight_smile: I’ve pushed a fix that avoids a possible deadlock and some new stuff that allows for better cooperation between preemptive SWI-Prolog and cooperative Python threads. Multi-threaded applications should now work pretty much ok, although bugs are not impossible. I think I now understand most of the Python thread and GIL limitations, but not everything :frowning:

The latter solution works fine. Finally a bidirectional MIDI interface for SWI Prolog ! It took some time but it was definitely worth it.

Here is a script that filters incoming MIDI messages according to the 5 types that I needed : note on, note off, control change, program change and sysex.
It also creates notes with time stamps and durations : note(Channel, Note, Velocity, Start, Duration)

:- use_module(library(janus)).

inputs :-
    py_call(mido:get_input_names(), Names),
    format('MIDI Inputs: ~p~n', [Names]).

capture(Dev) :- % call this with your inport's name, for instance, capture('loopMIDI Port 0').
    py_call(mido:open_input(Dev), Port),
    thread_create(input_from(Port), _).

:- dynamic
	 note_on/4, 
	 note/5.

input_from(Port) :-
    forall(repeat, receive_one(Port)).

receive_one(Port) :-
    py_call(Port:receive(), Msg),
    py_call(Msg:type, Type),
    handle_midi_message(Type, Msg),
    py_free(Msg).  
    	
handle_midi_message('note_on', Msg) :- 
	get_time(Now),
	py_call(Msg:channel, Channel),
	py_call(Msg:note, Note),
	py_call(Msg:velocity, Velocity),
	assertz(note_on(Channel, Note, Velocity, Now)).
	
handle_midi_message('note_off', Msg) :- 
	get_time(Now),
	py_call(Msg:channel, Channel),
	py_call(Msg:note, Note),
	(	note_on(Channel, Note, Velocity, Start)
	->	retract(note_on(Channel, Note, Velocity, Start)),
		Duration is Now - Start,
		write_ln(note(Channel, Note, Velocity, Start, Duration)),
		assertz(note(Channel, Note, Velocity, Start, Duration))
	;	write_ln(unmatched_note_off(Channel, Note))
	).
	
handle_midi_message('control_change', Msg) :-
	py_call(Msg:channel, Channel),
	py_call(Msg:control, Control),
	py_call(Msg:value, Value),
	write_ln(control_change(Channel, Control, Value)).
	
handle_midi_message('program_change', Msg) :-
	py_call(Msg:channel, Channel),
	py_call(Msg:program, Program),
	write_ln(program_change(Channel, Program)).
	
handle_midi_message('sysex', Msg) :- 
	py_call(Msg:data, Data),
	write_ln(sysex(Data)).
	
handle_midi_message(Type, _) :-
	write_ln(not_supported_message(Type)).

:slight_smile: It did indeed take some time, but made Janus a lot more mature, notably in handling threads. While Janus was first of all designed for single threaded usage because it was designed for XSB and while XSB has threads, unlike SWI-Prolog it doesn’t focus on them. With support for threads we can use Janus to embed Prolog in Python (web-)services, etc. Multiple Python threads can now call Prolog and while Prolog is working on the query, Python can switch to other threads.

Simply use the code below. retract/1 instantiates its arguments and fails if no matching clause is found, just like calling it. Many people miss this :slight_smile:

    (   retract(note_on(Channel, Note, Velocity, Start))
    ->  ...

? write_ln/1 does not exist. writeln/1 does. What you probably want though is debug/3 to write

debug(note(duration), 'unmatched_note_off(~p, ~p)', [Channel, Note]).

Now you can use debug/1 to enable/disable these messages. The first argument is the debug channel. That can be any term (atom, compound, …) and only debug/3 calls with an enabled channel (using debug/1) are printed. If you load the code using -O (optimized), the debug/3 calls are removed by the compiler.

Thanks for reviewing my code. It’s a miracle that the non-existing write_ln/1 works. :slight_smile:

I now wanted to get rid of the sleep/1 predicate, separating note on and off events. I guess those sleep/1 predicates cannot be interrupted and block all input for the thread the code is running in.

I wrote some conceptual and very simplified code to test the alarm/3 predicate and it seems to work. However, SWI Prolog (I installed swipl-w64-2023-10-10.exe) crashes as soon as I add

py_call(OutPort:send(eval(mido:'Message'(note_on, channel=Channel, note=Note, velocity=Velocity))))

Here is the code where I commented out two py_call/1 calls.

generate_chords(OutPortName) :-	%2023-10-10
	py_call(mido:open_output(OutPortName), OutPort),
	create_chord(OutPort, 9, 70),
	py_call(OutPort:close()).

% main chord creation predicate, simplified to a single note per chord
create_chord(_, _, 75).
create_chord(OutPort, Channel, Chord) :- 
  Velocity is 90,
  start_chord(OutPort, Channel, Chord, Velocity),
  Duration is 1, % 1 sec
  % preparation of chord
  NewChord is Chord + 1,	% here I simplify to the bare minimum
  % ...further preparation
  alarm(Duration, stop_and_create_new_chord(OutPort, Channel, Chord, NewChord), _, [remove(true)]).

start_chord(OutPort, Channel, Note, Velocity) :-
  writeln(start_chord(OutPort, Channel, Note)).
  %py_call(OutPort:send(eval(mido:'Message'(note_on, channel=Channel, note=Note, velocity=Velocity)))).

stop_and_create_new_chord(OutPort, Channel, Note, NewChord) :-
  writeln(stop_and_create_new_chord(OutPort, Channel, Note, NewChord)),
  %py_call(OutPort:send(eval(mido:'Message'(note_off, channel=Channel, note=Note, velocity=0)))),
  create_chord(OutPort, Channel, NewChord).

The output is correct but in a strange format because of the timer. The query returns with true initially, but then seems to hang after the last timer halts. Can this be avoided?

?- generate_chords('Microsoft GS Wavetable Synth 1').
start_chord(<py_Output>(0000021eb772c210),9,70)
true.

?- stop_and_create_new_chord(<py_Output>(0000021eb772c210),9,70,71)
start_chord(<py_Output>(0000021eb772c210),9,71)
stop_and_create_new_chord(<py_Output>(0000021eb772c210),9,71,72)
start_chord(<py_Output>(0000021eb772c210),9,72)
stop_and_create_new_chord(<py_Output>(0000021eb772c210),9,72,73)
start_chord(<py_Output>(0000021eb772c210),9,73)
stop_and_create_new_chord(<py_Output>(0000021eb772c210),9,73,74)
start_chord(<py_Output>(0000021eb772c210),9,74)
stop_and_create_new_chord(<py_Output>(0000021eb772c210),9,74,75)

It is in the “backcomp” (backward compatibility) library, from which it is lazily loaded. Had long forgotten about that.

sleep/1 doesn’t do anything, but it does process signals (interrupts such as user ^C and alarm triggers.

This is not going to work because create_chord/3 returns, after which you close the output. Later though, the alarm triggers, using OutPort again. So, either you let the last alarm trigger the close or you open the port and leave it open during your music generation. Note that, AFAIK, you can use the same output port from multiple threads and send commands to it.

It is more misinterpretation. I already explained generate_chords/1 returns. Therefore the true. But, the alarms are pending and will eventually fire. That is what you see. The prompt is not repeating, but if you hit RETURN, you’ll see it is reading next command. The exact behavior depends on the platform.

If you call alarm/3, you’ll have to make sure the thread remains alive to process the expiration of the timer and runs something that handles interrupts. e.g., sleep/1 does so, and so does the interactive toplevel.

I have my doubt that using alarm/3 is going to be a comfortable way to program this. Alternatively, you can create a thread and make it play this chord. There you can comfortably use sleep/1. You could also create a central timer that maintains the rhythm (forgive my poor knowledge on musical terms, especially in English) and use thread_wait/2 in several threads that each play an instrument (or part of an instrument).

There are lots of choices. The challenge is to find something that is easy to program and manage.
One option is to start at the other end: define how you would like to write down the music to be played and then think about making this notation work.