Somehow I shied away from implementing call/n for my new
Prolog system. I thought my new Prolog system has only monomorphic
predicate lookup caches , I will never be able to replicate what
I did for my old Prolog system with arity polymorphic predicate
lookup caches. This changed when I had the idea to dynamically
add a cache for the duration of a higher order loop such as maplist/n,
foldl/n etc.. What can we do with these new toys, we can implement
vector operations and matrice operations. An then apply it for example
to layered neural networks by representing them as:
/**
* Network is represented as [N0,M1,N1,...,Mn,Nn]
* - Where N0 are the input neurons vector
* - Where N1 .. Nn-1 are the hidden neurons vectors
* - Where Nn are the output neurons vector
* . Where M1 .. Mn are the transition weights matrice
*/
?- mknet([3,2], X).
X = [''(-1, 1, 1), ''(''(1, 1, -1), ''(1, 1, -1)), ''(-1, 1)].
Here is a visualization of the [3,2] example:
The model evaluation at a data point is straight forward:
eval([V], [V]) :- !.
eval([V,M,_|L], [V,M|R]) :- !,
matmul(M, V, H),
vecact(H, expit, J),
eval([J|L], R).
The backward calculation of deltas is straight forward:
back([V], U, [D]) :- !,
vecact(U, V, sub, E),
vecact(E, V, mulderiv, D).
back([V,M,W|L], U, [D2,M,D|R]) :-
back([W|L], U, [D|R]),
mattran(M, M2),
matmul(M2, D, E),
vecact(E, V, mulderiv, D2).
You can use this to compute weight changes and drive
a gradient algorithm.
Edit 11.03.2025
But where is Autograd, automatic derivation from some symbolic input?
In general you can objectify neural networks which I already did
with the Prolog list, and routines such as back/3 are pure Prolog.
Basically you could symbolically derive expit (activation), mulderiv
(the product with the derivative of the activation) and matrran (the
jacobian without activation) from a DAG of vector functions. In a linear
neural network, the jacobian without activation is the same as the
weights, and expit has a simple derivative that is based on the
expit result itself which is already stored as the activation:
/* g(x) = logistic function */
expit(X, Y) :- Y is 1/(1+exp(-X)).
/* g'(x) = g(x)*(1-g(x)) */
mulderiv(X, Y, Z) :- Z is X*Y*(1-Y).
See also:
A Gentle Introduction to
torch.autograd
https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html