The more natural way to model “adjacent” in Prolog would be:
:- dynamic adjacent/2.
adjacent(a, b).
adjacent(a, c).
adjacent(a, d).
Do you see how things got easier already, both for maintaining the adjacency matrix and for querying?
If you need to efficiently manipulate a list, look at the code that Jan kindly shared in this post.