Rational trees A, B are incomparable if (modified) compare_with_stack/3
detects same pairs
in the stack, that is, so far, it is procedural notion. However, I think it could be defined as follows: distinct X and Y are incomparable if neither X< Y nor Y< X. A binary relation < on rational trees is defined first, It follows that < is transitive, < is not total, if X<Y then X<Z<Y for some Z.
EDIT I remember lengthy definition of mine.
Let A, B, C, D be terms with A\ne B, C\ne D. Define (A, B) \to (C, D) if A = f(A_1,A_2,\ldots,A_i, \ldots,A_n), B = f(B_1,B_2,\ldots,B_i, \ldots,B_n), A_j=B_j (1\leq j<i) A_i\neq B_i, for some n, f, 1\leq i\leq n.
Two cyclic terms X, Y are incomparable if there are infinite series of pairs
(X, Y)=(X_1,Y_1)\to(X_2,Y_2)\to\cdots\to(X_k,Y_k)\to\cdots.
Let X=f(Y,0), Y=f(X, 1) (Mats Carlsson). Note that X\ne Y. X,Y are incomparable because of infinite series
(X, Y)\to(Y,X)\to(X,Y)\to(Y,X)\to\cdots.
END
The following is just for curiosity, where incomparable pairs are identified, though still I can’t imagine situations in which sorting rational trees are useful.
% ?- predsort(x_compare, [c, b, b, a], R).
%@ R = [a, b, c].
% ?- X=f(Y, 0), Y=f(X, 1), predsort(x_compare, [X, Y, X, Y], R).
%@ X = f(f(X, 1), 0),
%@ Y = f(X, 1),
%@ R = [f(f(X, 1), 0)].
x_compare(C, X, Y):- compare_with_stack(C0, X, Y),
( C0 = incomparable -> C = (=)
; C = C0
).