I’m not sure if there is another way to optimize this.
Assuming you are dealing with N images, your model would have to calculate
N + N-1 + N-2 + ... + 1 = (N+1) * N / 2 comparisons.
You could speed it up using batches, but if I understand your explanation correctly, you are avoiding unnecessary calculations.