There is no magic way to do this I’m afraid.
allclose will work for an op that is element-wise or with very small inputs.
But operations that contain reductions (sum, mm, conv, etc) will have a tendency to increase the error, the bigger the reduced dimension is.
So you can actually end up with very significant differences, I would even say arbitrarily large differences if you can make the inputs as big as you want.