Looking at this a bit more your code seems problematic. You can’t just detach and then reattach tensors to the computational graph, sorry that I didn’t really read carefully before. Can you explain why you don’t just use (gt_coord-pred_coord)**2 ?
Is it because you somehow don’t care about the ordering of the coordinates, just that some of them overlap?
cv2.circle is not differentiable in pytorch, so I think it’s not possible to use it, however there might be some easy way to implement it in pytorch instead.