To be honest I don’t quite get your explanation but I think it is good that you try to experiment by yourself.
A few additional insights:

It seems that your batches are completely independant, therefore it is almost certain that you can vectorize it

(refer_h - cur_select) ** 2 looks like the squared distance to me. When you encounter a common operation always use the pytorch implementation instead of rewriting yourself. You can use torch.nn.functional.mse_loss here. It is implemented by the backend and may be significantly faster

Again let me answer again. Are you sure this is a “learnable” loss. Can you provide a mathematical formulation of it so I can have a better sense of what it is suppsed to do ?