Hello,
Avoid any for loop and only use vectorized operations. For example for exp_sum I would do:
def optimizer_sum(refer_h, others, sigma):
    result = torch.exp(- (refer_h - others) ** 2 / (2 * sigma ** 2))
    return result.view(others.size(0), -1).mean(1).sum()

To be honest I did not read the entire thing but If you keep that in mind you should be able to get rid of all your for loops.
Since I did not read your entire cost function so, I’m not sure, but your loss seems WAY too complicated to be a cost function. If you want to do gradient descent on it, make sure it is smooth enough so that you can find a decent solution.
Hope it helped