You could try to use broadcasting as seen here:
a = torch.arange(4).float().view(4, 1)
b = torch.arange(4).float().view(4, 1)

# element-wise
print(a - b)

# pair-wise
print(a.unsqueeze(1) - b)

which would result in a higher memory footprint, but might be faster than your sequential approach.