Printing out print(x.shape, x.min(), x.max(), y.shape, y_t.min(), y_t.max()) looks like this:
torch.Size([32, 360, 2]) tensor(0., device='cuda:0') tensor(255., device='cuda:0') torch.Size([32, 360, 2]) tensor(-46.9311, device='cuda:0', grad_fn=<MinBackward1>) tensor(371.8865, device='cuda:0', grad_fn=<MaxBackward1>)
torch.Size([32, 360, 2]) tensor(0., device='cuda:0') tensor(255., device='cuda:0') torch.Size([32, 360, 2]) tensor(-101.9687, device='cuda:0', grad_fn=<MinBackward1>) tensor(349.1602, device='cuda:0', grad_fn=<MaxBackward1>)
torch.Size([11, 360, 2]) tensor(0., device='cuda:0') tensor(255., device='cuda:0') torch.Size([11, 360, 2]) tensor(nan, device='cuda:0', grad_fn=<MinBackward1>) tensor(nan, device='cuda:0', grad_fn=<MaxBackward1>)

I am running this loss function on top of an object detection model, hence when it becomes nan, the region proposal no longer proposes 32 boxes. Hence, the shape at the end is [11,360,2].
Furthermore, I ran torch.cdist which computes the pairwise distance for a single input. I used a for loop to compute a batch of 32. This gives the same value as the batch_pairwise_squared_distances function, however, it does not cause the weights to be NaN. This tells me there is something about torch.bmm that is causing the model to be unstable during training.