promach:

So, are you implying that torch.cdist() function itself contains contiguous() call ?


Probably yes, and these lines 10 are probably the right ones.



 promach:

I tried to replace torch.cdist() using fast_cdist() as shown below. However, I still have GPU out-of-memory error.


In your code other methods, such as torch.cat will create contiguous tensors as seen here:
a = torch.randn(1, 1).expand(10, 10)
print(a.is_contiguous())
> False

b = torch.randn(10, 10)
print(b.is_contiguous())
> True

c = torch.cat((a, b), dim=1)
print(c.is_contiguous())
> True

The main issue, is that your data is too large for the applied operations, as at least some of them work on contiguous tensors, which will create the memory increase.
For mixed-precision training, I would recommend to install the nightly and use native amp as described here 2.