nn.CosineSimilarity casts to float32 using the mixed-precision utility via torch.cuda.amp.autocast as given here 1 so I assume the limited numerical precision of float16 creates the large error. Since you are manually calling half() on the data float16 will be used and ill thus create the mismatch.