I guess there no issue with the model code.
what would I suggest is to print the size of each intermediate variable, because the error is related to sizes only.
I feel there is a size mismatch with the similarity matrix,  look into torch.diag() specifically.