I assume you are using nn.BCELoss based on your description and the usage of the sigmoid activation? If so, could you remove the sigmoid and replace the criterion with nn.BCEWithLogitsLoss, as it would give you more numerical stability?
Based on your usage of self.dot = torch.matmul it seems you are creating an output tensor of the shape [batch_size, batch_size]. I assume this is expected?