self.cos_sim(utt_embedding.repeat(batch_embedding.shape[0],1)

What does self.cos_sim do?
Besides it may happen because spk_embedding comes from batch_embeeding
Think that utt_embedding comes from batch_emmbeding so if you modify it, it counts also as an in-place operation.