I think you’re seeing numerical precision:
a = torch.empty(3, 3)
a = torch.nn.init.orthogonal_(a)
almost_eye = torch.mm(a, a.t())
print((almost_eye - torch.eye(3)).abs().max().item())

gives something < 1e-6 or so.