A bit late to the party, but starting in 1.10 this feature is supported natively in PyTorch:
https://pytorch.org/docs/stable/generated/torch.nn.utils.parametrizations.orthogonal.html 32
Leaving this here in case someone else bumps into this post in the future.