@tom Hi Thomas, and YES I am trying to implement Orthogonal / Unitary RNN.

Waw, such an elegant solution, only one thing: what do you mean with "after someone called backward " ? Do you mean that before calling this optimizer the code should have already called backward in order for param to have .grad calculated? And by the way now I understand why in the PyTorch optimizer it is needed to pass only param-s.
I will discuss with you further this point because I am in the middle of some testings, but it is good to have your opinion because now I know at least that there is a way to do it and I just have to search to find it.

I thank you very much again for your support.
Cheers.
Ergnoor