Solved by KFrank in post #5 
              
              
                Hi Leon! 

I don’t really follow your embedding scheme, and I can’t comment 
on your code. 
But if I understand goal correctly, perhaps you can project your 
gradients onto your desired subspace after an unmodified 
backward() step. 
The common optimizers use some variant of gradient descent 
where p…