Solved by tom in post #2 
              
              
                Last year, quite a few people talked about KFAC, but it might not be too easy to fit into the pytorch optimizer framework. 
One of the most prominent implementations seems to be this: 
 
 
Best regards 
Thomas