Solved by ffuuugor in post #2 
              
              
                Hi! 
It is expected that Opacus has a certain memory overhear. At the very least, we have to store per-sample gradients for all model parameters - that alone increases the memory required to store gradient by the factor of batch size. 
In order to address that I suggest using virtual_step() method i…