Solved by Mathemagier in post #2 
              
              
                Okay I think, that I can answer my own question. The reason here was really, that in the Feed-Forward pass the stored matrices are too large. I use attention and the my matrices for Q,K,V (multi-headed self attention) are of shape (BS, heads=8, ~4000, 1024). One matrix is 9Gb large!  You will see th…