The amount of memory used by the model’s parameters may not make up the bulk of memory usage during training so in general this is a difficult problem to solve without more complexity like activation checkpointing or model-parallelism with multiple GPUs. The first thing you might try is to use some form of mixed-precision e.g., moving parts of the model to fp16 or bfloat16 which would potentially also improve speed