How to calculate memory requirements of different GPT models? 
## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

What's the best way to calculate how much GPU ram is needed to finetune a given GPT size? The GPT-xl model is 6.5GB worth of parameters but clearly this isn't the full story when it comes to fine-tuning. I've tried running a batch size of 1 on a 11GB 1080ti but that ran out of memory. My next step is to test it on a 2080ti with FP16 training enabled to further reduce memory requirements. It would be great if there was some back of the envelope calculation to see in advance if this would work or if instead I should go straight to 16GB Titan V. Or maybe even that's too small!

Current commercial GPU VRAM sizes are not large enough to process the XL model. You'll need probably around 32-64 GB of RAM in order to finetune successfully. So for now, you'll have to train on a CPU with a lot of memory attached.<|||||>How did you come to that number though? 

In Thomas Wolf's medium post on training large models he mentioned gradient checkpointing for models that couldn't fit a batch onto a single gpu. Is it possible to implement something like for GPT-2 and friends?<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
