Training large huggingface models on Azure with CUDA? [OPT]
I am trying to finetune the 1.3b and 30b OPT variant models respectively, however each time I try to train them I will always get the following error:
```
RuntimeError: CUDA out of memory. Tried to allocate 16.00 MiB (GPU 0; 11.17 GiB total capacity; 1.95 GiB already allocated; 8.50 MiB free; 1.95 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
```

I have tried reducing the batch size to 1. However short of running the model on CPU which would take way too much time, it will crash each time. If it's relevant I am using an Azure environment which uses the Tesla K80 accelerator. Has anyone managed to train or use these models on GPU?
Hi, for training such large models, a Tesla K80 won't suffice. You typically need # of parameters * 18 in terms of bytes of RAM when fine-tuning. So for the 30 billion parameter model, it's 30 billion * 18 = 540 GB. It's because you need 18 bytes per parameter to store not only the parameter itself, but also its gradient and optimizer states.

However there are tricks like mixed precision and frameworks like DeepSpeed to fit giant models, you can read more in our guide here: https://huggingface.co/docs/transformers/performance