Using the accelerate MLM example still results in CUDA out of memory
Hi,
I am using [this example](https://github.com/huggingface/transformers/blob/master/examples/pytorch/language-modeling/run_mlm_no_trainer.py) and instructions on [this discussion](https://discuss.huggingface.co/t/how-to-use-specified-gpus-with-accelerator-to-train-the-model/10967) to adapt the code from [run_summarization.py](https://github.com/huggingface/transformers/blob/master/examples/pytorch/summarization/run_summarization.py), and train (fine tune) a summarization BART model (specifically     model_name="sshleifer/distilbart-xsum-12-6" that I got from HF model repo).  I have one GPU and my batch size is 8.  My training data sample size is 15k. 
 However, as soon as the training starts, I get the following error:

`RuntimeError: CUDA out of memory. Tried to allocate 512.00 MiB (GPU 0; 14.76 GiB total capacity; 13.49 GiB already allocated; 419.75 MiB free; 13.54 GiB reserved in total by PyTorch)`

My full code can be found [here](https://github.com/tanyaroosta/test/blob/master/accl_trainer.py).  

Other relevant parameters:
cuda available" True
current device: 0
device count: 1
device name: Tesla T4
Allocated: 1.1 GB
accelerator device" cuda
***** Running training *****
  Num examples = 15909
  Num Epochs = 3
  Instantaneous batch size per device = 8
  Total train batch size (w. parallel, distributed & accumulation) = 8
  Gradient Accumulation steps = 1
  Total optimization steps = 5967

I would appreciate any help as I am at a loss of what else to try to get this training to go through.  I am using a SageMaker instance of ml.g4dn.16xlarge.

Thanks for your help.
Hi,

Can you try out this: https://github.com/rentruewang/koila

Let me know if it resolves your issue.<|||||>koila didn't work for me. I basically had to use a p3.24 instance to get it to work with a batch size of 1.