Validation loop gives OOM when finetuning T5
While finetuning T5-base on summarization task, using `--sortish_sampler` it gives an OOM error starting from a particular index during the validation loop. After removing those indices and training again, I still get the OOM error, but in the second validation loop, whereas the validation loop worked well the first time in the 0th epoch. I have finetuned t5-base previously as well on the same dataset and the same environment, and it never gave this error. 
I am using a batch size of 1 during training and evaluation both. 

I am using Colab to finetune the model on a single GPU.
GPU specs:
GPU: Tesla K80
RAM: 12GB

This is the traceback:

Epoch 0:  91% 5644/6231 [25:04<02:36,  3.75it/s, loss=2.089, v_num=16]
Epoch 0:  91% 5645/6231 [25:06<02:36,  3.75it/s, loss=2.089, v_num=16]
Epoch 0:  91% 5646/6231 [25:07<02:36,  3.75it/s, loss=2.089, v_num=16]
Epoch 0:  91% 5647/6231 [25:08<02:35,  3.74it/s, loss=2.089, v_num=16]
Epoch 1:  84% 5247/6231 [18:08<03:24,  4.82it/s, loss=1.859, v_num=16]
Validating: 0it [00:00, ?it/s]
Epoch 1:  84% 5248/6231 [18:10<03:24,  4.81it/s, loss=1.859, v_num=16]
Epoch 1:  84% 5249/6231 [18:11<03:24,  4.81it/s, loss=1.859, v_num=16]Traceback (most recent call last):
  File "finetune.py", line 441, in <module>
    main(args)
  File "finetune.py", line 416, in main
    logger=logger,
  File "/content/drive/My Drive/Colab Notebooks/transformers_20_Oct_summarization/transformers/examples/lightning_base.py", line 386, in generic_train
    trainer.fit(model)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/trainer.py", line 440, in fit
    results = self.accelerator_backend.train()
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/accelerators/gpu_accelerator.py", line 54, in train
    results = self.train_or_test()
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/accelerators/accelerator.py", line 66, in train_or_test
    results = self.trainer.train()
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/trainer.py", line 483, in train
    self.train_loop.run_training_epoch()
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/training_loop.py", line 569, in run_training_epoch
    self.trainer.run_evaluation(test_mode=False)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/trainer.py", line 568, in run_evaluation
    output = self.evaluation_loop.evaluation_step(test_mode, batch, batch_idx, dataloader_idx)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/evaluation_loop.py", line 171, in evaluation_step
    output = self.trainer.accelerator_backend.validation_step(args)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/accelerators/gpu_accelerator.py", line 76, in validation_step
    output = self.__validation_step(args)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/accelerators/gpu_accelerator.py", line 86, in __validation_step
    output = self.trainer.model.validation_step(*args)
  File "finetune.py", line 181, in validation_step
    return self._generative_step(batch)
  File "finetune.py", line 221, in _generative_step
    max_length=self.eval_max_length,
  File "/usr/local/lib/python3.6/dist-packages/torch/autograd/grad_mode.py", line 15, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/generation_utils.py", line 489, in generate
    model_kwargs=model_kwargs,
  File "/usr/local/lib/python3.6/dist-packages/transformers/generation_utils.py", line 665, in _generate_beam_search
    outputs = self(**model_inputs, return_dict=True)  # (batch_size * num_beams, cur_len, vocab_size)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_t5.py", line 1212, in forward
    return_dict=return_dict,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_t5.py", line 767, in forward
    output_attentions=output_attentions,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_t5.py", line 556, in forward
    output_attentions=output_attentions,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_t5.py", line 478, in forward
    output_attentions=output_attentions,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_t5.py", line 374, in forward
    q, k.transpose(3, 2)
RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 14.73 GiB total capacity; 13.70 GiB already allocated; 13.88 MiB free; 13.72 GiB reserved in total by PyTorch)
Epoch 1:  84%|████████▍ | 5249/6231 [18:13<03:24,  4.80it/s, loss=1.859, v_num=16]


Any help is appreciated. :)
Maybe @sshleifer or @patil-suraj can help here? :-) <|||||>We can't help without:

1) a command that used to work and now OOMs.
2) Some notion of when it worked. (Ideally version numbers but a guess is fine.)
3) current `transformers-cli env` and `pip freeze | grep torch` outputs.<|||||>Maybe colab allocated you something larger than 12GB K80 the last time you ran your command?<|||||>> We can't help without:
> 
> 1. a command that used to work and now OOMs.
> 2. Some notion of when it worked. (Ideally version numbers but a guess is fine.)
> 3. current `transformers-cli env` and `pip freeze | grep torch` outputs.

Yeah, I understand what you're saying. Itrained T5 like a month back on colab. Also I wanted to know, if the OOM error comes sporadically, i.e., sometimes in the first epoch, sometimes in the second epoch, and it's always during the validation loop, what should I conclude from it? It is an error in my code, or it is just the lack of appropriate memory.<|||||>Unfortunately, it's not clear what to conclude. 
You can force eval to use less memory by controlling `val_max_target_length`, `eval_max_gen_length`, `val_batch_size`, and `eval_num_beams`.
<|||||>Alright. Thank you so much.<|||||>Alternatively, you can try to use `Seq2SeqTrainer`.
`Trainer` has recently added `eval_accumulation_step` argument which offloads the `logits/predictions` to `cpu` every `eval_accumulation_steps` to avoid OOM, you can use this with `Seq2SeqTrainer` as well.<|||||>> Alternatively, you can try to use `Seq2SeqTrainer`.
> `Trainer` has recently added `eval_accumulation_step` argument which offloads the `logits/predictions` to `cpu` every `eval_accumulation_steps` to avoid OOM, you can use this with `Seq2SeqTrainer` as well.

Thank you so much. Tried this and it worked well. :)<|||||>> > Alternatively, you can try to use `Seq2SeqTrainer`.
> > `Trainer` has recently added `eval_accumulation_step` argument which offloads the `logits/predictions` to `cpu` every `eval_accumulation_steps` to avoid OOM, you can use this with `Seq2SeqTrainer` as well.
> 
> Thank you so much. Tried this and it worked well. :)

would you share what `eval_accumulation_steps` you used? Thanks.<|||||>I am having the exact same issue, this is happening only during evaluation and not training. thanks<|||||>This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.