@lhoestq thanks for quick (and helpful!) response. Decided to deal with things one at a time, so have concentrated on trying to get your more information on the memory issue when loading larger files. I tried various runs with my data (a T5 denoising fine-tuning exercise). I have two sets of train/validate/test files one set where the training csv file is about 1.4GB (400k rows), one set where the training csv file is 600MB (225K rows). I tried t5-small, t5-base and t5-large models, with and without fp16. So large/small files x t5-small/base/large x fp16/not fp16 = 12 runs. I am on a Windows 10 machine with 1 NVIDIA GPU (RTX 3090) with 24GB of memory. I was running the run_summarization.py with max_train_samples 5000 and max_eval_samples 500, per_device_train_batch_size = 4, eval_steps 500.
I logged the first 500 steps and the 1st eval on wandb and put together this set of results showing GPU memory consumption.

LargeFileProbs1159×879 73.1 KB

Findings:

t5-large with fp16/no fp16 just blew memory which are the two single dots on the left of chart (not showing them above 100% in this wandb chart because effectively they didn’t start)
t5-large with the small files without/with fp16 is up near 100% and 80% GPU memory allocated
t5-small with the small files without/with fp16 is up near 20% and 15% GPU memory allocated
t5-small with the large files without/with fp16 is up near 40% and 30% GPU memory allocated
t5-base with the small files without/with fp16 is up near 45% and 39% GPU memory allocated
t5-base with the large files without fp16 rose to 100% then settled down to 60% (this seemed strange/interesting so I did two runs of this to check it repeated)
t5-base with the large files with fp16 rose to 100% and stayed there (this seemed strange/interesting so I did two runs of this to check it repeated)

Comments:

With t5-large no fp16 with small files I am banging up against the 24GB available on the GPU
(fp16 gives me some head room at 80% but also returns NaN, a different but maybe related issue?)
Using input files (aka more samples) seems to require more GPU memory for the model?
The up to 100% then back down to 60% behaviour of t5-base with large files seems interesting
(adding fp16 takes it up to 100% and there it stays for t5-base)

And is one of the two stack text dumps for when t5-large immediately runs out of GPU memory:
wandb: Run wandb offline to turn off syncing.
0%|          | 1/3750 [00:02<2:09:23,  2.07s/it]Traceback (most recent call last):
File “C:\Users\BrianS\PycharmProjects\PyTorch\transformers\examples\pytorch\summarization\run_summarization.py”, line 624, in 
main()
File “C:\Users\BrianS\PycharmProjects\PyTorch\transformers\examples\pytorch\summarization\run_summarization.py”, line 548, in main
train_result = trainer.train(resume_from_checkpoint=checkpoint)
File “C:\Users\BrianS.virtualenvs\summarization-DUOCBs9B\lib\site-packages\transformers\trainer.py”, line 1287, in train
tr_loss += self.training_step(model, inputs)
File “C:\Users\BrianS.virtualenvs\summarization-DUOCBs9B\lib\site-packages\transformers\trainer.py”, line 1782, in training_step
loss = self.compute_loss(model, inputs)
File “C:\Users\BrianS.virtualenvs\summarization-DUOCBs9B\lib\site-packages\transformers\trainer.py”, line 1814, in compute_loss
outputs = model(**inputs)
File “C:\Users\BrianS.virtualenvs\summarization-DUOCBs9B\lib\site-packages\torch\nn\modules\module.py”, line 1051, in _call_impl
return forward_call(*input, **kwargs)
File “C:\Users\BrianS.virtualenvs\summarization-DUOCBs9B\lib\site-packages\transformers\models\t5\modeling_t5.py”, line 1561, in forward
encoder_outputs = self.encoder(
File “C:\Users\BrianS.virtualenvs\summarization-DUOCBs9B\lib\site-packages\torch\nn\modules\module.py”, line 1051, in _call_impl
return forward_call(*input, **kwargs)
File “C:\Users\BrianS.virtualenvs\summarization-DUOCBs9B\lib\site-packages\transformers\models\t5\modeling_t5.py”, line 998, in forward
layer_outputs = layer_module(
File “C:\Users\BrianS.virtualenvs\summarization-DUOCBs9B\lib\site-packages\torch\nn\modules\module.py”, line 1051, in _call_impl
return forward_call(*input, **kwargs)
File “C:\Users\BrianS.virtualenvs\summarization-DUOCBs9B\lib\site-packages\transformers\models\t5\modeling_t5.py”, line 691, in forward
hidden_states = self.layer-1
File “C:\Users\BrianS.virtualenvs\summarization-DUOCBs9B\lib\site-packages\torch\nn\modules\module.py”, line 1051, in _call_impl
return forward_call(*input, **kwargs)
File “C:\Users\BrianS.virtualenvs\summarization-DUOCBs9B\lib\site-packages\transformers\models\t5\modeling_t5.py”, line 301, in forward
forwarded_states = self.DenseReluDense(forwarded_states)
File “C:\Users\BrianS.virtualenvs\summarization-DUOCBs9B\lib\site-packages\torch\nn\modules\module.py”, line 1051, in _call_impl
return forward_call(*input, **kwargs)
File “C:\Users\BrianS.virtualenvs\summarization-DUOCBs9B\lib\site-packages\transformers\models\t5\modeling_t5.py”, line 261, in forward
hidden_states = self.dropout(hidden_states)
File “C:\Users\BrianS.virtualenvs\summarization-DUOCBs9B\lib\site-packages\torch\nn\modules\module.py”, line 1051, in _call_impl
return forward_call(*input, **kwargs)
File “C:\Users\BrianS.virtualenvs\summarization-DUOCBs9B\lib\site-packages\torch\nn\modules\dropout.py”, line 58, in forward
return F.dropout(input, self.p, self.training, self.inplace)
File “C:\Users\BrianS.virtualenvs\summarization-DUOCBs9B\lib\site-packages\torch\nn\functional.py”, line 1168, in dropout
return VF.dropout(input, p, training) if inplace else _VF.dropout(input, p, training)
RuntimeError: CUDA out of memory. Tried to allocate 38.00 MiB (GPU 0; 24.00 GiB total capacity; 21.38 GiB already allocated; 11.31 MiB free; 21.97 GiB reserved in total by PyTorch)
wandb: Waiting for W&B process to finish, PID 1268
wandb: Program failed with code 1.  Press ctrl-c to abort syncing.
wandb: - 0.00MB of 0.00MB uploaded (0.00MB deduped)
wandb:
wandb: Find user logs for this run at: C:\Users\BrianS\PycharmProjects\PyTorch\transformers\examples\pytorch\summarization\wandb\run-20210821_122954-1rtixwev\logs\debug.log
wandb: Find internal logs for this run at: C:\Users\BrianS\PycharmProjects\PyTorch\transformers\examples\pytorch\summarization\wandb\run-20210821_122954-1rtixwev\logs\debug-internal.log
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb:
wandb: Synced wandb T5-Large no FP16 MIMIC_III large file: Weights & Biases
Process finished with exit code 1