T5-large on multiple gpus.
Hi! I'm trying to fine-tune a T5-large on multiple gpus, so I basically use `torch.nn.DataParallel`, but when I get the output of the model which contains the loss and I do `loss.mean().backward()` I run into `cuda out of memory` as I think the loss is just on the first gpu and that's taken. What should I do?
For OOM errors, I guess you need to reduce batch_size or parallelize over more GPUs.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
