How to fine-tune BERT on a large training dataset?
## ❓ Questions & Help

Hi,

I want to fine-tune BERT on a large training dataset. With around 1.5million training examples, this is currently consuming around 60GB of RAM. Is there any way to reduce the RAM usage or load the training examples in parts? Thanks!.

I think you should check out pytorch dataloaders https://pytorch.org/docs/stable/data.html
Also, gradient accumulation is helpful.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
