Hi, I would like to pretrain BERT by using DDP.
I saved pretrain dataset(350GB of large corpus) as torch.tensor.
When I run the code below, dataset is loaded in memory 8 times.
python -m torch.distributed.launch --nproc_per_node=8 train.py
How can I prevent it?
Thanks.