Electra multi-gpu pretraining.
Hi, I am pretraining the Electra model with my own data, for now, I am pretraining using one GPU in my machine. Can we use multi GPUs to pretrain Electra? Thanks for your reply

That would depend of the training script. All of our models are `nn.Module`s so all can be trained on multi-GPU, it just depends on the script. What script are you using?<|||||>@LysandreJik  Hi, thanks for your reply, i am not pretty sure about script, what I am doing now is using TensorFlow google Electra, but it seems I haven't figure out how to use multi gpu in TensorFlow, that's why i am here, Do u have any advice, I mean maybe I can just use Electra in this project to achieve multi-gpu pretraining?<|||||>Yes, you could check [this thread](https://discuss.huggingface.co/t/electra-training-reimplementation-and-discussion/1004) and use their project to train ELECTRA, which is based on this repository.<|||||>@LysandreJik  thanks, however, according to this issue https://github.com/richarddwang/electra_pytorch/issues/5 , I guess they haven't figure out how to use multigpu yet,<|||||>Ah, then I don't think I can help you further, unfortunately.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
