Hi there,
I am pretty new, I hope to do it right:)
I have two gpus nvidia, which work fine. I can train model on each of them, I can use data parallelism. I wonder if I can parallelize the model itself. Surfing the internet I found it is possible but no one tells how. Some frameworks do it as torchgpipe, deepspeed PipelineModule, Fairscale but they wants sequential models but transformers are hard to turn sequential.
Can you point me in the right direction?
Specs:
I want to parallelize BERT model on two gpus titan xp.
Thank you, every hints or helps will be appreciated
valgi0