I’m running a bilstm + crf model. If I chose vocab_size of 10000, batch_size = 512, the forward speed is equal to use dataparalle , gpu = 4, batch_size = 2048. However, if I chose vocab_size of 0.5million, the forward speed on sigle gpu is 8 times faster than dataparallel, gpu = 4, batch_size = 2048. The speed on two case of single gpu is similar. I don’t know why . I have already set embedding.require_grad = False