I am facing a very starnge problem with torch.nn.DataParallel(). I have a system with 8 GPUs and I want to use multiple GPUs for training my model. Now when I wrap the model with nn.DataParallel, it works only for batch_size 10! This is very odd because for any batch size other than 10 ( even smaller), the execution just gets stuck. When I am not using parallelism and running on single GPU, it is working properly. But for batch_size more than 16, cuda is running out of memory because my input vectors are very large and model is very big. So I am unable to take advantage of multiple GPUs. Any soltuion out there? Thank you in advance…