What do you mean by “ground truths”?
Also, have you tried using the Model Parallel method? That will divide the model layers between GPUs instead of batches.
https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html 2