makslevental:

        for i in range(10):
            img_tensor = next(iter(train_dataloader))
            img_tensor = img_tensor.to(rank)
            torch.cuda.synchronize(rank)
            dogs.append(dog(img_tensor))



Could you please wrap the above code with some time measure and check how much percentage does it contribute to the total delay of end-start?
BTW, you might be able to avoid the torch.cuda.synchronize(rank) above, as the copy  (tensor.to) should have synchronize the destination device properly.