pytorch.org
  
  
    

Getting Started with Distributed Data Parallel — PyTorch Tutorials 1.7.0... 3



  
  
    
    
  
  


I have read the tutorial and I run the demo_basic with 2 visible gpus. In the function, I output the device of labels and outputs, and they are on gpus, while the inputs of torch.randn(20, 10) is on cpu. The tutorial is a little confused that the backend is gloo, but the main function tries to use gpus.
My question is that why don’t we need to transfer the inputs to gpus ?
Also, the tutorial might be more practical if dummy datasets and dataloaders are provided.
def demo_basic(rank, world_size):
    print(f"Running basic DDP example on rank {rank}.")
    setup(rank, world_size)

    # create model and move it to GPU with id rank
    model = ToyModel().to(rank)
    ddp_model = DDP(model, device_ids=[rank])

    loss_fn = nn.MSELoss()
    optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)

    optimizer.zero_grad()
    outputs = ddp_model(torch.randn(20, 10))
    labels = torch.randn(20, 5).to(rank)
    loss_fn(outputs, labels).backward()
    optimizer.step()

    cleanup()