@albanD I was readint the api/docs for DistributedDataParallel — PyTorch 1.7.1 documentation 9 and it says this (which worries me for my application):

This container parallelizes the application of the given module by splitting the input across the specified devices by chunking in the batch dimension

the chunking is problematic. My models do not receive thesors. They received special parsed objects that I need to process. Batching this is not easy this is why I am resorting to distributed. Let assume that I am right that batching is hard. If that is the case then how would DDP “chunk” my list of python objects? Would it work at all?
e.g.
def demo_basic(rank, world_size):
    print(f"Running basic DDP example on rank {rank}.")
    setup(rank, world_size)

    # create model and move it to GPU with id rank
    model = ToyModel().to(rank)
    ddp_model = DDP(model, device_ids=[rank])

    loss_fn = nn.MSELoss()
    optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)

    optimizer.zero_grad()
    x = torch.randn(20, 10)  # imagine we change this to a list [ obj1, ..., obj_batch_size]
    outputs = ddp_model(x)
    labels = torch.randn(20, 5).to(rank)
    loss_fn(outputs, labels).backward()
    optimizer.step()

    cleanup()

what would happen if x is a list of objects instead?