You would probably need to make your own custom data loader to handle data structures other than tensor, but the general scheme could be something like this?
model = DDP(model, ...)
sampler = DistributedSampler(dataset)
dataloader = CustomDataLoader(dataset, shuffle=False, sampler=train_sampler, batch_size=1)

for sample in dataloader:
  loss = model(sample)
  optimizer.zero_grad()
  loss.backward() #this is synchronized automatically
  optimizer.step()