Thank you ptrblck. Spliting the batch and sending them to multiple devices is all that I need, but it doesn’t do that. In each epoch I have a loop as below:
for images, targets in train_loader:	
        images = list(torchvision.transforms.ToTensor()(image).to(device) for image in images)
        targets = [{k: v.to(device) for k, v in t.items()} for t in targets]
        loss_dict = model(images, targets)
        ...

After loading each batch I send them to cuda devices, but it utilizes only the first gpu. If instead of train_loader I use an array (already filled with the data), this code works with all gpus.