Hi. I want to use multi-gpus but I don’t know how to use it.
I used the DataParallel function because it seems to be the most basic way.
When I use 2 gpus, the images are well assigned to each gpus, but the speed at which the actual code runs is the same as when using one gpu.
Below is the code I used.
# create model
print("=> creating model 'resnet18'")
model = resnet.resnet18()

if torch.cuda.device_count() > 1:
    print("Using", torch.cuda.device_count(), "GPUs")
    # DataParallel will divide and allocate batch_size to all available GPUs
    model = nn.DataParallel(model.cuda(), device_ids=[0,1])

# define loss function (criterion) and optimizer
criterion = nn.NLLLoss()
softmax = nn.Softmax(1)

optimizer = torch.optim.SGD(model.parameters(), learning_rate,
                            momentum=momentum,
                            weight_decay=weight_decay)

cudnn.benchmark = True


## train code
def train(train_loader, model, criterion, optimizer, epoch):

    # switch to train mode
    model.train()

    for i, (images, target) in enumerate(train_loader):

        images = images.to(device)
        target = target.to(device)

        out = model(images)
        out_sm = softmax(out)
        log = torch.log(out_sm)
        
        loss = criterion(log, target)

        # compute gradient and do SGD step
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

    return loss

Is there anything I missed?
Thanks