I’ve replaced the code with the following cuda.amp code:
scaler = torch.cuda.amp.GradScaler()

for epoch in range(n_epochs):
    for i, (X_batch, y_gt) in enumerate(batches):
        X_batch = X_batch.cuda()
        y_gt = y_gt.cuda()
        optimizer.zero_grad()
        with torch.cuda.amp.autocast():
            y_pred = model(X_batch)
            loss = Cross_entropy(y_pred, y_gt)

        scaler.scale(loss).backward()
        scaler.step(optimizer)
        scaler.update()


It is training stably so far but it is really slow. I’m using Densenet121 backbone and some convolutions to output 19 segmentation classes with a batch size of 2 with 2 GPUs GTX1080Ti. The input size is 512x512. When I don’t use cuda.amp the training speed per iteration is ~2 seconds involving forward and backward passes. However, if I use cuda.amp, the speed per iteration is ~6.6 seconds.
System:
OS: 18.04
Pytorch: 1.6 (1.6.0+cu101)
Cuda: 10.1 (cannot get cuda 10.2 because I don’t have superuser access)
GPU: 2 GTX 1080Ti