In the past I use nviddia-apex O1 mode to train my model. And the torch.cuda.amp seems to save more memory. I compared apex and torch.cuda.amp respectively for 1000 iterations, and find the performance gap on val set was 8 (mIoU for Semantic segmentation). The model with apex was 30 mIoU and with torch.cuda.amp was 22 mIoU. Is this normal ?
Here are some of the key implementations::
def __init__(self,):
    self.scaler = GradScaler()
def step(X, label)
    with autocast(enabled=True):
        _, output = self.forward(input=X)
        outputUp = F.interpolate(output, size=X.size()[2:], mode='bilinear', align_corners=True)
        loss = self.loss_source(input=outputUp, target=label)
    self.amp_backward(loss, self.BaseOpti)


And the amp_backward function was:
def amp_backward(self, loss, optim):
    self.scaler.scale(loss).backward()
    self.scaler.step(optim)
    self.scaler.update()