You could still save memory with opt_level='O1', as activations might be stored in FP16 as seen in this dummy code snippet:
import torch
import torch.nn as nn

import torchvision.models as models

from apex import amp

use_amp = True

model = models.resnet50()
model.cuda()


model.avgpool.register_forward_hook(lambda m, x, y: print(x[0].type(), y[0].type()))
x = torch.randn(1, 3, 224, 224).cuda()

torch.cuda.synchronize()
print(torch.cuda.max_memory_allocated()/1024**2)

if use_amp:
    model = amp.initialize(model, opt_level='O1')
output = model(x)

torch.cuda.synchronize()
print(torch.cuda.max_memory_allocated()/1024**2)

besides the speedup using TensorCores.