Mixing native amp (via torch.cuda.amp) and apex/amp might yield these issues and we recommend to stick to the native implementation. multi_tensor_apply is already implemented in upstream PyTorch and more optimizers are being worked on to enable it.