There are a couple of different issues here:

the illegal memory access should never happen and we are looking into it
gradient overflow is expected as long as it’s not constantly in every iteration and the loss scaler will reduce its scaling factor
a NaN output should not happen and there might be issues in the model architecture or a bug in amp
we recommend to install the nightly binaries, which come with native amp via torch.cuda.amp instead of building apex/amp