The code logic is correct as seen in the amp example 7, so you might need to reduce the batch size further.
Is the training running fine without gradient accumulation and if so how large is the memory usage?