Thanks for your reply. I am looking at nvidia-smi, also.
Sorry, I have little experience. I have verified that amp does work, although it does not provide any memory gain. Only the checkpoint helps to somehow reduce memory consumption. I had to delete more than 40% of the dataset already because the sequence is long (64 tokens) and batch size 16.
Unfortunately, the checkpoint will not help train the layers of the transformer.
If you can tell me what else can be done, besides the checkpoint and amp, I would be very grateful.
I have a feeling that I am doing something very wrong. 
I read Performance Tuning Guide — PyTorch Tutorials 1.9.1+cu102 documentation and Training With Mixed Precision :: NVIDIA Deep Learning Performance Documentation 2