Hey,
Thank you for the advice!
I have tried to reduce the batch size to 4, the checkpoints, the detach() for the batch loss and I have also tried the torch.cuda.amp.GradScaler().
But nothing changed, the error still happened at the same place  which I found quite weird.
I will try to investigate it further, maybe reducing the images size (but for me, if I do that it is more about putting a plaster on the problem instead of solving it).