benjs:

Maybe the model is too large for my GPU, but it only needs 500MB in the beginning + 450MB when doing predicition?. How can it be, that it does not fit on my GPU?


The intermediate tensors, which are needed to calculate the gradients, the gradients themselves, as well as the CUDA context will all use memory on your device, which could explain the OOM issue.
You could try to decrease the batch size, if possible, and check the memory usage for a smaller batch size. Alternatively, you could use torch.utils.checkpoint to trade compute for memory.
Also, you could try to run your code in e.g. Colab, which would give you a GPU with more memory for a limited time, and check the memory usage there for your current code.