I started an experiment yesterday and it is still running and it never went that far.
If the training finishes I will check the solution.
I guess I will never know why it runs OOM when there are no annotations.
Thank you for the help!