Thank you so much for your suggestions! I will surely try training with different initial seeds. I was thinking that maybe CUDA cached the models weights of the previous run and somehow used these weights for the next run even though I re-initialized the model. Although I think this is mostly not the case. Thanks again!