I think I might have figured out what the problem is. I think I have a vanishing/exploding gradient issue, because:

The first epoch loss is ever so slightly different from the subsequent ones, and those are all exactly the same.
The model trains when I trim down the model

I didn’t consider this problem, as it works flawlessly in Keras. Afaik, I’m using the exact same hyperparameters.
I’m still very much interested if anyone knows why this happens in PyTorch, but not in Keras!