First of all, thank you for your answer!
I had a look at it because I was also worried that the reshape would mess the data up but it doesn’t seem to be the case. I removed the .view(-1, self.num_labels) because it was indeed useless (thanks!) but it doesn’t change the problem.
I also tried the toy example where all the target labels are [0, 0, …, 0] and the model fails as well. It predicts [1, 1, …, 1] for every sample. When I look at the evolution of the loss, I saw that it decreased during the first few steps within the first epoch and then it goes up.

I have no idea where I go wrong.