Hi,
But the fact here is that once you use torch no grad, eval mode and no shuffling the behaviour is deterministic (the same samples must fail the whole time).
You can identify those to check whether they have some issue or not. Try to train removing them. Try to identify the 1st layer inside the text model which is producing this output.
In the end using the same data doesn’t mean you are using the same preprocessing. There are always subtle details which are not written in the paper.