Solved by AndreaSottana in post #2 
              
              
                Hi Mahdi, 
My guess is that with enough training data, the transformer model, and in particular its attention heads, will learn to recognise what they should be paying most attention to, i.e. which parts of the text are more important for the classification, and which parts are not that relevant, so…