I agree with PereLluis13.
Were you thinking of using a pre-trained transformer, or training from scratch?
Are you sure you need a transformer at all?   What are you hoping it will do with the text?   Have you tried regular expressions and a rule-based system?