I agree that SMOTE does not generally work for text data. But a few tricks might make it work. For example, instead of randomly mixing different parts of multiple texts, you may concatenate two pieces of text from different classes, and expect your model to output the same probability for those classes. Nothing is for sure, the best option is to try!


I assume you are using a pre-trained BERT. BERT is just a stack of encoders, so there is no decoder! What you are going to do is a Masked Language Modelling, which you can refer to Huggingface documentation for detailed information. In short, you mask some tokens and ask the model to predict those tokens.


About your last point, I agree that this training regime might introduce bias in your model. You may want to preprocess your data to reduce this bias in the dataset. As a side note, I think there are better options than BERT for this task. For instance, you can extract tf-idf features from your documents and use a simple SVM classifier, which is robust to word positions.