@lewtun can I bother you again with a follow-up?
So, I just now realized that it will be very difficult for me to process long documents as BERT’s max input length is 512. So now I’m looking for checkpoints that can process longer input sequences and this is where my question arises…
How much does it matter which task the model was trained for? For example there is a large xlm-roberta model in my target language but it was trained for token-classification. Would I even be able to utilize that?
My understanding is that any task can potentially be usefull as the models do learn things about the language they process but maybe this isn’t as universaly true as I thought?