Hi @lewtun ,
Thanks for the suggestion, I think the sliding window approach looks very promising indeed.
My documents are body texts from scraped web pages, so the length varies widely, from a few tokens till very long texts.
Current performance of the fine-tuned classifier is around 0.8-0.9 accuracy (2 classes). I’ll have to look deeper into whether this is good enough for my application, and do more data analysis.
Anyway for clarification: i understand the tokenizer behavior I described above is expected and the Bert model is supposed to handle input texts with multiple sentences in a single string well, right?