Thank you for your detailed answer!
Yes, that is my aim and the documents are between 300 and 5000 words and we’re talking about domain specific documents on top of that.
Simply truncating them will probably work well on the shorter ones but the median length is about 2000 words. I could still try it but from what I can gather there is no obvious parts of the documents that I could discard and just truncating at 512 will definitely cut out important details.
I have looked at LongFormer but sadly there isn’t one in my target language (German btw) and I don’t think I’ll be able to train from scratch as my ressources are fairly limited.
Thanks for pointing me to that thread. It does seem promising to first pick the k most important sentences and then use those.