hey @carschno, how long are your documents (on average) and what kind of performance do you get with the current approach (i.e. tokenizing + truncating the whole document)?
if performance is a problem then since you’re doing text classification, you could try chunking each document into smaller passages with a sliding window (see this tutorial 112 for details on how that’s done), and then aggregate the [CLS] representations for each window in a manner similar to this 23 paper.
it’s not the most memory efficient strategy, but if your documents are not super long it might be a viable alternative to simple truncation