Hi @eirikdahlen,
Wordpiece and tf-idf are different concepts. Wordpiece takes a peice of text and converts it into a sequence of tokens. tf-idf on the other hand, takes a sequence of tokens and converts them into a vector (a fixed size list of numbers).
Most implementations of tf-idf take raw text as input and do the tokenization step implicitly, which is why it "doesn’t feel " like tf-idf takes tokens as input, but it does.
In other words, tf-idf has nothing to do with truncation strategies.
That still leaves you with your original problem, you have 28K tokens in a document but can only input 512, that’s a lot of data loss…
The strategy to use here depends on your end goals with the model. Since the data comes in the form tweets and is then aggregated, you might benefit from giving up on the aggregation of the tweets and instead inputting individual tweets.
Can you share more about what you are trying to achieve ?
Tal