Hello,
My study partner and I are doing research on Twitter data for our Master’s Thesis. We have collected a dataset of tweets, aggregated on user-level. Each entry in the dataset corresponds to a user, and each user has a text document and a classification label. These text documents consist of several tweets from one user in one long string (not sentences or word-tokens, just one long string).
We use BERTForSequenceClassification for this, but have a problem with truncation. The average number of tokens for these text documents is 28.000(!), and with a sequence length of 512, there are obviously a huge amount of tokens that are dropped.
Our question is the truncation strategy. We set parameter truncation=True when initializing the BertTokenizer. Will the truncation just keep the 512 first tokens with this strategy, or will it keep the 512 tokens with the highest weights/WordPiece “score”. In other words, if the tokenizer strategy was e.g. TF-IDF, would the truncation process keep the top-512 TF-IDF scoring tokens or just the 512 first tokens. We don’t fully understand how WordPiece gives weights/scores to tokens and if these “scores” are used in truncation.