More than the labels, I’d get the pooler output of each chunk and then average them before using the classifier. This will require you to change the code of BertForSequenceClassification a little bit.