Chunking Long Documents for Classification Tasks
## 🚀 Feature

A way to process long documents for downstream classification tasks. One approach is to chunk long sequences with a specific stride similar to what is done in the run_squad example.

## Motivation

For classification tasks using datasets that are on average longer than 512 tokens, I believe it would improve performance.

## Additional context

https://github.com/google-research/bert/issues/27#issuecomment-435265194
I'm not sure that I understand. As you say, you can see it implemented in the run_squad example. What else would you like? <|||||>Hello Bram,

I mean I want to apply it with a sequence classification task like BertForSequenceClassification, for example, versus what is being done in squad.

I don't think it should be too hard but I'm not exactly sure how a long document that is being chunked gets trained. Do we ignore the fact that these are chunks of the same document and just treat them as independent docs? Or do we do some sort of trick to join the tokens/embeddings with the first chunk? 

How would this be implemented for sequence classification?<|||||>I quickly glared over the `convert_examples_to_features` function, and it seems that given some stride different parts are used as input. So, yes, as far as I can see they are treated as independent docs.

https://github.com/huggingface/transformers/blob/ae50ad91ea2fedb64ecd2e7c8e2d0d4778dc03aa/examples/utils_squad.py#L189-L397

<|||||>isn't there a way to deal with long documents without ignoring the fact that the chunks represent the same doc? 

Maybe something along the lines of https://finetune.indico.io/chunk.html?highlight=long or https://explosion.ai/blog/spacy-pytorch-transformers#batching<|||||>After a first look, I don't see how `spacy-pytorch-transformers` does anything special rather than processing a document sentence-per-sentence. `finetune`'s approach might be what you after (taking the mean over all the slided windows), but as always: "a mean is just a mean", so the question remains how representative it is of the whole document. I am not saying that slicing is _better_ by any means, but averaging can distort "real" values greatly.<|||||>Yeah I see your point. I'm starting to think that maybe trying out chunking with a couple of different strides and maybe at inference time taking a voting approach would be a better option.

In any case, thank you for your feedback!<|||||>I agree that that might be the more efficient approach. No worries, thanks for the interesting question. If you think it's okay the question, please close it so it's easy to keep track of all open issues.<|||||>Hi, just to let you know that there is an option to manage strides in the `encode_plus` method. It handles special tokens and returns the overflowing elements in the `overflowing_tokens` field of the returned dictionary.