Help with implementing strides into features for multi-label classifier
As you might know, BERT has a maximum wordpiece token sequence length of 512. 
The SQuAD example actually uses strides to account for this: https://github.com/google-research/bert/issues/27
I want to implement something like what Jacob Devlin described in that post for Kaushal's multi-label BERT classifier: https://medium.com/huggingface/multi-label-text-classification-using-bert-the-mighty-transformer-69714fa3fb3d

What I did was basically just copy and edit a bit the doc_stride functions from the SQuAD example in huggingface into Kaushal's code: https://colab.research.google.com/drive/1aqcIdm2Pn2rvmWHvOSEMUTzw_CNmoseA

However, this is excruciatingly slow, I don't know if it's an issue of stride length?

Also, I'm not sure how to implement the thing about reshaping the combined minibatches and getting predictions from there. 

Anyone who has tried to adapt BERT for longer texts, could you please help?

Thanks

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>@alvin-leong did this make any progress? I am interested in doing something similar for the sequence classifier. <|||||>@alvin-leong @maxzzze , just wondering if there is any progress on this? 
Thanks.<|||||>No 😞 I ended up implementing several methods to grab different spans of length X (hyperparamter) from all over my documents which have lengths > 512.<|||||>Thanks @maxzzze . I am doing the same now, and just concerned about the answers from different spans. You may have right answer with low scores and wrong answer with high scores. Finger cross not. Not sure if scores from different spans is comparable?!<|||||>I modified https://github.com/kyzhouhzau/BERT-NER to implement strides https://github.com/anupamsingh610/bert_ner_stride

Hope it helps :)<|||||>@alvin-leong Hi, I have you made any progress on this issue? I checked your notebook. What is the problem with that? Is just the train very slow?<|||||>Hi, we have implemented BERT binary classifier for longer texts [here](https://github.com/mim-solutions/bert_for_longer_texts).