Hi, instead of Bert, you may be interested in Longformer which has a pretrained weights on seq. length of 4096

  
      
      huggingface.co
  
  
    

Longformer — transformers 3.4.0 documentation 17