Yes handling a long context it is an open problem and an active research topic.
E.g. see
  
      

      arXiv.org
  

  
    

$\infty$-former: Infinite Memory Transformer 1

  Transformers struggle when attending to long contexts, since the amount of
computation grows with the context length, and therefore they cannot model
long-term memories effectively. Several variations have been proposed to
alleviate this problem, but...