A recent paper by Google show it’s possible to greatly reduce the memory footprint of transformers.
If and when would something like this be implemented in Pytorch?

  
      

      arXiv.org
  

  
    

Self-attention Does Not Need $O(n^2)$ Memory 3

  We present a very simple algorithm for attention that requires $O(1)$ memory
with respect to sequence length and an extension to self-attention that
requires $O(\log n)$ memory. This is in contrast with the frequently stated
belief that...