LONGNET


1B sequence length
![](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2Fd27a24d7-be6f-48ce-a27e-bc5a5a168c1c%2FUntitled.png?table=block&id=7c4d0da8-b2bd-4baf-ad2d-21edbd1014a0&cache=v2)
[Dilated Attention](https://texonom.com/dilated-attention-d9000e3460f547bca3f15ac9b8d7e36c) 
![](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2Fdf83ede8-c08e-498f-b494-c2adc9226fbd%2FUntitled.png?table=block&id=5b951e2a-d9ba-47d3-9e65-95358c82c58c&cache=v2)



> [Microsoft’s LongNet Scales Transformer to One Billion Tokens](https://medium.com/syncedreview/microsofts-longnet-scales-transformer-to-one-billion-tokens-af02ff657d87)

> [Untitled](https://arxiv.org/pdf/2307.02486.pdf)





