Dilated Attention


![](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2Fe84a0f0b-ccda-469a-978d-c0ef60558a4b%2FUntitled.png?table=block&id=6014ac5c-6ab5-4d96-affe-7f504b2c4355&cache=v2)




> [Microsoft’s LongNet Scales Transformer to One Billion Tokens](https://medium.com/syncedreview/microsofts-longnet-scales-transformer-to-one-billion-tokens-af02ff657d87)





