Unlimiformer: Long-Range Transformers with Unlimited Length Input
1. This is not exact attention, but an approximation of it. Specifically, they use k-nearest neighbors to retrieve the top-k most similar tokens, out of an "unlimited-length input" say of size N, where k << N.<p>2. This idea is quite similar to retrieval transformers and Hopfield networks which have been known and published for several years now. It's not really that novel.<p>3. Due to the preceding points, the title can easily mislead people. It's not really a conventional transformer, and it's not a breakthrough.<p>4. This paper is a preprint and not peer-reviewed.<p>"I generally don't enjoy seeing preprints like this going to the top of Hacker News. This would be a higher quality submission if the paper was peer-reviewed or put into a greater context, like a blog post discussion or something like that."<p>Let me retract this and say something a bit nicer :) 
I personally think there this specific preprint making it to the top of HN is potentially harmful, because of the hype around LLMs, the diverse audience of readers here, and the specific title that implies a claim of "transformer with unlimited context length", when this is misleading. I don't have anything against preprints in general - a lot of work outside of the peer-review process ends up being very impactful.