Retentive Network: A Successor to Transformer for Large Language Models
I only spent a few minutes skimming thr paper, but:<p>1) there are a lot of papers claiming to be the successor to the Transformer, and not all of them are cited; e.g., the MetaFormer is missing 
<a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2111.11418" rel="nofollow noreferrer">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2111.11418</a>.
Another candidate that wasn&#x27;t compares against (or at least argued why it wouldn&#x27;t make sense to compare against) are the Hopfield Networks
<a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2008.02217" rel="nofollow noreferrer">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2008.02217</a>.<p>So until a more solid Related Work section is written (their section is actually called &quot;Relation to and Differences from Previous Methods&quot;) I reserve the right to be skeptical whether their model is the &quot;best&quot; successor to the Transformer.<p>2) they say in the abstract &quot;We theoretically derive the connection between recurrence and attention&quot; but I couldn&#x27;t find a longer theorem-proof section. So either this is done only in a cursory manner, or the proof is very easy.
Recurrence and attention have been around for a long time as concepts, so surely there are already proofs in similar contexts of this fact (I am not working in this particular area of Machine Learning, so I don&#x27;t know the SOTA by heart, but I strongly suspect that these aspects have been discusses previously; thr Hopfield Network paper I linked to unearthes some theoretical facts about attention, for example).<p>So -based on my very cursory reading- this paper seems like an interesting approach, but I do see some holes in thr execution. Time will tell whether Rentetive Network will become mainstream or not.<p>Ok, this was my five minute review of the paper. Now I have to urgently return to completing my actual reviews for NeurIPS, haha.