Towards A Token-Free Future In NLP (2022)
I&#x27;m a native Hebrew speaker, where we have wild word morphology, and also worked on NLP at a large bank where we trained models to anlyze Bloomberg chats (that are almost english but behave like a DSL).<p>In both cases, &quot;token-free&quot; has been the way to go for extractive tasks like entity recognition.<p>It&#x27;s unfortunately a technique that is off the beaten path, so a lot of extra engineering effort needs to be paid down. For example, when dealing with spans but embedding the text at the byte level, one needs to account for multi-byte codepoints that can throw off the alignment between &quot;encoded&quot; and raw bytes.<p>As the article alludes, the increase in sequence length can be very expensive. In extractive tasks, I&#x27;ve found it effective to use much lighter models with limited  (but large) context length like ByteNet[0].<p>Thinking out loud, as always, there is no one-tool for everything. Often the summarization&#x2F;few-shot capabilities of an off-the-shelf transformer are so far ahead that it&#x27;s not worth building a model from ±scratch to solve the subtlties of tokenization. Other times, you don&#x27;t need the unbounded context or have a simpler task and can forgo the power of off-the-shelf models for the specificity of a token free one.<p>[0] Neural Machine Translation in Linear Time (<a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;1610.10099" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;1610.10099</a>)