Llama: Add grammar-based sampling
Here's my understanding of how this works (please someone correct me if I'm getting this wrong).<p>Language models emit tokens one at a time, starting with the prompt that you give them.<p>If you have a conversation with an LLM, effectively you can think of that as you giving it a sequence of tokens, then it generates some, then you generate more and so-on.<p>This grammar trick effectively takes advantage of this by giving you much more finely grained control over the tokens. So you can do things like this:<p><pre><code>    Give me the address of the
    White House as JSON:
    
    {"street": "
</code></pre>
Then the LLM can return:<p><pre><code>    1600 Pennsylvania Ave NW"
</code></pre>
The moment you see that closing double quote, you take over again and inject:<p><pre><code>    ",
    "City": "
</code></pre>
It fills in:<p><pre><code>    Washington, DC"
</code></pre>
And so on.<p>But because this is all based on a grammar, you can do way more with it than just JSON.<p>I saw a brilliant suggestion relating to this on Twitter a while ago:<p>> @OpenAI should add an API argument allowing passing up a deterministic context free grammar.<p>> [...]<p>> While I think DCFL is what you want here in the short term, the really best thing is passing up a small WASM binary that simply <i>is</i> the sampler.<p>> Allow a user to pass up a few KB of WASM binary and give it a few megabytes of RAM to run. Would enable next level LLM superpowers.<p><a href="https://twitter.com/grantslatton/status/1637692033115762688" rel="nofollow noreferrer">https://twitter.com/grantslatton/status/1637692033115762688</a>