Title: Outlines is a Python library that focuses on text generation with large language models. Brandon and I are not LLM experts and started the project a few months ago because we wanted to understand better how the generation process works. Our original background is probabilistic, relational and symbolic programming.<p>Recently we came up with a fast way to generate text that matches a regex (<a href="https:&#x2F;&#x2F;blog.normalcomputing.ai&#x2F;posts&#x2F;2023-07-27-regex-guided-generation&#x2F;regex-guided-generation.html" rel="nofollow noreferrer">https:&#x2F;&#x2F;blog.normalcomputing.ai&#x2F;posts&#x2F;2023-07-27-regex-guide...</a>). The basic idea is simple: regular expressions have an equivalent Deterministic-Finite Automaton (DFA) representation. We can transform this DFA into a generative model: in each state we get a list of symbols which correspond to completions that partially match the regular expression. We mask the other symbols in the logits returned by a large language model, sample a new symbol and move to the next state. The subtelty is that language models work with tokens, not symbols, so we derive a new FSM whose alphabet is the model&#x27;s vocabulary. We can do this in only one pass over the vocabulary.<p>Generating the token masks thus only requires a dictionary lookup at each state. Our method blows other libraries like Microsoft&#x27;s guidance out of the water.<p>From there it was only a small leap to be able to generate text that follows a JSON schema (<a href="https:&#x2F;&#x2F;json-schema.org&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;json-schema.org&#x2F;</a>), or is parseable into a Pydantic model (<a href="https:&#x2F;&#x2F;docs.pydantic.dev&#x2F;latest&#x2F;usage&#x2F;models&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;docs.pydantic.dev&#x2F;latest&#x2F;usage&#x2F;models&#x2F;</a>). The method works with union types, optional types, nested schemas, arrays, everything. It is guaranteed that the output is parseable.<p>I think it&#x27;s cool, and I&#x27;ve spent a lot of time watching even tiny models output valid JSON over the weekend. Hope you will too.<p>I look forward to feedback, bug reports, feature requests and discussions!<p>Edit: Link to our pre-print explaining the method and how this can be extended to generate text that follows a Context-Free Grammar <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2307.09702" rel="nofollow noreferrer">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2307.09702</a>
Upvote: