Answers.Body: <p>There are no guarantees that an LLM used directly will output with any specific constraints.</p>
  <p>You can add to prompts, fine-tuning or other customisation to request a specific output, and that often helps, but does not <em>guarantee</em> output will conform to what you requested. At best you can get a certain level of reliablility e.g. 90% accurate or 99% accurate. How accurate you can get depends on the complexity of the task you have stated in the prompt.</p>
  <p>I have found that ChatGPT v4 (and I assume others at the same level) will output code snippets quite reliably, with either Python or JSON structure, and using markdown formatting, when this is specified in setting up an Assistant, a GPT or in a system prompt.</p>
  <p>A simplified example of a system prompt that might do this:</p>
  <blockquote>
  <p>You are a sentiment analysis tool for user reviews on a shopping site. For each user prompt, decide whether it is &quot;negative&quot;, &quot;neutral&quot; or &quot;positive&quot;. Then output a JSON snippet with key of &quot;sentiment&quot; and the decision as the value.</p>
  </blockquote>
  <p>Example user input:</p>
  <blockquote>
  <p>This television had a scratch on the screen when it was delivered.</p>
  </blockquote>
  <p>Example LLM output:</p>
  <blockquote>
  <p>A scratched screen is a sign of fault that may have occurred in manufacture or during delivery. Although the customer did not express any emotions in their post, they are likely to be unhappy with this defect.</p>
  <pre><code>{
   &quot;sentiment&quot;: &quot;negative&quot;
  }
  </code></pre>
  </blockquote>
  <p>In addition, it usually helps with accuracy of processing if you allow the chat session to add preamble to this, as per my example. I.e. do not demand, in the prompt, to only receive a final processed result. That is because the LLM does not &quot;think&quot; about the text output, it simply outputs text. The coherency of the text can be used to emulate some forms of processing.</p>
  <p>This is a common misunderstanding when using services like ChatGPT, that many people seem to assume that posting a question causes the AI to &quot;think&quot; then return an answer. It doesn't do that. So if you want any kind of logical analysis of your input, it can help to have the LLM work through the issue - a bit like thinking out loud - before it outputs structured data.</p>
  <p>To get your structured data:</p>
  <ul>
  <li>In the web UI, code snippets with markdown will have a <em>copy</em> icon, and you can click it to get a copy of the code block to paste into something else.</li>
  <li>Via the API, you will need to write code that can look for the markdown block, and extract it and parse. Regular expressions are good for this, and available in most programming languages.</li>
  </ul>
  <p>In both cases, you need to allow for the times when the LLM doesn't output what you want. You can retry, fail with an error, or whatever is appropriate for your use case.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-30T12:28:35.553Z
Answers.Text: Note that this answer was posted before ChatGPT o1 models that do "think" before answering. Under the hood in those models, something very similar to the chain-of-thought suggested in this answer is going on. I.e. the bot is generating some reasoning/explaining text, but hiding it
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-09T10:53:20.670Z
Answers.LastActivityDate: 2024-02-09T11:12:05.593Z
Answers.LastEditDate: 2024-02-09T11:12:05.593Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>I agree with @Neil Slater -- it's never guaranteed because inherently the LLM are a stochastic model, not deterministic.</p>
  <p>Given the need for JSON output, <code>OpenAI</code> heeded to the call of the community and made it easy to <strong>increase</strong> your likelihood of generating a JSON output even with the cheaper <code>gpt-3.5-turbo</code> model.</p>
  <p>Here's a code snippet from <a href="https://platform.openai.com/docs/guides/text-generation/json-mode" rel="nofollow noreferrer">their documentation</a>:</p>
  <pre><code>from openai import OpenAI
  client = OpenAI()
  
  response = client.chat.completions.create(
    model=&quot;gpt-3.5-turbo-0125&quot;,
    response_format={ &quot;type&quot;: &quot;json_object&quot; },
    messages=[
      {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: &quot;You are a helpful assistant designed to output JSON.&quot;},
      {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Who won the world series in 2020?&quot;}
    ]
  )
  print(response.choices[0].message.content)
  </code></pre>
  <p>A tip I would offer is to create a simple test in which you pass a set of typical inputs and have that occur over 20 iterations. Then you examine the accuracy of the output.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-10T10:53:13.527Z
Answers.LastActivityDate: 2024-02-10T10:53:13.527Z
Answers.UserDisplayName: Ari
Answers.Body: <p>I'd say it's possible, although it depends on the model you're using.</p>
  <p>If you're using llama.cpp (or its python bindings), you can use their <a href="https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md" rel="nofollow noreferrer">grammars</a> support to constrain the output</p>
  <blockquote>
  <p>GBNF (GGML BNF) is a format for defining formal grammars to constrain model outputs in llama.cpp. For example, you can use it to force the model to generate valid JSON, or speak only in emojis.</p>
  </blockquote>
  <p>For a more generic approach, and if you're using Python, take a look at <a href="https://github.com/guidance-ai/guidance" rel="nofollow noreferrer">guidance</a>.</p>
  <blockquote>
  <p>guidance is a programming paradigm that offers superior control and efficiency compared to conventional prompting and chaining. It allows users to constrain generation (e.g. with regex and CFGs) as well as to interleave control (conditional, loops) and generation seamlessly.</p>
  </blockquote>
  <p>Here's a snippet of what a guidance configuration would look like (borrowed from their docs):</p>
  <pre><code>@guidance
  def character_maker(lm, id, description, valid_weapons):
      lm += f&quot;&quot;&quot;\
      The following is a character profile for an RPG game in JSON format.
      ```json
      {{
          &quot;id&quot;: &quot;{id}&quot;,
          &quot;description&quot;: &quot;{description}&quot;,
          &quot;name&quot;: &quot;{gen('name', stop='&quot;')}&quot;,
          &quot;age&quot;: {gen('age', regex='[0-9]+', stop=',')},
          &quot;armor&quot;: &quot;{select(options=['leather', 'chainmail', 'plate'], name='armor')}&quot;,
          &quot;weapon&quot;: &quot;{select(options=valid_weapons, name='weapon')}&quot;,
          &quot;class&quot;: &quot;{gen('class', stop='&quot;')}&quot;,
          &quot;mantra&quot;: &quot;{gen('mantra', stop='&quot;')}&quot;,
          &quot;strength&quot;: {gen('strength', regex='[0-9]+', stop=',')},
          &quot;items&quot;: [&quot;{gen('item', list_append=True, stop='&quot;')}&quot;, &quot;{gen('item', list_append=True, stop='&quot;')}&quot;, &quot;{gen('item', list_append=True, stop='&quot;')}&quot;]
      }}```&quot;&quot;&quot;
      return lm
  
  
  lm = llama2 + character_maker(1, 'A nimble fighter', ['axe', 'sword', 'bow'])
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-10T15:36:01.960Z
Answers.LastActivityDate: 2024-02-10T15:36:01.960Z
Answers.UserDisplayName: Vlad Iliescu
Body: <p>What prompt (or other technique) should I use with an LLM so that</p>
  <ol>
  <li>The result is guaranteed to be reliably parseable as a list of values (e.g. a Python list of strings)</li>
  <li>LLM would understand that a variable list of values is expected during generation</li>
  </ol>
  <h1>EDIT: Chosen solution</h1>
  <p>What I personally ended up using is BAML. Works with all LLMs and guarantees correct output.</p>
  <p><a href="https://github.com/BoundaryML/baml" rel="nofollow noreferrer">https://github.com/BoundaryML/baml</a></p>
  <p><code>pip install baml-py</code></p>
  <p>Example:</p>
  <p><a href="https://www.promptfiddle.com/BAML-Examples-7xLvl" rel="nofollow noreferrer">https://www.promptfiddle.com/BAML-Examples-7xLvl</a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-09T02:39:34.557Z
Comments.Text: Related https://ai.stackexchange.com/questions/42127/how-to-prompt-open-ai-for-a-variable-list-length?rq=1
Comments.UserDisplayName: Elijas Dapšauskas
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-10T16:16:03.560Z
Comments.Text: This question is off-topic here. This is about specific tool or tools and how to use them in practice. It has little to do with any scientific theory because there aren't real scientific theories about prompt engineering. Just rules of thumb, which often don't work. This is not science but opinions. Still, this question may be useful, but it's off-topic here. It would be on-topic at https://genai.stackexchange.com/ I think.
Comments.UserDisplayName: nbro
Tags: <large-language-models><chatgpt><prompt>
Title: How to force LLM (like OpenAI ChatGPT) to output a variable list of values?
UserDisplayName: Elijas Dapšauskas
ThreadText: # How to force LLM (like OpenAI ChatGPT) to output a variable list of values?
  
  (Asked by: username_0 on 2024-02-09)
  
  What prompt (or other technique) should I use with an LLM so that
  
  1. The result is guaranteed to be reliably parseable as a list of values (e.g. a Python list of strings)
  2. LLM would understand that a variable list of values is expected during generation
  
  # EDIT: Chosen solution
  
  What I personally ended up using is BAML. Works with all LLMs and guarantees correct output.
  
  <https://github.com/BoundaryML/baml>
  
  `pip install baml-py`
  
  Example:
  
  <https://www.promptfiddle.com/BAML-Examples-7xLvl>
  
  **Tags:** large-language-models, chatgpt, prompt
  > **Comments:**
  > **username_0**: Related https://ai.stackexchange.com/questions/42127/how-to-prompt-open-ai-for-a-variable-list-length?rq=1  
  > **username_1**: This question is off-topic here. This is about specific tool or tools and how to use them in practice. It has little to do with any scientific theory because there aren't real scientific theories about prompt engineering. Just rules of thumb, which often don't work. This is not science but opinions. Still, this question may be useful, but it's off-topic here. It would be on-topic at https://genai.stackexchange.com/ I think.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2024-02-09. Score: 2
  
  There are no guarantees that an LLM used directly will output with any specific constraints.
  You can add to prompts, fine-tuning or other customisation to request a specific output, and that often helps, but does not guarantee output will conform to what you requested. At best you can get a certain level of reliablility e.g. 90% accurate or 99% accurate. How accurate you can get depends on the complexity of the task you have stated in the prompt.
  I have found that ChatGPT v4 (and I assume others at the same level) will output code snippets quite reliably, with either Python or JSON structure, and using markdown formatting, when this is specified in setting up an Assistant, a GPT or in a system prompt.
  A simplified example of a system prompt that might do this:
  
  You are a sentiment analysis tool for user reviews on a shopping site. For each user prompt, decide whether it is "negative", "neutral" or "positive". Then output a JSON snippet with key of "sentiment" and the decision as the value.
  
  Example user input:
  
  This television had a scratch on the screen when it was delivered.
  
  Example LLM output:
  
  A scratched screen is a sign of fault that may have occurred in manufacture or during delivery. Although the customer did not express any emotions in their post, they are likely to be unhappy with this defect.
  {
   "sentiment": "negative"
  }
  
  
  In addition, it usually helps with accuracy of processing if you allow the chat session to add preamble to this, as per my example. I.e. do not demand, in the prompt, to only receive a final processed result. That is because the LLM does not "think" about the text output, it simply outputs text. The coherency of the text can be used to emulate some forms of processing.
  This is a common misunderstanding when using services like ChatGPT, that many people seem to assume that posting a question causes the AI to "think" then return an answer. It doesn't do that. So if you want any kind of logical analysis of your input, it can help to have the LLM work through the issue - a bit like thinking out loud - before it outputs structured data.
  To get your structured data:
  
  In the web UI, code snippets with markdown will have a copy icon, and you can click it to get a copy of the code block to paste into something else.
  Via the API, you will need to write code that can look for the markdown block, and extract it and parse. Regular expressions are good for this, and available in most programming languages.
  
  In both cases, you need to allow for the times when the LLM doesn't output what you want. You can retry, fail with an error, or whatever is appropriate for your use case.
  
  > **Comments:**
  > **username_2**: Note that this answer was posted before ChatGPT o1 models that do "think" before answering. Under the hood in those models, something very similar to the chain-of-thought suggested in this answer is going on. I.e. the bot is generating some reasoning/explaining text, but hiding it  
  
  ---
  
  ### Answer by username_3 on 2024-02-10. Score: 2
  
  I agree with @Neil Slater -- it's never guaranteed because inherently the LLM are a stochastic model, not deterministic.
  Given the need for JSON output, OpenAI heeded to the call of the community and made it easy to increase your likelihood of generating a JSON output even with the cheaper gpt-3.5-turbo model.
  Here's a code snippet from their documentation:
  from openai import OpenAI
  client = OpenAI()
  
  response = client.chat.completions.create(
    model="gpt-3.5-turbo-0125",
    response_format={ "type": "json_object" },
    messages=[
      {"role": "system", "content": "You are a helpful assistant designed to output JSON."},
      {"role": "user", "content": "Who won the world series in 2020?"}
    ]
  )
  print(response.choices[0].message.content)
  
  A tip I would offer is to create a simple test in which you pass a set of typical inputs and have that occur over 20 iterations. Then you examine the accuracy of the output.
  
  
  ---
  
  ### Answer by username_4 on 2024-02-10. Score: 0
  
  I'd say it's possible, although it depends on the model you're using.
  If you're using llama.cpp (or its python bindings), you can use their grammars support to constrain the output
  
  GBNF (GGML BNF) is a format for defining formal grammars to constrain model outputs in llama.cpp. For example, you can use it to force the model to generate valid JSON, or speak only in emojis.
  
  For a more generic approach, and if you're using Python, take a look at guidance.
  
  guidance is a programming paradigm that offers superior control and efficiency compared to conventional prompting and chaining. It allows users to constrain generation (e.g. with regex and CFGs) as well as to interleave control (conditional, loops) and generation seamlessly.
  
  Here's a snippet of what a guidance configuration would look like (borrowed from their docs):
  @guidance
  def character_maker(lm, id, description, valid_weapons):
      lm += f"""\
      The following is a character profile for an RPG game in JSON format.
      ```json
      {{
          "id": "{id}",
          "description": "{description}",
          "name": "{gen('name', stop='"')}",
          "age": {gen('age', regex='[0-9]+', stop=',')},
          "armor": "{select(options=['leather', 'chainmail', 'plate'], name='armor')}",
          "weapon": "{select(options=valid_weapons, name='weapon')}",
          "class": "{gen('class', stop='"')}",
          "mantra": "{gen('mantra', stop='"')}",
          "strength": {gen('strength', regex='[0-9]+', stop=',')},
          "items": ["{gen('item', list_append=True, stop='"')}", "{gen('item', list_append=True, stop='"')}", "{gen('item', list_append=True, stop='"')}"]
      }}```"""
      return lm
  
  
  lm = llama2 + character_maker(1, 'A nimble fighter', ['axe', 'sword', 'bow'])
  
  
  ---
  
  