Answers.Body: <p>Uncertainty is usually seen under the lens of entropy (aka chaos).</p>
  <p>The higher the entropy, the less predictable the outcome. Indeed, it's a paraphrasis of what you are saying... if the entropy is high, the generations will be very different (you are confused). For example, think about asking an LLM to predict a coin flip... it will assign 0.5 probability of both outcomes, and thus if you generate, you will have &quot;very confused&quot; answers.</p>
  <p>Instead, for events that are certain, theoretically the distribution will be deterministic, because there is no entropy (you have close to no uncertainty in which state London is, thus it's low entropy)</p>
  <p>Given this, the entropy of a distribution is given by <span class="math-container">$\int p(x) \log p(x)$</span>, and given that an LLM is nothing more than a function that predict distribution, specifically factorized as <span class="math-container">$p(x) = \prod p(x_i|x_{j&lt;i})$</span>, you can used a <a href="https://math.stackexchange.com/questions/986330/decomposing-factorized-entropy">factorized entropy</a> to find the overall entropy</p>
  <p>In poor words, at every token generation step, you check the softmax distribution the LLM is predicting, you calculate the entropy, and at the end you sum all of the entropies from the generation... the higher it is, the more uncertain is the LLM about the generation</p>
  <p>A proxy of entropy, is called perplexity, and it's a very well known metric in the LLM community, <a href="https://mbernste.github.io/posts/perplexity/" rel="nofollow noreferrer">here's a nice blogpost about it</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-28T21:08:30.290Z
Answers.Text: Are you saying the confidence level is the entropy or perplexity of the answer ChatGPT puts out? If so, do you have any published reference to confirm this claim?
Answers.UserDisplayName: Hans
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-28T21:23:06.940Z
Answers.Text: @Hans arxiv is public for everybody... https://arxiv.org/html/2404.15993v1 _"Existing uncertainty estimation approaches for LLMs usually involve designing uncertainty metrics for their outputs. For black-box LLMs, these metrics are computed by examining aspects like the generated outputs’ consistency, similarity, entropy, and other relevant characteristics"_
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-28T21:38:19.047Z
Answers.Text: This paper proposes a measure for uncertainty. What I am asking is how ChatGPT constructs its confidence level for each answer. Is their method the same or close to what ChatGPT uses?
Answers.UserDisplayName: Hans
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-29T00:46:26.127Z
Answers.Text: @Hans chatGPT is a proprietary system, they do not disclose such information, I'm answering to the _ What is the mathematical description of it_
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-29T00:47:01.413Z
Answers.Text: @Hans not that there are not many other formalization of uncertainty without having a bayesian neural network, which chatgpt is not made of definitely
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-29T19:22:09.080Z
Answers.Text: I understood that ChatGPT did not disclose its proprietary information. But not all OpenAI information is secret. So I was banking on the possibility that some of the relevant information was not proprietary. I know all the relevant mathematics. So basically you are saying entropy/perplexity is the best guess you have for formulating uncertainty/confidence, right? But regarding "not that there are not many other... without..., which ... is not ... ", is there a typo here, what with 4 negations? Can you put it in the form of an affirmative assertion just so as to leave no doubt?
Answers.UserDisplayName: Hans
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-30T00:28:09.213Z
Answers.Text: @Hans i’m saying is the cheapest and most enstablished method, and wrt the second question, the naive form would be “there are very few other methods as reliable and grounded as this one out there”
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-28T20:09:53.757Z
Answers.LastActivityDate: 2024-04-28T20:09:53.757Z
Answers.UserDisplayName: Alberto
Answers.Body: <p>ChatGPT does not output confidence estimates for its responses.
  You can ask the LLM to verbalize a confidence-value for its response, but this is terribly unreliable.</p>
  <p>A better approach is to use OpenAI's API which offers <a href="https://cookbook.openai.com/examples/using_logprobs" rel="nofollow noreferrer">logprobs</a>. Using these, you can  compute the probability of the response according to the model (remember LLMs are fundamentally probabilistic models). However this approach is still not particularly reliable -- logprobs tend to vary based on how exactly the response happens to be phrased and do not reflect unknown unknowns (sometimes called <a href="https://www.cs.ox.ac.uk/people/yarin.gal/website/thesis/1_introduction.pdf" rel="nofollow noreferrer">epistemic uncertainty</a> in the ML literature). Moreover, other LLM APIs like Anthropic do not offer logprobs.</p>
  <p>I've built a <a href="https://tlm.cleanlab.ai/" rel="nofollow noreferrer">tool</a> to more reliably estimate uncertainty in responses from any LLM, including OpenAI or Anthropic models. My approach accounts for both <a href="https://www.cs.ox.ac.uk/people/yarin.gal/website/thesis/1_introduction.pdf" rel="nofollow noreferrer">aleatoric and epistemic uncertainty</a> in the model, via a combination of LLM self-reflection, probabilistic prediction, and measuring the semantic consistency between alternative responses the LLM finds plausible.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-25T06:25:37.047Z
Answers.LastActivityDate: 2024-11-25T06:25:37.047Z
Answers.UserDisplayName: Jonas Mueller
Body: <p>How does ChatGPT assign confidence level to its answer? What is the mathematical description of it?</p>
  <p>Here is my guess. It produces many answers from auto-regression. It already has a probability assigned to each of its outputs. It can just output this probability or some function of this probability for the chosen output. Is this correct?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-29T11:24:37.587Z
Comments.Text: Where is it written that ChatGPT assigns "confidence levels" to its answers? I've just used ChatGPT again and I don't see that. So, either you're mixing concepts here, or you're asking about "how ChatGPT decides what's the most likely answer".
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-30T03:45:41.163Z
Comments.Text: @nbro: Hmm, you may well be right. Someone was saying it gave confidence level. I thought he might be talking about the paid version...
Comments.UserDisplayName: Hans
Tags: <natural-language-processing><chatgpt><question-answering>
Title: How does ChatGPT assign confidence level?
UserDisplayName: Hans
ThreadText: # How does ChatGPT assign confidence level?
  
  (Asked by: username_0 on 2024-04-28)
  
  How does ChatGPT assign confidence level to its answer? What is the mathematical description of it?
  
  Here is my guess. It produces many answers from auto-regression. It already has a probability assigned to each of its outputs. It can just output this probability or some function of this probability for the chosen output. Is this correct?
  
  **Tags:** natural-language-processing, chatgpt, question-answering
  
  > **Comments:**
  > **username_1**: Where is it written that ChatGPT assigns "confidence levels" to its answers? I've just used ChatGPT again and I don't see that. So, either you're mixing concepts here, or you're asking about "how ChatGPT decides what's the most likely answer".  
  > **username_0**: @username_1: Hmm, you may well be right. Someone was saying it gave confidence level. I thought he might be talking about the paid version...  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2024-04-28. Score: 1
  
  Uncertainty is usually seen under the lens of entropy (aka chaos).
  
  The higher the entropy, the less predictable the outcome. Indeed, it's a paraphrasis of what you are saying... if the entropy is high, the generations will be very different (you are confused). For example, think about asking an LLM to predict a coin flip... it will assign 0.5 probability of both outcomes, and thus if you generate, you will have "very confused" answers.
  
  Instead, for events that are certain, theoretically the distribution will be deterministic, because there is no entropy (you have close to no uncertainty in which state London is, thus it's low entropy)
  
  Given this, the entropy of a distribution is given by $\int p(x) \log p(x)$, and given that an LLM is nothing more than a function that predict distribution, specifically factorized as $p(x) = \prod p(x\_i|x\_{j<i})$, you can used a [factorized entropy](https://math.stackexchange.com/questions/986330/decomposing-factorized-entropy) to find the overall entropy
  
  In poor words, at every token generation step, you check the softmax distribution the LLM is predicting, you calculate the entropy, and at the end you sum all of the entropies from the generation... the higher it is, the more uncertain is the LLM about the generation
  
  A proxy of entropy, is called perplexity, and it's a very well known metric in the LLM community, [here's a nice blogpost about it](https://mbernste.github.io/posts/perplexity/)
  
  > **Comments:**
  > **username_0**: Are you saying the confidence level is the entropy or perplexity of the answer ChatGPT puts out? If so, do you have any published reference to confirm this claim?  
  > **username_2**: @username_0 arxiv is public for everybody... https://arxiv.org/html/2404.15993v1 _"Existing uncertainty estimation approaches for LLMs usually involve designing uncertainty metrics for their outputs. For black-box LLMs, these metrics are computed by examining aspects like the generated outputs’ consistency, similarity, entropy, and other relevant characteristics"_  
  > **username_0**: This paper proposes a measure for uncertainty. What I am asking is how ChatGPT constructs its confidence level for each answer. Is their method the same or close to what ChatGPT uses?  
  > **username_2**: @username_0 chatGPT is a proprietary system, they do not disclose such information, I'm answering to the _ What is the mathematical description of it_  
  > **username_2**: @username_0 not that there are not many other formalization of uncertainty without having a bayesian neural network, which chatgpt is not made of definitely  
  > **username_0**: I understood that ChatGPT did not disclose its proprietary information. But not all OpenAI information is secret. So I was banking on the possibility that some of the relevant information was not proprietary. I know all the relevant mathematics. So basically you are saying entropy/perplexity is the best guess you have for formulating uncertainty/confidence, right? But regarding "not that there are not many other... without..., which ... is not ... ", is there a typo here, what with 4 negations? Can you put it in the form of an affirmative assertion just so as to leave no doubt?  
  > **username_2**: @username_0 i’m saying is the cheapest and most enstablished method, and wrt the second question, the naive form would be “there are very few other methods as reliable and grounded as this one out there”  
  
  ---
  
  ### Answer by username_3 on 2024-11-25. Score: 0
  
  ChatGPT does not output confidence estimates for its responses.
  You can ask the LLM to verbalize a confidence-value for its response, but this is terribly unreliable.
  
  A better approach is to use OpenAI's API which offers [logprobs](https://cookbook.openai.com/examples/using_logprobs). Using these, you can compute the probability of the response according to the model (remember LLMs are fundamentally probabilistic models). However this approach is still not particularly reliable -- logprobs tend to vary based on how exactly the response happens to be phrased and do not reflect unknown unknowns (sometimes called [epistemic uncertainty](https://www.cs.ox.ac.uk/people/yarin.gal/website/thesis/1_introduction.pdf) in the ML literature). Moreover, other LLM APIs like Anthropic do not offer logprobs.
  
  I've built a [tool](https://tlm.cleanlab.ai/) to more reliably estimate uncertainty in responses from any LLM, including OpenAI or Anthropic models. My approach accounts for both [aleatoric and epistemic uncertainty](https://www.cs.ox.ac.uk/people/yarin.gal/website/thesis/1_introduction.pdf) in the model, via a combination of LLM self-reflection, probabilistic prediction, and measuring the semantic consistency between alternative responses the LLM finds plausible.
  
  
  ---
  
  