Answers.Body: <p>I don't know if <span class="math-container">$N(X)$</span> has a name or has any applicability in AI, but I can comment on how this function varies as the <span class="math-container">$H(X)$</span> based on your equation</p>
  <p><span class="math-container">$$N(X) = \dfrac{1}{2^{H(X)}}$$</span></p>
  <p>which looks correct to me (just apply the <span class="math-container">$\log_2$</span> to both sides).</p>
  <p>In the case of a Bernoulli random variable (which is a categorical r.v. that can take 2 values, <span class="math-container">$0$</span> or <span class="math-container">$1$</span>, which is a special case of your categorical r.v., if you set <span class="math-container">$k=2$</span>), then this is the relationship between the probability that this random variable <span class="math-container">$X = 1$</span> and the entropy of this r.v.</p>
  <p><a href="https://i.sstatic.net/lXNSJ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/lXNSJ.png" alt="enter image description here" /></a></p>
  <p>So, the entropy is <span class="math-container">$1$</span> when the probability is <span class="math-container">$0.5$</span> and decreases as the probability tends to <span class="math-container">$0$</span> or <span class="math-container">$1$</span>, which makes sense, because the entropy quantifies the uncertainty about the r.v. Here, the entropy is computed in bits because we use the logarithm <span class="math-container">$\log_2$</span>.</p>
  <p>So, for <span class="math-container">$k=2$</span> (in your example), then, if <span class="math-container">$H(X) = 1$</span>,</p>
  <p><span class="math-container">$$N(X) = \dfrac{1}{2^{H(X)}} = \frac{1}{2} = \frac{1}{2}^{\frac{1}{2}} * \frac{1}{2}^{\frac{1}{2}}$$</span></p>
  <p>which is equal to <span class="math-container">$P(X = 1) = 1 - P(X = 0)$</span>.</p>
  <p>Now, as <span class="math-container">$H(X)$</span> decreases to <span class="math-container">$0$</span>, then <span class="math-container">$\dfrac{1}{2^{H(X)}}$</span> increases, because the denominator <span class="math-container">$2^{H(X)}$</span> becomes smaller, where the smallest value is <span class="math-container">$H(X) = 0$</span></p>
  <p><span class="math-container">$$N(X) = \dfrac{1}{2^{H(X)}} = 1 = 1^1 * ? $$</span></p>
  <p>This is already problematic because <a href="https://en.wikipedia.org/wiki/Zero_to_the_power_of_zero" rel="nofollow noreferrer"><span class="math-container">$0^0$</span> is not well-defined</a>. So, I already see a problem with <span class="math-container">$N(X)$</span>.</p>
  <p>For <span class="math-container">$k &gt; 2$</span>, I think the same reasoning applies, but now the maximum value of <span class="math-container">$H(X)$</span> should be <span class="math-container">$\log_2 k$</span>.</p>
  <p>So, I don't think that <span class="math-container">$N(X)$</span> is of any practical value as it can lead to expressions like <span class="math-container">$0^0$</span>.</p>
  <p>However, this type of problem also arises in the original formulation of the entropy, because <span class="math-container">$\log_2 0$</span> is not defined. The convention for when <span class="math-container">$P(X) = 0$</span> is to set <span class="math-container">$P(X) \log P(X)$</span> to zero <a href="https://www.inference.org.uk/itprnn/book.pdf#page=44" rel="nofollow noreferrer">[1]</a>.</p>
  <p>Anyway, it seems to me that one way to interpret <span class="math-container">$N(X)$</span> is as the (average?) probability that <span class="math-container">$X$</span> takes one of the values, and it could be that the <strong>information content</strong> is what you are looking for. The definition can be found in <a href="https://www.inference.org.uk/itprnn/book.pdf#page=44" rel="nofollow noreferrer">[1]</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-23T12:52:20.073Z
Answers.LastActivityDate: 2021-12-23T13:03:54.243Z
Answers.LastEditDate: 2021-12-23T13:03:54.243Z
Answers.UserDisplayName: nbro
Body: <p>Suppose <span class="math-container">$X$</span> is a random variable taking <span class="math-container">$k$</span> values.</p>
  <p><span class="math-container">$$Val(X) = \{x_1, x_2, x_3, \cdots, x_k\} $$</span></p>
  <p>Then what is the following expression of <span class="math-container">$N(X)$</span> called in literature if exists? What does it signify?</p>
  <p><span class="math-container">$$ N(X) =  \prod \limits_{i = 1}^{k}  p(x_i)^{p(x_i)}$$</span></p>
  <p>I am using the notation <span class="math-container">$N(X)$</span> for the sake of my convenience only.</p>
  <hr />
  <p><strong>Background</strong>: I am asking this question because of the <a href="https://people.cs.umass.edu/%7Eelm/Teaching/Docs/mutInf.pdf" rel="nofollow noreferrer">definition of entropy</a> I encountered. Entropy is calculated as follows.</p>
  <p><span class="math-container">$$ H(X) = - \sum\limits_{i = 1}^{k} p(x_i) \log p(x_i) $$</span></p>
  <p>If I further solve <span class="math-container">$H(X)$</span> as follows, I will get <span class="math-container">$H(X)$</span> in terms of N(X).</p>
  <p><span class="math-container">$$ H(X) = - \sum\limits_{i = 1}^{k} p(x_i) \log p(x_i)  = - \sum\limits_{i = 1}^{k}  \log p(x_i)^{p(x_i)}  $$</span>
  <span class="math-container">$$\implies H(X)= - \log  \prod \limits_{i = 1}^{k} p(x_i)^{p(x_i)}   
   = - \log N(X)$$</span></p>
  <p>Entropy is used to characterize the unpredictability of a random variable.</p>
  <hr />
  <p>A logarithm is generally applied to many quantities in AI in order to bring them into <a href="https://ai.stackexchange.com/a/28243/18758">the desirable range</a> where overflow and underflow won't happen. Hence I am thinking that <span class="math-container">$\dfrac{1}{N(X)}$</span> is the actual quantity one has to measure (the entropy?). Hence I am guessing that <span class="math-container">$N(X)$</span> can be treated as reciprocal of entropy. So, does <span class="math-container">$N(X)$</span> is a quantity that has quantified the predictability of a random variable?</p>
  <p><span class="math-container">$$N(X) = \dfrac{1}{2^{H(X)}} = \dfrac{1}{2^{entropy}}$$</span></p>
  <p>So, I am wondering whether there is any quantity that <span class="math-container">$N(X)$</span> quantify.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-13T07:52:55.933Z
Comments.Text: I don't know if it signifies anything or has a term, but it looks likely to underflow on a computer if you use floating point.
Comments.UserDisplayName: Neil Slater
Tags: <terminology><probability><entropy>
Title: What does the product of probabilities raised to own powers used for entropy calculation quantify?
UserDisplayName: hanugm
ThreadText: # What does the product of probabilities raised to own powers used for entropy calculation quantify?
  
  (Asked by: username_0 on 2021-06-13)
  
  Suppose $X$ is a random variable taking $k$ values.
  
  $$Val(X) = \{x\_1, x\_2, x\_3, \cdots, x\_k\} $$
  
  Then what is the following expression of $N(X)$ called in literature if exists? What does it signify?
  
  $$ N(X) = \prod \limits\_{i = 1}^{k} p(x\_i)^{p(x\_i)}$$
  
  I am using the notation $N(X)$ for the sake of my convenience only.
  
  ---
  
  **Background**: I am asking this question because of the [definition of entropy](https://people.cs.umass.edu/%7Eelm/Teaching/Docs/mutInf.pdf) I encountered. Entropy is calculated as follows.
  
  $$ H(X) = - \sum\limits\_{i = 1}^{k} p(x\_i) \log p(x\_i) $$
  
  If I further solve $H(X)$ as follows, I will get $H(X)$ in terms of N(X).
  
  $$ H(X) = - \sum\limits\_{i = 1}^{k} p(x\_i) \log p(x\_i) = - \sum\limits\_{i = 1}^{k} \log p(x\_i)^{p(x\_i)} $$
  $$\implies H(X)= - \log \prod \limits\_{i = 1}^{k} p(x\_i)^{p(x\_i)}
  = - \log N(X)$$
  
  Entropy is used to characterize the unpredictability of a random variable.
  
  ---
  
  A logarithm is generally applied to many quantities in AI in order to bring them into [the desirable range](https://ai.stackexchange.com/a/28243/18758) where overflow and underflow won't happen. Hence I am thinking that $\dfrac{1}{N(X)}$ is the actual quantity one has to measure (the entropy?). Hence I am guessing that $N(X)$ can be treated as reciprocal of entropy. So, does $N(X)$ is a quantity that has quantified the predictability of a random variable?
  
  $$N(X) = \dfrac{1}{2^{H(X)}} = \dfrac{1}{2^{entropy}}$$
  
  So, I am wondering whether there is any quantity that $N(X)$ quantify.
  
  **Tags:** terminology, probability, entropy
  > **Comments:**
  > **username_1**: I don't know if it signifies anything or has a term, but it looks likely to underflow on a computer if you use floating point.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-12-23. Score: 1
  
  I don't know if $N(X)$ has a name or has any applicability in AI, but I can comment on how this function varies as the $H(X)$ based on your equation
  
  $$N(X) = \dfrac{1}{2^{H(X)}}$$
  
  which looks correct to me (just apply the $\log\_2$ to both sides).
  
  In the case of a Bernoulli random variable (which is a categorical r.v. that can take 2 values, $0$ or $1$, which is a special case of your categorical r.v., if you set $k=2$), then this is the relationship between the probability that this random variable $X = 1$ and the entropy of this r.v.
  
  [![enter image description here](https://i.sstatic.net/lXNSJ.png)](https://i.sstatic.net/lXNSJ.png)
  
  So, the entropy is $1$ when the probability is $0.5$ and decreases as the probability tends to $0$ or $1$, which makes sense, because the entropy quantifies the uncertainty about the r.v. Here, the entropy is computed in bits because we use the logarithm $\log\_2$.
  
  So, for $k=2$ (in your example), then, if $H(X) = 1$,
  
  $$N(X) = \dfrac{1}{2^{H(X)}} = \frac{1}{2} = \frac{1}{2}^{\frac{1}{2}} \* \frac{1}{2}^{\frac{1}{2}}$$
  
  which is equal to $P(X = 1) = 1 - P(X = 0)$.
  
  Now, as $H(X)$ decreases to $0$, then $\dfrac{1}{2^{H(X)}}$ increases, because the denominator $2^{H(X)}$ becomes smaller, where the smallest value is $H(X) = 0$
  
  $$N(X) = \dfrac{1}{2^{H(X)}} = 1 = 1^1 \* ? $$
  
  This is already problematic because [$0^0$ is not well-defined](https://en.wikipedia.org/wiki/Zero_to_the_power_of_zero). So, I already see a problem with $N(X)$.
  
  For $k > 2$, I think the same reasoning applies, but now the maximum value of $H(X)$ should be $\log\_2 k$.
  
  So, I don't think that $N(X)$ is of any practical value as it can lead to expressions like $0^0$.
  
  However, this type of problem also arises in the original formulation of the entropy, because $\log\_2 0$ is not defined. The convention for when $P(X) = 0$ is to set $P(X) \log P(X)$ to zero [[1]](https://www.inference.org.uk/itprnn/book.pdf#page=44).
  
  Anyway, it seems to me that one way to interpret $N(X)$ is as the (average?) probability that $X$ takes one of the values, and it could be that the **information content** is what you are looking for. The definition can be found in [[1]](https://www.inference.org.uk/itprnn/book.pdf#page=44).
  
  
  ---
  
  