Answers.Body: <p>You can find a description of this distribution (which is also known as <strong>categorical distribution</strong>, which you probably already heard of) in section 2.3.2 (p. 35) of the book <a href="http://noiselab.ucsd.edu/ECE228/Murphy_Machine_Learning.pdf#page66" rel="nofollow noreferrer">Machine Learning: A Probabilistic Perspective</a> (by K. Murphy). You can also find there and in the previous section a description of the related Bernoulli, binomial and multinomial (the most general of the four) distributions. The word <em>multi<strong>noulli</strong></em> is used in order to remind you that this distribution is a generalization of the Ber<strong>noulli</strong>.</p>
  <p>In any case, this is how you may remember these four probability distributions.</p>
  <ul>
  <li><a href="https://en.wikipedia.org/wiki/Bernoulli_distribution" rel="nofollow noreferrer">Bernoulli</a>: you throw a coin only <strong>once</strong> (<span class="math-container">$n=1$</span>), and a coin has <strong><span class="math-container">$k = 2$</span> outcomes</strong> (heads or tails)</li>
  <li><a href="https://en.wikipedia.org/wiki/Binomial_distribution" rel="nofollow noreferrer">Binomial</a>: you throw a coin <span class="math-container">$n$</span> times, where <span class="math-container">$n$</span> can be greater than <span class="math-container">$1$</span>, and a coin has <strong><span class="math-container">$k = 2$</span> outcomes</strong></li>
  <li><a href="https://en.wikipedia.org/wiki/Categorical_distribution" rel="nofollow noreferrer">Categorical</a>: you throw a dice, with <strong><span class="math-container">$k$</span> sides</strong> (e.g. a side may be <span class="math-container">$1$</span> or <span class="math-container">$5$</span>), where <span class="math-container">$k$</span> can be greater than <span class="math-container">$2$</span>, only <strong>once</strong> (<span class="math-container">$n=1$</span>)</li>
  <li><a href="https://en.wikipedia.org/wiki/Multinomial_distribution" rel="nofollow noreferrer">Multinomial</a>: you throw a dice, with <strong><span class="math-container">$k$</span> sides, <span class="math-container">$n$</span> times</strong></li>
  </ul>
  <p>So, these are all discrete probability distributions, with an associated probability mass function (p.m.f). In all of them, we have two parameters <span class="math-container">$n$</span> (the number of trials of the experiment) and <span class="math-container">$k$</span> (the number of outcomes).</p>
  <p>I don't think it is correct to say that a softmax is a probability distribution. The softmax is a function used to compute the probabilities that you associate with a categorical distribution, i.e. you use the softmax to produce a <a href="https://en.wikipedia.org/wiki/Probability_vector" rel="nofollow noreferrer">probability vector</a> (although some people will say that the softmax produces a <a href="https://en.wikipedia.org/wiki/Probability_distribution" rel="nofollow noreferrer">probability distribution</a>), as the excerpt that you quote states. In principle, I think you could use other functions to do that (an alternative to the regular softmax, <em>sigsoftmax</em>, is proposed in <a href="https://papers.nips.cc/paper/2018/file/9dcb88e0137649590b755372b040afad-Paper.pdf" rel="nofollow noreferrer">this paper</a>, section 3.3, p. 6). So, the <em>softmax</em> is used to <em>model</em> a categorical distribution, but I wouldn't say it's a categorical distribution. You can find an explanation of why the softmax is used instead of e.g. just normalizing by the sum <a href="https://ai.stackexchange.com/q/17325/2444">here</a> (and a long list of probability distributions <a href="https://en.wikipedia.org/wiki/List_of_probability_distributions" rel="nofollow noreferrer">here</a>).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-19T10:40:19.647Z
Answers.LastActivityDate: 2021-08-19T10:52:09.740Z
Answers.LastEditDate: 2021-08-19T10:52:09.740Z
Answers.UserDisplayName: nbro
Body: <p>I encountered the term <strong>multinoulli distribution</strong> in the following sentence from <a href="https://www.deeplearningbook.org/contents/numerical.html" rel="nofollow noreferrer">Chapter 4: Numerical Computation</a> of the deep learning book.</p>
  <blockquote>
  <p>The softmax function is often used to predict the probabilities
  associated with a <strong>multinoulli distribution</strong>.</p>
  </blockquote>
  <p>I am guessing that multinouli distribution is any probability distribution that has been defined on a <strong>random variable taking multiple values</strong>. I know that SoftMax function is used in converting a vector into another vector of the same length with probability values that signify the chance of input falling into that particular class.</p>
  <p>Suppose <span class="math-container">$C$</span> is a random variable with support <span class="math-container">$\{c_1, c_2, c_3, \cdots, c_k\} $</span>. Then I am guesssing that any probability distribution on <span class="math-container">$C$</span> is a multinouli distribution. SoftMax is an example of such multinouli distribution that uses the expression <span class="math-container">$\dfrac{e^x}{\sum e^x}$</span> for calculating probabilities.</p>
  <p>Is my guess correct about multinoulli distribution? The reason for my doubt is that I never came across the word multinoulli and I cannot find even on internet. If my guess is wrong, where can I read about multinoulli distribution?</p>
  
Tags: <terminology><probability-distribution><softmax>
Title: Where can I read about the multinoulli distribution?
UserDisplayName: hanugm
ThreadText: # Where can I read about the multinoulli distribution?
  
  (Asked by: username_0 on 2021-08-17)
  
  I encountered the term **multinoulli distribution** in the following sentence from [Chapter 4: Numerical Computation](https://www.deeplearningbook.org/contents/numerical.html) of the deep learning book.
  
  > The softmax function is often used to predict the probabilities
  > associated with a **multinoulli distribution**.
  
  I am guessing that multinouli distribution is any probability distribution that has been defined on a **random variable taking multiple values**. I know that SoftMax function is used in converting a vector into another vector of the same length with probability values that signify the chance of input falling into that particular class.
  
  Suppose $C$ is a random variable with support $\{c\_1, c\_2, c\_3, \cdots, c\_k\} $. Then I am guesssing that any probability distribution on $C$ is a multinouli distribution. SoftMax is an example of such multinouli distribution that uses the expression $\dfrac{e^x}{\sum e^x}$ for calculating probabilities.
  
  Is my guess correct about multinoulli distribution? The reason for my doubt is that I never came across the word multinoulli and I cannot find even on internet. If my guess is wrong, where can I read about multinoulli distribution?
  
  **Tags:** terminology, probability-distribution, softmax
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-08-19. Score: 1
  
  You can find a description of this distribution (which is also known as **categorical distribution**, which you probably already heard of) in section 2.3.2 (p. 35) of the book [Machine Learning: A Probabilistic Perspective](http://noiselab.ucsd.edu/ECE228/Murphy_Machine_Learning.pdf#page66) (by K. Murphy). You can also find there and in the previous section a description of the related Bernoulli, binomial and multinomial (the most general of the four) distributions. The word *multi**noulli*** is used in order to remind you that this distribution is a generalization of the Ber**noulli**.
  
  In any case, this is how you may remember these four probability distributions.
  
  * [Bernoulli](https://en.wikipedia.org/wiki/Bernoulli_distribution): you throw a coin only **once** ($n=1$), and a coin has **$k = 2$ outcomes** (heads or tails)
  * [Binomial](https://en.wikipedia.org/wiki/Binomial_distribution): you throw a coin $n$ times, where $n$ can be greater than $1$, and a coin has **$k = 2$ outcomes**
  * [Categorical](https://en.wikipedia.org/wiki/Categorical_distribution): you throw a dice, with **$k$ sides** (e.g. a side may be $1$ or $5$), where $k$ can be greater than $2$, only **once** ($n=1$)
  * [Multinomial](https://en.wikipedia.org/wiki/Multinomial_distribution): you throw a dice, with **$k$ sides, $n$ times**
  
  So, these are all discrete probability distributions, with an associated probability mass function (p.m.f). In all of them, we have two parameters $n$ (the number of trials of the experiment) and $k$ (the number of outcomes).
  
  I don't think it is correct to say that a softmax is a probability distribution. The softmax is a function used to compute the probabilities that you associate with a categorical distribution, i.e. you use the softmax to produce a [probability vector](https://en.wikipedia.org/wiki/Probability_vector) (although some people will say that the softmax produces a [probability distribution](https://en.wikipedia.org/wiki/Probability_distribution)), as the excerpt that you quote states. In principle, I think you could use other functions to do that (an alternative to the regular softmax, *sigsoftmax*, is proposed in [this paper](https://papers.nips.cc/paper/2018/file/9dcb88e0137649590b755372b040afad-Paper.pdf), section 3.3, p. 6). So, the *softmax* is used to *model* a categorical distribution, but I wouldn't say it's a categorical distribution. You can find an explanation of why the softmax is used instead of e.g. just normalizing by the sum [here](https://ai.stackexchange.com/q/17325/2444) (and a long list of probability distributions [here](https://en.wikipedia.org/wiki/List_of_probability_distributions)).
  
  
  ---
  
  