Answers.Body: <p>When you say likelihood, you are invoking several other concepts like events, sample, parameters, model, probability density function (PDF), etc (it would be helpful if you learn more about these concepts). In essence, a likelihood function <span class="math-container">$l(x|\theta)$</span> is a PDF that quantifies how likely is that event <span class="math-container">$x$</span> happens out of a set of possible events, given the parameters <span class="math-container">$\theta$</span> that define your model.  </p>
  
  <p>In the specific case of images, the set of possible events is usually one of two options: 1) all the available images in a dataset, or 2) all the existing images. Usually you want to model the likelihood in the option 2), but only having access to a sample of all the possible images. In either case, the likelihood is just the probability that you select one image out of all the possible ones. If you consider only images of <span class="math-container">$1048\times 720$</span> pixels, the possible amount of images is <span class="math-container">$(256\times3)^{1048\times 720}$</span>, where I am assuming that each pixel consists of 3 colors and each color can take 256 values. Since the amount of possible images is so so big, it is very common that the probability of selecting a specific one is very very small. This is a reason why you usually work with log-likelihood (the logarithm of the likelihood) instead of directly using likelihoods. For example, if all your images were equally probable, the likelihood would be in the order of <span class="math-container">$10^{-{10^7}}$</span>, while the log-likelihood would be around <span class="math-container">$-10^7$</span>.  </p>
  
  <p>To solve your paradox with the probability of images and pixels, consider that instead of pixels you have coins and instead of images you have sequences of coins. Let's say that you have a fair coin, so the probability of tails (<span class="math-container">$T$</span>) after tossing the coin is 0.5. If you toss a second coin, the probability of having <span class="math-container">$T$</span> again is naturally 0.5 as well, but what is the probability that both results where <span class="math-container">$T$</span>? It is the product (0.25) since the events are independent. Similarly, the probability of the other three sequences <span class="math-container">$TH$</span>, <span class="math-container">$HT$</span> and <span class="math-container">$HH$</span> is just 0.25. You can see that since the probability needs to be shared between 4 sequences equally probable, they are less probable with respect to the probabilities of the sequences of length 1. If you toss the coin 3 times, then the probability of all these coins being tails is just <span class="math-container">$0.5^{3}$</span>. Again, there are now 8 possible sequences, all sharing the same amount of probability. You can see what's going on. Since the amount of possible options becomes large, the probability of each possible sequence of coins becomes small. Clearly, you would never toss a coin 10 times and expect to obtain all <span class="math-container">$T$</span>, right? Well, exactly the same happens in the case of the pictures.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-07T01:51:31.833Z
Answers.Text: Thanks for the answer! It makes sense why you would multiply, and use a log. I'm still a bit confused as to why a PDF in this case is being used for a discrete set (while it is massive, the set of all images of defined dimensions is finite). Moreover, normally when you want to find the probability of an event you don't find the exact events probability, but a set of events with some tolerance as an estimate, as an exact event occurring has a likelihood of 0. Is that done in unsupervised learning?
Answers.UserDisplayName: Recessive
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-08T21:42:34.977Z
Answers.Text: @Recessive, you are right in saying that the likelihood is a prob. mass function if the set of events is discrete. It just seemed more general to refer to a density function. Now, in the case of images, actually it is more common to suppose that the pixel values are continuous and that the likelihood is a PDF. This supposition is made precisely because the set of events is so massive and the probabilities become so insignificant. The numbers I used were mostly to try to give you an intuition.
Answers.UserDisplayName: Diego Gomez
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-06T17:35:18.477Z
Answers.LastActivityDate: 2019-12-06T17:35:18.477Z
Answers.UserDisplayName: Diego Gomez
Body: <p>The specific problem I'm having is with a Fully Visible Belief Network. It is an explicit density model (though I don't know what quantifies something being such) that uses the chain rule to decompose the likelihood of an image x into a product of 1-d distributions.</p>
  
  <p><a href="https://i.sstatic.net/fg8PC.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/fg8PC.png" alt="enter image description here"></a></p>
  
  <p>What is meant by "the likelihood of an image x"? With respect to what? I assume it refers to how common this image would be in the data set it is selected from? Like if you had 1000 images, 800 of which were white and 200 of which were black, the model should ideally output 0.2 for any black image inputted? Obliviously with more complicated clustering like dogs vs cats it'd be a bit different, but that's my intuition. Is that correct?</p>
  
  <p>Also as a side question, that equation looks very wrong. If you have an image of <span class="math-container">$1048\times720$</span> pixels, and say every pixel evaluates to have a probability of 0.9, you'd expect the final probability of the image to be 0.9 or 90%. But according to that equation, it's <span class="math-container">$0.9^{720*1048}$</span>, which is stupidly small, essentially 0. What's going on here?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-06T13:47:55.297Z
Comments.Text: @DuttaA Ok that makes sense. Do you know if my first assumption about the meaning of "likelihood of an image x" is correct?
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-06T17:16:46.210Z
Comments.Text: In practice, you often work with log probabilities, rather than probabilities. Maybe later I will have a closer look at this question and maybe try to provide an answer.
Comments.UserDisplayName: nbro
Tags: <unsupervised-learning>
Title: In unsupervised learning, what is meant by "finding the probability of an image"?
UserDisplayName: Recessive
ThreadText: # In unsupervised learning, what is meant by "finding the probability of an image"?
  
  (Asked by: username_0 on 2019-12-06)
  
  The specific problem I'm having is with a Fully Visible Belief Network. It is an explicit density model (though I don't know what quantifies something being such) that uses the chain rule to decompose the likelihood of an image x into a product of 1-d distributions.
  
  [![enter image description here](https://i.sstatic.net/fg8PC.png)](https://i.sstatic.net/fg8PC.png)
  
  What is meant by "the likelihood of an image x"? With respect to what? I assume it refers to how common this image would be in the data set it is selected from? Like if you had 1000 images, 800 of which were white and 200 of which were black, the model should ideally output 0.2 for any black image inputted? Obliviously with more complicated clustering like dogs vs cats it'd be a bit different, but that's my intuition. Is that correct?
  
  Also as a side question, that equation looks very wrong. If you have an image of $1048\times720$ pixels, and say every pixel evaluates to have a probability of 0.9, you'd expect the final probability of the image to be 0.9 or 90%. But according to that equation, it's $0.9^{720\*1048}$, which is stupidly small, essentially 0. What's going on here?
  
  **Tags:** unsupervised-learning
  
  > **Comments:**
  > **username_0**: @DuttaA Ok that makes sense. Do you know if my first assumption about the meaning of "likelihood of an image x" is correct?  
  > **username_1**: In practice, you often work with log probabilities, rather than probabilities. Maybe later I will have a closer look at this question and maybe try to provide an answer.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-12-06. Score: 2 | Accepted answer
  
  When you say likelihood, you are invoking several other concepts like events, sample, parameters, model, probability density function (PDF), etc (it would be helpful if you learn more about these concepts). In essence, a likelihood function $l(x|\theta)$ is a PDF that quantifies how likely is that event $x$ happens out of a set of possible events, given the parameters $\theta$ that define your model.
  
  In the specific case of images, the set of possible events is usually one of two options: 1) all the available images in a dataset, or 2) all the existing images. Usually you want to model the likelihood in the option 2), but only having access to a sample of all the possible images. In either case, the likelihood is just the probability that you select one image out of all the possible ones. If you consider only images of $1048\times 720$ pixels, the possible amount of images is $(256\times3)^{1048\times 720}$, where I am assuming that each pixel consists of 3 colors and each color can take 256 values. Since the amount of possible images is so so big, it is very common that the probability of selecting a specific one is very very small. This is a reason why you usually work with log-likelihood (the logarithm of the likelihood) instead of directly using likelihoods. For example, if all your images were equally probable, the likelihood would be in the order of $10^{-{10^7}}$, while the log-likelihood would be around $-10^7$.
  
  To solve your paradox with the probability of images and pixels, consider that instead of pixels you have coins and instead of images you have sequences of coins. Let's say that you have a fair coin, so the probability of tails ($T$) after tossing the coin is 0.5. If you toss a second coin, the probability of having $T$ again is naturally 0.5 as well, but what is the probability that both results where $T$? It is the product (0.25) since the events are independent. Similarly, the probability of the other three sequences $TH$, $HT$ and $HH$ is just 0.25. You can see that since the probability needs to be shared between 4 sequences equally probable, they are less probable with respect to the probabilities of the sequences of length 1. If you toss the coin 3 times, then the probability of all these coins being tails is just $0.5^{3}$. Again, there are now 8 possible sequences, all sharing the same amount of probability. You can see what's going on. Since the amount of possible options becomes large, the probability of each possible sequence of coins becomes small. Clearly, you would never toss a coin 10 times and expect to obtain all $T$, right? Well, exactly the same happens in the case of the pictures.
  
  > **Comments:**
  > **username_0**: Thanks for the answer! It makes sense why you would multiply, and use a log. I'm still a bit confused as to why a PDF in this case is being used for a discrete set (while it is massive, the set of all images of defined dimensions is finite). Moreover, normally when you want to find the probability of an event you don't find the exact events probability, but a set of events with some tolerance as an estimate, as an exact event occurring has a likelihood of 0. Is that done in unsupervised learning?  
  > **username_2**: @username_0, you are right in saying that the likelihood is a prob. mass function if the set of events is discrete. It just seemed more general to refer to a density function. Now, in the case of images, actually it is more common to suppose that the pixel values are continuous and that the likelihood is a PDF. This supposition is made precisely because the set of events is so massive and the probabilities become so insignificant. The numbers I used were mostly to try to give you an intuition.  
  
  ---
  
  