Answers.Body: <h3>Introduction: MAP finds a point estimate!</h3>
  
  <p>As opposed to your apparently current belief, in maximum a posteriori (MAP) estimation, you are looking for a <em>point estimate</em> (a number or vector) rather than a full probability distribution. The MAP estimation can be seen as a <a href="http://www.scholarpedia.org/article/Bayesian_statistics" rel="nofollow noreferrer">Bayesian version</a> of the maximum likelihood estimation (MLE). Therefore, I will first remind you of the objective of MLE.</p>
  
  <h3>Maximum likelihood estimation (MLE)</h3>
  
  <p>Let <span class="math-container">$\theta$</span> be the parameters you want to find. For example, <span class="math-container">$\theta$</span> can be the weights of your neural network. In MLE, we want to find a point estimate (rather than a full distribution). The objective in MLE is</p>
  
  <p><span class="math-container">\begin{align}
  \theta^* 
  &amp;= \operatorname{argmax}_\theta p(X \mid \theta) \tag{1}\label{1}
  \end{align}</span></p>
  
  <p>where <span class="math-container">$p(X \mid \theta)$</span> is the likelihood of the data <span class="math-container">$X$</span> given the parameters <span class="math-container">$\theta$</span>. In other words, we want to find the parameters <span class="math-container">$\theta$</span> such that <span class="math-container">$p(X \mid \theta)$</span> is the highest, where <span class="math-container">$X$</span> is your given training data, so <span class="math-container">$X$</span> is fixed. </p>
  
  <p>The notation <span class="math-container">$p(X \mid \theta)$</span> can be confusing because, in a conditional probability distribution, <span class="math-container">$p(a\mid b)$</span>, we often assume that <span class="math-container">$b$</span> is given and <span class="math-container">$p(a\mid b)$</span> is a distribution over <span class="math-container">$a$</span>. However, in the case of MLE, <span class="math-container">$\theta$</span> in <span class="math-container">$p(X \mid \theta)$</span> is not fixed, but it is a variable, while <span class="math-container">$X$</span> is given and fixed. Hence we call <span class="math-container">$p(X \mid \theta)$</span> a <strong>likelihood</strong> rather than a probability density or mass function. Moreover, we often denote the likelihood as <span class="math-container">$\mathcal{L}(\theta; X) = p_{\theta}(X)$</span> (and there are other notations, but this is, in my opinion, the least confusing one), because we want to emphasize that the likelihood is actually a function of the variable <span class="math-container">$\theta$</span>. However, this is notation can also be confusing because we equate a function of a variable <span class="math-container">$\theta$</span> to a probability distribution over <span class="math-container">$X$</span>. However, you should note that <span class="math-container">$p_{\theta}(X)$</span> is parametrized by <span class="math-container">$\theta$</span>.</p>
  
  <p>Therefore, the MLE estimation \ref{1} can also be written as follows</p>
  
  <p><span class="math-container">\begin{align}
  \theta^* 
  &amp;= 
  \operatorname{argmax}_\theta \mathcal{L}(\theta; X) \\
  &amp;=\operatorname{argmax}_\theta p_{\theta}(X)
  \tag{2}\label{2}
  \end{align}</span>
  where <span class="math-container">$\theta^*$</span> is the <strong>point estimate</strong> of the objective function.</p>
  
  <p>This notation emphasizes the fact that we want to find <span class="math-container">$\theta$</span>, such that the probability of the given data <span class="math-container">$X$</span> is maximized.</p>
  
  <h3>Maximum a posteriori (MAP)</h3>
  
  <p>MAP is similar to MLE, but the objective is slightly different. First of all, we assume that <span class="math-container">$\theta$</span> is a random variable, so we have an associated probability distribution <span class="math-container">$p(\theta)$</span>.</p>
  
  <p>Recall that the Bayes' rule is the following</p>
  
  <p><span class="math-container">\begin{align}
  p(\theta \mid X) = \frac{p(X \mid \theta) p(\theta)}{p(X)}
  \tag{3}\label{3}
  \end{align}</span></p>
  
  <p>The objective function in MAP estimation is</p>
  
  <p><span class="math-container">\begin{align}
  \theta^* 
  &amp;= \operatorname{argmax}_\theta \frac{p(X \mid \theta) p(\theta)}{p(X)} \\
  &amp;= \operatorname{argmax}_\theta p(\theta \mid X) \tag{4}\label{4}
  \end{align}</span></p>
  
  <p>Given that <span class="math-container">$p(X)$</span> does not depend on <span class="math-container">$\theta$</span>, for the purposes of optimization, we can ignore it, so equation \ref{4} becomes</p>
  
  <p><span class="math-container">\begin{align}
  \theta^* 
  &amp;= \operatorname{argmax}_\theta p(X \mid \theta) p(\theta) \\
  &amp;= \operatorname{argmax}_\theta p(\theta \mid X)
  \tag{5}\label{5}
  \end{align}</span>
  which is the MAP objective.</p>
  
  <h3>What is the relationship between MLE and MAP?</h3>
  
  <ul>
  <li><p>In MAP, the objective is \ref{5}, which includes a prior over <span class="math-container">$\theta$</span>, while, in MLE, equation \ref{1}, there is no such thing. </p>
  
  <ul>
  <li>Therefore, in MAP, we can assume that the parameters <span class="math-container">$\theta$</span> follow a certain distribution, thanks to the usage of <span class="math-container">$p(\theta)$</span>.</li>
  </ul></li>
  <li><p>In both MAP and MLE, we want to find a <strong>point estimate</strong> (which can be a number, if you have just one parameter, or a vector of size <span class="math-container">$N$</span>, if you have <span class="math-container">$N$</span> parameters).</p></li>
  <li><p>MAP is equivalent to MLE if you use a uniform prior, that is, if <span class="math-container">$p(\theta)$</span> is a uniform distribution.</p></li>
  </ul>
  
  <h3>Which distribution fits the data <span class="math-container">$X$</span>?</h3>
  
  <p>In MAP, the human (you, me, etc.) chooses the family of distributions. For example, you can assume that your parameters <span class="math-container">$\theta$</span> follow a Gaussian distribution, so <span class="math-container">$p(\theta)$</span> will be a Gaussian distribution over the parameters. Why do I say "family"? For example, in the case of a Gaussian distribution, you have two parameters that control the shape of the distribution, namely, the mean and variance. Depending on the concrete values of these two parameters, you will have different Gaussian distributions, so you call all these Gaussian distributions a family.</p>
  
  <h3>How do you find <span class="math-container">$\theta$</span>?</h3>
  
  <p>To find <span class="math-container">$\theta^*$</span>, you can use an optimization method like <a href="https://en.wikipedia.org/wiki/Gradient_descent" rel="nofollow noreferrer">gradient descent</a> or, in certain cases, you can find a <a href="http://mathworld.wolfram.com/Closed-FormSolution.html" rel="nofollow noreferrer">closed-form solution</a>. See also <a href="https://stats.stackexchange.com/q/32103/82135">Which distributions have closed-form solutions for maximum likelihood estimation?</a>.</p>
  
  <h3>Resources</h3>
  
  <p>The following blog post <a href="https://wiseodd.github.io/techblog/2017/01/01/mle-vs-map/" rel="nofollow noreferrer">MLE vs MAP: the connection between Maximum Likelihood and Maximum A Posteriori Estimation</a>, by Agustinus Kristiadi (a Ph.D. student in machine learning), might also be useful, so I suggest you read it. It will give you more details that I've left out on purpose to avoid cluttering this answer.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-23T17:08:42.543Z
Answers.LastActivityDate: 2019-11-23T17:14:12.193Z
Answers.LastEditDate: 2019-11-23T17:14:12.193Z
Answers.UserDisplayName: nbro
Body: <p>I was learning about the <em>maximum a posteriori probability (MAP)</em> estimation for machine learning and I found <a href="https://www.youtube.com/watch?v=XepXtl9YKwc" rel="nofollow noreferrer">a nice short video</a> that essentially explained it as finding a distribution and tweaking the parameters to fit the observed data in a way that makes the observations most likely (makes sense). </p>
  
  <p>However, in mathematical terms, how does it determine which distribution best fits the data? </p>
  
  <p>There are so many distributions out there that it could be any of them and the parameters you could fit them could be infinitely large.</p>
  
Tags: <probability-distribution><bayesian-probability>
Title: How does maximum approximation of the posterior choose a distribution?
UserDisplayName: user8714896
ThreadText: # How does maximum approximation of the posterior choose a distribution?
  
  (Asked by: username_0 on 2019-11-21)
  
  I was learning about the *maximum a posteriori probability (MAP)* estimation for machine learning and I found [a nice short video](https://www.youtube.com/watch?v=XepXtl9YKwc) that essentially explained it as finding a distribution and tweaking the parameters to fit the observed data in a way that makes the observations most likely (makes sense).
  
  However, in mathematical terms, how does it determine which distribution best fits the data?
  
  There are so many distributions out there that it could be any of them and the parameters you could fit them could be infinitely large.
  
  **Tags:** probability-distribution, bayesian-probability
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-11-23. Score: 3 | Accepted answer
  
  ### Introduction: MAP finds a point estimate!
  
  As opposed to your apparently current belief, in maximum a posteriori (MAP) estimation, you are looking for a *point estimate* (a number or vector) rather than a full probability distribution. The MAP estimation can be seen as a [Bayesian version](http://www.scholarpedia.org/article/Bayesian_statistics) of the maximum likelihood estimation (MLE). Therefore, I will first remind you of the objective of MLE.
  
  ### Maximum likelihood estimation (MLE)
  
  Let $\theta$ be the parameters you want to find. For example, $\theta$ can be the weights of your neural network. In MLE, we want to find a point estimate (rather than a full distribution). The objective in MLE is
  
  \begin{align}
  \theta^\*
  &= \operatorname{argmax}\_\theta p(X \mid \theta) \tag{1}\label{1}
  \end{align}
  
  where $p(X \mid \theta)$ is the likelihood of the data $X$ given the parameters $\theta$. In other words, we want to find the parameters $\theta$ such that $p(X \mid \theta)$ is the highest, where $X$ is your given training data, so $X$ is fixed.
  
  The notation $p(X \mid \theta)$ can be confusing because, in a conditional probability distribution, $p(a\mid b)$, we often assume that $b$ is given and $p(a\mid b)$ is a distribution over $a$. However, in the case of MLE, $\theta$ in $p(X \mid \theta)$ is not fixed, but it is a variable, while $X$ is given and fixed. Hence we call $p(X \mid \theta)$ a **likelihood** rather than a probability density or mass function. Moreover, we often denote the likelihood as $\mathcal{L}(\theta; X) = p\_{\theta}(X)$ (and there are other notations, but this is, in my opinion, the least confusing one), because we want to emphasize that the likelihood is actually a function of the variable $\theta$. However, this is notation can also be confusing because we equate a function of a variable $\theta$ to a probability distribution over $X$. However, you should note that $p\_{\theta}(X)$ is parametrized by $\theta$.
  
  Therefore, the MLE estimation \ref{1} can also be written as follows
  
  \begin{align}
  \theta^\*
  &=
  \operatorname{argmax}\_\theta \mathcal{L}(\theta; X) \\
  &=\operatorname{argmax}\_\theta p\_{\theta}(X)
  \tag{2}\label{2}
  \end{align}
  where $\theta^\*$ is the **point estimate** of the objective function.
  
  This notation emphasizes the fact that we want to find $\theta$, such that the probability of the given data $X$ is maximized.
  
  ### Maximum a posteriori (MAP)
  
  MAP is similar to MLE, but the objective is slightly different. First of all, we assume that $\theta$ is a random variable, so we have an associated probability distribution $p(\theta)$.
  
  Recall that the Bayes' rule is the following
  
  \begin{align}
  p(\theta \mid X) = \frac{p(X \mid \theta) p(\theta)}{p(X)}
  \tag{3}\label{3}
  \end{align}
  
  The objective function in MAP estimation is
  
  \begin{align}
  \theta^\*
  &= \operatorname{argmax}\_\theta \frac{p(X \mid \theta) p(\theta)}{p(X)} \\
  &= \operatorname{argmax}\_\theta p(\theta \mid X) \tag{4}\label{4}
  \end{align}
  
  Given that $p(X)$ does not depend on $\theta$, for the purposes of optimization, we can ignore it, so equation \ref{4} becomes
  
  \begin{align}
  \theta^\*
  &= \operatorname{argmax}\_\theta p(X \mid \theta) p(\theta) \\
  &= \operatorname{argmax}\_\theta p(\theta \mid X)
  \tag{5}\label{5}
  \end{align}
  which is the MAP objective.
  
  ### What is the relationship between MLE and MAP?
  
  * In MAP, the objective is \ref{5}, which includes a prior over $\theta$, while, in MLE, equation \ref{1}, there is no such thing.
  
    + Therefore, in MAP, we can assume that the parameters $\theta$ follow a certain distribution, thanks to the usage of $p(\theta)$.
  * In both MAP and MLE, we want to find a **point estimate** (which can be a number, if you have just one parameter, or a vector of size $N$, if you have $N$ parameters).
  * MAP is equivalent to MLE if you use a uniform prior, that is, if $p(\theta)$ is a uniform distribution.
  
  ### Which distribution fits the data $X$?
  
  In MAP, the human (you, me, etc.) chooses the family of distributions. For example, you can assume that your parameters $\theta$ follow a Gaussian distribution, so $p(\theta)$ will be a Gaussian distribution over the parameters. Why do I say "family"? For example, in the case of a Gaussian distribution, you have two parameters that control the shape of the distribution, namely, the mean and variance. Depending on the concrete values of these two parameters, you will have different Gaussian distributions, so you call all these Gaussian distributions a family.
  
  ### How do you find $\theta$?
  
  To find $\theta^\*$, you can use an optimization method like [gradient descent](https://en.wikipedia.org/wiki/Gradient_descent) or, in certain cases, you can find a [closed-form solution](http://mathworld.wolfram.com/Closed-FormSolution.html). See also [Which distributions have closed-form solutions for maximum likelihood estimation?](https://stats.stackexchange.com/q/32103/82135).
  
  ### Resources
  
  The following blog post [MLE vs MAP: the connection between Maximum Likelihood and Maximum A Posteriori Estimation](https://wiseodd.github.io/techblog/2017/01/01/mle-vs-map/), by Agustinus Kristiadi (a Ph.D. student in machine learning), might also be useful, so I suggest you read it. It will give you more details that I've left out on purpose to avoid cluttering this answer.
  
  
  ---
  
  