Answers.Body: <p>In section 2.1 of the <a href="https://www.aclweb.org/anthology/C18-1139" rel="nofollow noreferrer">paper</a>, the authors state that the goal of <em>character-level language model</em> is to estimate the following <a href="https://en.wikipedia.org/wiki/Joint_probability_distribution" rel="nofollow noreferrer">joint probability distribution</a></p>
  
  <p><span class="math-container">$$P(\boldsymbol{x}_{0:T}) = P(\boldsymbol{x}_{0}, \boldsymbol{x}_{1}, \dots, \boldsymbol{x}_{T}),$$</span> </p>
  
  <p>which is a joint probability distribution of all characters of a sequence of <span class="math-container">$T+1$</span> characters, where <span class="math-container">$\boldsymbol{x}_i$</span> is the <span class="math-container">$i$</span>th character of the sequence. Ideally, the joint probability distribution, <span class="math-container">$P(\boldsymbol{x}_{0:T})$</span>, should put more mass (or density) on the combination of <span class="math-container">$T-1$</span> characters that are more likely (in a given language). For example, in English, the combination of the characters "the" is more likely than the combination "bibliopole". Hence, <span class="math-container">$P(\text{t}, \text{h}, \text{e})$</span> should be higher than <span class="math-container">$P(\text{b}, \text{i}, \text{b}, \text{l}, \text{i}, \text{o}, \text{p}, \text{o}, \text{l}, \text{e})$</span>, where, in this case, <span class="math-container">$T=2$</span>. So, <span class="math-container">$P(\boldsymbol{x}_{0:T})$</span> is the actual character-level language model, which is represented by a joint probability distribution.</p>
  
  <p>Similarly and intuitively, the conditional probability distribution <span class="math-container">$$P(\boldsymbol{x}_{t} \mid \boldsymbol{x}_{0:t-1})$$</span></p>
  
  <p>tells us the probability of the next character of the sequence <span class="math-container">$\boldsymbol{x}_{t}$</span> given (that is, having observed) the previous characters of the sequence <span class="math-container">$\boldsymbol{x}_{0:t-1} = (\boldsymbol{x}_{0}, \dots, \boldsymbol{x}_{t-1})$</span>. In general, the next character of a sequence <span class="math-container">$\boldsymbol{x}_{t}$</span> can depend on all previous characters of the same sequence, <span class="math-container">$\boldsymbol{x}_{0:t-1}$</span>. If we are able to learn the conditional model <span class="math-container">$P(\boldsymbol{x}_{t} \mid \boldsymbol{x}_{0:t-1})$</span>, given a sequence of <span class="math-container">$t$</span> characters, <span class="math-container">$\boldsymbol{x}_{0:t-1}$</span>, we can sample the next character according to the same conditional probability distribution.</p>
  
  <p>Recall that, given two events (or random variables) <span class="math-container">$A$</span> and <span class="math-container">$B$</span>, the joint distribution of them is defined as <span class="math-container">$P(A, B) = P(A \mid B)P(B) = P(B \mid A)P(A)$</span>, which gives rise to the Bayes' theorem. This can be easily generalised to multiple variables. More specifically, if you consider <span class="math-container">$B$</span> to a set of <span class="math-container">$N$</span> events (rather than just one), that is, <span class="math-container">$B=B_1 \cap B_2\cap \dots \cap B_N = B_1, B_2, \dots, B_N$</span>, then <span class="math-container">$P(A, B) = P(A \mid B)P(B)$</span> still holds, but we can further decompose it</p>
  
  <p><span class="math-container">\begin{align}
  P(A, B) = P(A, B_1, B_2, \dots, B_N)
  &amp;= P(A \mid B_1, B_2, \dots, B_N)P(B_1, B_2, \dots, B_N) \\
  &amp;= P(A \mid B_1, B_2, \dots, B_N)P(B_1 \mid B_2, \dots, B_N) P(B_2, \dots, B_N) \\
  &amp;= \cdots
  \end{align}</span></p>
  
  <p>This is called the <a href="https://en.wikipedia.org/wiki/Chain_rule_(probability)" rel="nofollow noreferrer">chain rule (or product rule)</a> of probability. Essentially, we apply the rule <span class="math-container">$P(A, B) = P(A \mid B)P(B)$</span> recursively.</p>
  
  <p>Analogously, in the <a href="https://www.aclweb.org/anthology/C18-1139" rel="nofollow noreferrer">paper</a>, the authors apply this chain rule to express the joint distribution <span class="math-container">$P(\boldsymbol{x}_{0:T})$</span> as a product of conditional probability distributions, that is</p>
  
  <p><span class="math-container">$$
  P(\boldsymbol{x}_{0:T}) = \prod_{t=0}^T P(\boldsymbol{x}_{t}\mid \boldsymbol{x}_{0:t-1})
  $$</span> </p>
  
  <p>In the case of the LSTM, the <em>vector</em> <span class="math-container">$\boldsymbol{h}_t$</span> is supposed to keep track of the past. More specifically, in the case of the character-level language model, we train an LSTM-based RNN, so that <span class="math-container">$\boldsymbol{h}_t$</span> is approximately equal to <span class="math-container">$\boldsymbol{x}_{0:t-1}$</span>, that is, <span class="math-container">$\boldsymbol{h}_t \approx \boldsymbol{x}_{0:t-1}$</span>. Hence, the joint probability distribution of the characters above can be now be approximately defined as a function of the vector <span class="math-container">$\boldsymbol{h}_t$</span></p>
  
  <p><span class="math-container">$$
  P(\boldsymbol{x}_{0:T}) \approx \prod_{t=0}^T P(\boldsymbol{x}_{t}\mid \boldsymbol{h}_t; \boldsymbol{\theta})
  $$</span> </p>
  
  <p>where <span class="math-container">$\boldsymbol{\theta}$</span> are the parameters of the LSTM-based RNN. Note that this is not an equality but an <em>approximation</em>. Intuitively, we train the LSTM so that it learns the interactions of the past characters.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-30T14:44:53.203Z
Answers.LastActivityDate: 2019-05-30T16:46:16.487Z
Answers.LastEditDate: 2019-05-30T16:46:16.487Z
Answers.UserDisplayName: nbro
Body: <p>In the paper, <a href="https://www.aclweb.org/anthology/C18-1139" rel="nofollow noreferrer">Contextual String Embeddings for Sequence Labeling</a>, the authors state that</p>
  
  <p><span class="math-container">\begin{equation}
  P(x_{0:T}) = \prod_{t=0}^T P(x_t|x_{0:t-1})
  \end{equation}</span></p>
  
  <p>They also state that, in the LSTM architecture, the conditional probability <span class="math-container">$P(x_{t}|x_{0:t})$</span> is approximately a function of the network output <span class="math-container">$h_t$</span>.</p>
  
  <p><span class="math-container">\begin{equation}
  P(x_{t}|x_{0:t}) \approx \prod_{t=0}^{T} P(x_t|h_t;\theta)
  \end{equation}</span></p>
  
  <p>Why is this equation true?</p>
  
Tags: <machine-learning><natural-language-processing><long-short-term-memory><probability-distribution><sequence-modeling>
Title: Why can we approximate the joint probability distribution using the output vector of an LSTM?
UserDisplayName: Sarthak Mittal
ThreadText: # Why can we approximate the joint probability distribution using the output vector of an LSTM?
  
  (Asked by: username_0 on 2019-05-29)
  
  In the paper, [Contextual String Embeddings for Sequence Labeling](https://www.aclweb.org/anthology/C18-1139), the authors state that
  
  \begin{equation}
  P(x\_{0:T}) = \prod\_{t=0}^T P(x\_t|x\_{0:t-1})
  \end{equation}
  
  They also state that, in the LSTM architecture, the conditional probability $P(x\_{t}|x\_{0:t})$ is approximately a function of the network output $h\_t$.
  
  \begin{equation}
  P(x\_{t}|x\_{0:t}) \approx \prod\_{t=0}^{T} P(x\_t|h\_t;\theta)
  \end{equation}
  
  Why is this equation true?
  
  **Tags:** machine-learning, natural-language-processing, long-short-term-memory, probability-distribution, sequence-modeling
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-05-30. Score: 1 | Accepted answer
  
  In section 2.1 of the [paper](https://www.aclweb.org/anthology/C18-1139), the authors state that the goal of *character-level language model* is to estimate the following [joint probability distribution](https://en.wikipedia.org/wiki/Joint_probability_distribution)
  
  $$P(\boldsymbol{x}\_{0:T}) = P(\boldsymbol{x}\_{0}, \boldsymbol{x}\_{1}, \dots, \boldsymbol{x}\_{T}),$$
  
  which is a joint probability distribution of all characters of a sequence of $T+1$ characters, where $\boldsymbol{x}\_i$ is the $i$th character of the sequence. Ideally, the joint probability distribution, $P(\boldsymbol{x}\_{0:T})$, should put more mass (or density) on the combination of $T-1$ characters that are more likely (in a given language). For example, in English, the combination of the characters "the" is more likely than the combination "bibliopole". Hence, $P(\text{t}, \text{h}, \text{e})$ should be higher than $P(\text{b}, \text{i}, \text{b}, \text{l}, \text{i}, \text{o}, \text{p}, \text{o}, \text{l}, \text{e})$, where, in this case, $T=2$. So, $P(\boldsymbol{x}\_{0:T})$ is the actual character-level language model, which is represented by a joint probability distribution.
  
  Similarly and intuitively, the conditional probability distribution $$P(\boldsymbol{x}\_{t} \mid \boldsymbol{x}\_{0:t-1})$$
  
  tells us the probability of the next character of the sequence $\boldsymbol{x}\_{t}$ given (that is, having observed) the previous characters of the sequence $\boldsymbol{x}\_{0:t-1} = (\boldsymbol{x}\_{0}, \dots, \boldsymbol{x}\_{t-1})$. In general, the next character of a sequence $\boldsymbol{x}\_{t}$ can depend on all previous characters of the same sequence, $\boldsymbol{x}\_{0:t-1}$. If we are able to learn the conditional model $P(\boldsymbol{x}\_{t} \mid \boldsymbol{x}\_{0:t-1})$, given a sequence of $t$ characters, $\boldsymbol{x}\_{0:t-1}$, we can sample the next character according to the same conditional probability distribution.
  
  Recall that, given two events (or random variables) $A$ and $B$, the joint distribution of them is defined as $P(A, B) = P(A \mid B)P(B) = P(B \mid A)P(A)$, which gives rise to the Bayes' theorem. This can be easily generalised to multiple variables. More specifically, if you consider $B$ to a set of $N$ events (rather than just one), that is, $B=B\_1 \cap B\_2\cap \dots \cap B\_N = B\_1, B\_2, \dots, B\_N$, then $P(A, B) = P(A \mid B)P(B)$ still holds, but we can further decompose it
  
  \begin{align}
  P(A, B) = P(A, B\_1, B\_2, \dots, B\_N)
  &= P(A \mid B\_1, B\_2, \dots, B\_N)P(B\_1, B\_2, \dots, B\_N) \\
  &= P(A \mid B\_1, B\_2, \dots, B\_N)P(B\_1 \mid B\_2, \dots, B\_N) P(B\_2, \dots, B\_N) \\
  &= \cdots
  \end{align}
  
  This is called the [chain rule (or product rule)](https://en.wikipedia.org/wiki/Chain_rule_(probability)) of probability. Essentially, we apply the rule $P(A, B) = P(A \mid B)P(B)$ recursively.
  
  Analogously, in the [paper](https://www.aclweb.org/anthology/C18-1139), the authors apply this chain rule to express the joint distribution $P(\boldsymbol{x}\_{0:T})$ as a product of conditional probability distributions, that is
  
  $$
  P(\boldsymbol{x}\_{0:T}) = \prod\_{t=0}^T P(\boldsymbol{x}\_{t}\mid \boldsymbol{x}\_{0:t-1})
  $$
  
  In the case of the LSTM, the *vector* $\boldsymbol{h}\_t$ is supposed to keep track of the past. More specifically, in the case of the character-level language model, we train an LSTM-based RNN, so that $\boldsymbol{h}\_t$ is approximately equal to $\boldsymbol{x}\_{0:t-1}$, that is, $\boldsymbol{h}\_t \approx \boldsymbol{x}\_{0:t-1}$. Hence, the joint probability distribution of the characters above can be now be approximately defined as a function of the vector $\boldsymbol{h}\_t$
  
  $$
  P(\boldsymbol{x}\_{0:T}) \approx \prod\_{t=0}^T P(\boldsymbol{x}\_{t}\mid \boldsymbol{h}\_t; \boldsymbol{\theta})
  $$
  
  where $\boldsymbol{\theta}$ are the parameters of the LSTM-based RNN. Note that this is not an equality but an *approximation*. Intuitively, we train the LSTM so that it learns the interactions of the past characters.
  
  
  ---
  
  