Body: <p>From the part titled <em>Introducing Latent Variables</em> under subsection 2.2 in this <a href="https://link.springer.com/chapter/10.1007/978-3-642-33275-3_2" rel="nofollow noreferrer">tutorial</a>:</p>
  <blockquote>
  <p><strong>Introducing Latent Variables.</strong> Suppose we want to model an <span class="math-container">$m$</span>-dimensional
  unknown probability distribution <span class="math-container">$q$</span> (e.g., each component of a sample corresponds to one of m pixels of an image). Typically, not all variables <span class="math-container">$\mathbf{X} = (X_v)_{v \in V}$</span> in an MRF need to correspond to some observed component, and the number of nodes is larger than <span class="math-container">$m$</span>. We split <span class="math-container">$\mathbf{X}$</span> into visible (or observed) variables <span class="math-container">$\mathbf{V} = (V_1,...,V_m)$</span> corresponding to the components of the observations and latent (or hidden) variables <span class="math-container">$\mathbf{H} = (H_1,...,H_n)$</span> given by the remaining <span class="math-container">$n = |\mathbf{V}| − m$</span> variables. Using latent variables allows to describe complex distributions over the visible variables by means of simple (conditional) distributions. In this case the Gibbs distribution of an MRF describes the joint probability distribution of <span class="math-container">$(\mathbf{V},\mathbf{H})$</span> and one is usually interested in the marginal distribution of <span class="math-container">$\mathbf{V}$</span> which is given by:
  <span class="math-container">$$p(\mathbf{v}) = \sum_{\mathbf{h}} p(\mathbf{v},\mathbf{h}) = \frac{1}{Z} \sum_{\mathbf{h}} e^{-E(\mathbf{v},\mathbf{h})}$$</span>
  where <span class="math-container">$Z = \sum_{\mathbf{v},\mathbf{h}} e^{-E(\mathbf{v},\mathbf{h})}$</span>. While the visible variables correspond to the components of an observation, the latent variables introduce dependencies between the visible variables (e.g., between pixels of an input image).</p>
  </blockquote>
  <p>I have a question about this part:</p>
  <blockquote>
  <p>While the visible variables correspond to the components of an observation, the latent variables introduce dependencies between the visible variables (e.g., between pixels of an input image).</p>
  </blockquote>
  <p>Given a set of nodes <span class="math-container">$\mathbf{X}$</span> in a Markov Random Field <span class="math-container">$G$</span>, the joint distribution of all the nodes is given by:</p>
  <p><span class="math-container">$$p(\mathbf{X}) = \frac{1}{Z} \prod_{c \in C} \phi(c)$$</span></p>
  <p>Where <span class="math-container">$Z$</span> is the partition function and <span class="math-container">$C$</span> is the set of cliques in <span class="math-container">$G$</span>. To ensure that the joint distribution is positive, the following factors can be used:</p>
  <p><span class="math-container">$$\phi(c) = e^{-E(c)}$$</span></p>
  <p>Such that:</p>
  <p><span class="math-container">$$p(\mathbf{X}) = \frac{1}{Z} e^{-\sum_{c \in C} E(c)}$$</span></p>
  <p>Where <span class="math-container">$E$</span> is the energy function.</p>
  <p>I am not sure why there is a need to introduce hidden variables and express <span class="math-container">$p(\mathbf{v})$</span> as a marginalization of <span class="math-container">$p(\mathbf{v},\mathbf{h})$</span> over <span class="math-container">$\mathbf{h}$</span>. Why can't <span class="math-container">$p(\mathbf{v})$</span> be expressed as:</p>
  <p><span class="math-container">$$p(\mathbf{v}) = \frac{1}{Z} e^{-\sum_{v \in \mathbf{v}} E(v)}$$</span></p>
  <p>directly? I think it may be because the factors only encode dependencies between variables in cliques, and so may not be able to encode dependencies between variables that are in two separate cliques. The purpose of the hidden variables are then to encode these &quot;long-range&quot; dependencies between visible variables not in cliques. However, I am not sure about this reasoning.</p>
  <p>Any help would be greatly appreciated.</p>
  <p>By the way, I am aware of <a href="https://ai.stackexchange.com/questions/4084/what-is-the-role-of-the-hidden-vectors-in-restricted-boltzmann-machines?rq=1">this</a> question, but I think the answer is not specific enough.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-27T01:34:13.183Z
Comments.Text: Welcome to SE:AI!
Comments.UserDisplayName: DukeZhou
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-28T11:35:22.273Z
Comments.Text: @DuttaA thanks for the advice! Which chapter in the deep learning book are you referring to?
Comments.UserDisplayName: mhdadk
Tags: <restricted-boltzmann-machine><probabilistic-graphical-models>
Title: Purpose of the hidden variables in a Restricted Boltzmann Machine
UserDisplayName: mhdadk
ThreadText: # Purpose of the hidden variables in a Restricted Boltzmann Machine
  
  (Asked by: username_0 on 2020-10-26)
  
  From the part titled *Introducing Latent Variables* under subsection 2.2 in this [tutorial](https://link.springer.com/chapter/10.1007/978-3-642-33275-3_2):
  
  > **Introducing Latent Variables.** Suppose we want to model an $m$-dimensional
  > unknown probability distribution $q$ (e.g., each component of a sample corresponds to one of m pixels of an image). Typically, not all variables $\mathbf{X} = (X\_v)\_{v \in V}$ in an MRF need to correspond to some observed component, and the number of nodes is larger than $m$. We split $\mathbf{X}$ into visible (or observed) variables $\mathbf{V} = (V\_1,...,V\_m)$ corresponding to the components of the observations and latent (or hidden) variables $\mathbf{H} = (H\_1,...,H\_n)$ given by the remaining $n = |\mathbf{V}| − m$ variables. Using latent variables allows to describe complex distributions over the visible variables by means of simple (conditional) distributions. In this case the Gibbs distribution of an MRF describes the joint probability distribution of $(\mathbf{V},\mathbf{H})$ and one is usually interested in the marginal distribution of $\mathbf{V}$ which is given by:
  > $$p(\mathbf{v}) = \sum\_{\mathbf{h}} p(\mathbf{v},\mathbf{h}) = \frac{1}{Z} \sum\_{\mathbf{h}} e^{-E(\mathbf{v},\mathbf{h})}$$
  > where $Z = \sum\_{\mathbf{v},\mathbf{h}} e^{-E(\mathbf{v},\mathbf{h})}$. While the visible variables correspond to the components of an observation, the latent variables introduce dependencies between the visible variables (e.g., between pixels of an input image).
  
  I have a question about this part:
  
  > While the visible variables correspond to the components of an observation, the latent variables introduce dependencies between the visible variables (e.g., between pixels of an input image).
  
  Given a set of nodes $\mathbf{X}$ in a Markov Random Field $G$, the joint distribution of all the nodes is given by:
  
  $$p(\mathbf{X}) = \frac{1}{Z} \prod\_{c \in C} \phi(c)$$
  
  Where $Z$ is the partition function and $C$ is the set of cliques in $G$. To ensure that the joint distribution is positive, the following factors can be used:
  
  $$\phi(c) = e^{-E(c)}$$
  
  Such that:
  
  $$p(\mathbf{X}) = \frac{1}{Z} e^{-\sum\_{c \in C} E(c)}$$
  
  Where $E$ is the energy function.
  
  I am not sure why there is a need to introduce hidden variables and express $p(\mathbf{v})$ as a marginalization of $p(\mathbf{v},\mathbf{h})$ over $\mathbf{h}$. Why can't $p(\mathbf{v})$ be expressed as:
  
  $$p(\mathbf{v}) = \frac{1}{Z} e^{-\sum\_{v \in \mathbf{v}} E(v)}$$
  
  directly? I think it may be because the factors only encode dependencies between variables in cliques, and so may not be able to encode dependencies between variables that are in two separate cliques. The purpose of the hidden variables are then to encode these "long-range" dependencies between visible variables not in cliques. However, I am not sure about this reasoning.
  
  Any help would be greatly appreciated.
  
  By the way, I am aware of [this](https://ai.stackexchange.com/questions/4084/what-is-the-role-of-the-hidden-vectors-in-restricted-boltzmann-machines?rq=1) question, but I think the answer is not specific enough.
  
  **Tags:** restricted-boltzmann-machine, probabilistic-graphical-models
  > **Comments:**
  > **username_1**: Welcome to SE:AI!  
  > **username_0**: @DuttaA thanks for the advice! Which chapter in the deep learning book are you referring to?  
  
  ---
  
  ## 0 answers
  
  