Answers.Body: <h3>Generative models</h3>
  
  <blockquote>
    <p>The hidden units are just structural support and we don't care about what those hidden vectors really are.</p>
  </blockquote>
  
  <p>Generative modeling is concerned about <span class="math-container">$P(X)$</span>, to be able to compute it, we use <strong>representation learning</strong> (aka deep learning) to identify and disentangle the underlying explanatory factors hidden in the observed data, 
  because if we could separate them out like this  </p>
  
  <p><span class="math-container">$$P(X) = P(X \mid Z) P(Z),$$</span></p>
  
  <p>where <span class="math-container">$Z$</span> could encode not just the digit identity but the angle that the digit is drawn, the stroke width, and also abstract stylistic properties.  </p>
  
  <p>Any classifier for some task will decide which of those <span class="math-container">$z$</span> elements could help him to make a better decision, i.e. just the part that holds the digit identity and ignore the rest, this kind of modeling is called distributed representation, where each element is a simple indicator about some feature.</p>
  
  <h3>No conditional dependency within a layer</h3>
  
  <blockquote>
    <p>(my understanding for this reconstructed visible vector is this vector is a vector encoded in the defined RBM in the first place, we are not really construct something new, but we just happen to sample this vector from the defined RBM)</p>
  </blockquote>
  
  <p>RBM is an undirected graphical model and no conditional dependency within a layer. Thus, the conditional distribution over the hidden units <span class="math-container">$h$</span> given the input image <span class="math-container">$v$</span> factorizes, that is </p>
  
  <p><span class="math-container">$$p(h \mid v) = \prod_i p(h_i \mid v) $$</span> </p>
  
  <p>And the conditional distribution over the visible units <span class="math-container">$v$</span> given the hidden units <span class="math-container">$h$</span> also factorize. </p>
  
  <p><span class="math-container">$$p(v \mid h) = \prod_j p(v_j \mid h) $$</span> </p>
  
  <h3>Inference is tractable</h3>
  
  <p>Inference in RBM is tractable and we could compute <span class="math-container">$p(z|x)$</span>, but the tractability of the RBM does not extend to its partition function. </p>
  
  <p>To minimize the negative log-likelihood of the data, consider the following equation.</p>
  
  <p><span class="math-container">$$-\frac{\partial \log p(x)}{\partial \theta} = \frac{\partial \mathcal{F}(x)}{\partial \theta}-\sum_{\tilde{x}} p(\tilde{x}) \frac{\partial \mathcal{F}(\tilde{x})}{\partial \theta}$$</span></p>
  
  <p>The first term <span class="math-container">$\frac{\partial \mathcal{F}(x)}{\partial \theta}$</span> increases the probability of training data. The second term <span class="math-container">$\sum_{\tilde{x}} p(\tilde{x}) \frac{\partial \mathcal{F}(\tilde{x})}{\partial \theta}$</span> decreases the probability of samples generated by the model.  </p>
  
  <p>The second term is nothing less than an expectation over all possible configurations of the input <span class="math-container">$x$</span> (under the distribution <span class="math-container">$P$</span> formed by the model). </p>
  
  <h3>Monte Carlo-based algorithms</h3>
  
  <p>So, we use Monte Carlo based algorithms to replace the expectation with an average sum. Samples of <span class="math-container">$p(x)$</span> from the model can be obtained by running a Markov chain to convergence, using Gibbs sampling as the transition operator. </p>
  
  <p>For RBMs, the set of visible and hidden units, since they are conditionally independent, one can perform block Gibbs sampling. In this setting, visible units are sampled simultaneously given fixed values of the hidden units. Similarly, hidden units are sampled simultaneously given the visible units.  </p>
  
  <p>A step in the Markov chain is thus taken as follows:  </p>
  
  <ul>
  <li><span class="math-container">$v$</span>: input values</li>
  <li><span class="math-container">$h$</span>: hidden features, latent space, a.k.a <span class="math-container">$Z$</span></li>
  <li><span class="math-container">$W$</span>: the weight parameters; note that I am using the transposed <span class="math-container">$W$</span>,
  since it's an undirected graphical model. </li>
  </ul>
  
  <p>So, we have the formulas.</p>
  
  <p><span class="math-container">$$h^{(n+1)} \sim \operatorname{sigm}\left(W^{\prime} v^{(n)}+c\right)$$</span></p>
  
  <p>and </p>
  
  <p><span class="math-container">$$v^{(n+1)} \sim \operatorname{sigm}\left(W h^{(n+1)}+b\right)$$</span></p>
  
  <p>where <span class="math-container">$h^{n+1}$</span> refers to the set of all hidden units at the <span class="math-container">$n$</span>th step of the Markov chain. What it means is that, for example, <span class="math-container">$h^(n+1)_i$</span> is randomly chosen to be <span class="math-container">$1$</span> (versus <span class="math-container">$0$</span>) with the probability of the sigmoid function and similarly for <span class="math-container">$v^(n+1)_j$</span>.</p>
  
  <p>In theory, each parameter update in the learning process would require running one such chain to convergence. It is needless to say that doing so would be prohibitively expensive. As such, several algorithms have been devised for RBMs, see Contrastive Divergence.  </p>
  
  <p>We initialize the Markov chain with a training example (i.e., from a distribution that is expected to be close to <span class="math-container">$p$</span>, so that the chain will be already close to having converged to its final distribution <span class="math-container">$p$</span>).<br>
  CD does not wait for the chain to converge. Samples are obtained after only <span class="math-container">$k$</span>-steps of Gibbs sampling. In practice, <span class="math-container">$k=1$</span> has been shown to work surprisingly well.  </p>
  
  <h3>RBM for dimensionality reduction</h3>
  
  <blockquote>
    <p>RBM can be used to perform dimensionality reduction, and those hidden vectors are some abstract representations of the raw inputs:</p>
  </blockquote>
  
  <p>According to the paper <a href="https://www.cs.toronto.edu/~hinton/science.pdf" rel="nofollow noreferrer">Reducing the Dimensionality of Data with Neural Networks</a>  </p>
  
  <blockquote>
    <p>In 2006, a breakthrough in feature learning and deep learning took place (Hinton et al., 2006; Bengio et al., 2007; Ranzato et al., 2007), A central idea, referred to as greedy layerwise unsupervised pre-training, was to learn a hierarchy of features one level at a time, using unsupervised feature learning to learn a new transformation at each level
    to be composed with the previously learned transformations; essentially, each iteration of unsupervised feature learning adds one layer of weights to a deep neural network. Finally, the set of layers could be combined to initialize a deep supervised predictor, such as a neural network classifier.  </p>
  </blockquote>
  
  <p><a href="https://i.sstatic.net/8wyos.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/8wyos.png" alt="enter image description here"></a></p>
  
  <h3>Summary</h3>
  
  <ol>
  <li><p>Train single layer RBM to reconstruct to input and then keep adding layer after layer until we reach some good low-level representation as depicted in the figure below, W1, W2, W3 are the matrices that have some lower dimensionality of the input space.</p></li>
  <li><p>Transpose those matrices <span class="math-container">$W_1, W_2, W_3$</span> to be the decoder part.</p></li>
  <li><p>Fine-tune the whole encoder-decoder stack to reconstruct the input. image</p></li>
  </ol>
  
  <p><a href="https://i.sstatic.net/KLakD.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/KLakD.png" alt="enter image description here"></a></p>
  
  <h3>RBM vs VAE and GAN</h3>
  
  <p>Side note: RBM is now obsolete, see VAE, GAN they are way better at modeling the manifold in which the data is concentrated, yet you could sample from the manifold directly with a single inference step.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-04T14:47:21.157Z
Answers.LastActivityDate: 2020-05-16T15:03:41.733Z
Answers.LastEditDate: 2020-05-16T15:03:41.733Z
Answers.UserDisplayName: Fadi Bakoura
Body: <p>I'm learning about the <a href="https://en.wikipedia.org/wiki/Restricted_Boltzmann_machine" rel="nofollow noreferrer">restricted Boltzmann machine</a> (RBM), and I just came up with two naive understandings of this model. But it seems these two understandings are so different.</p>
  
  <p>My first understanding goes like this.</p>
  
  <p><strong>The  hidden units are just structural support and we don't care about what those hidden vectors really are.</strong> We introduce those hidden units just for RBM to gain more expressive power for a probability distribution.</p>
  
  <p>Let's say I have only one image, and I transform this image into a binary vector and feed this vector into an RBM with random variables (all the weights and biases are chosen randomly). Then, by turning on the machine, the first hidden vector would be constructed. But this hidden vector does not tell us anything, it would be only used to reconstruct a visible vector.
  <strong>(My understanding for this reconstructed visible vector is this vector is a vector encoded in the defined RBM in the first place, we are not really construct something new, but we just happen to sample this vector from the defined RBM).</strong>
  And we just run this loop of construction and reconstruction for infinitely many times. Finally, what we will get is just the probability distribution encoded in this RBM with random variables.</p>
  
  <p>My second understanding goes like this.</p>
  
  <p><strong>RBM can be used to perform dimensionality reduction, and those hidden vectors are some abstract representations of the raw inputs</strong>:</p>
  
  <p>Given an RBM, each hidden unit of the RBM would be a classifier, and what it does is to check the input vector lies in which side of the hyperplane defined by these hidden units (by its weights and bias). So, if we input an image into this RBM, the RBM will project this input vector onto many hyperplanes defined by all the hidden units, and thus for an input vector, the corresponding hidden vector is very important. It is some abstract representation. And we can further feed this representation into other learning models.</p>
  
  <p><em>What is the role of the hidden vectors in restricted Boltzmann machines?</em></p>
  
  <p>If you can answer this question by explaining how RBM is used for MNIST that would be extremely helpful for me.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-04T14:53:25.830Z
Comments.Text: This question was also asked on Data Science SE: [https://datascience.stackexchange.com/q/23194/10640](https://datascience.stackexchange.com/q/23194/10640).
Comments.UserDisplayName: nbro
Tags: <neural-networks><machine-learning><boltzmann-machine><restricted-boltzmann-machine>
Title: What is the role of the hidden vectors in restricted Boltzmann machines?
UserDisplayName: meTchaikovsky
ThreadText: # What is the role of the hidden vectors in restricted Boltzmann machines?
  
  (Asked by: username_0 on 2017-09-21)
  
  I'm learning about the [restricted Boltzmann machine](https://en.wikipedia.org/wiki/Restricted_Boltzmann_machine) (RBM), and I just came up with two naive understandings of this model. But it seems these two understandings are so different.
  
  My first understanding goes like this.
  
  **The hidden units are just structural support and we don't care about what those hidden vectors really are.** We introduce those hidden units just for RBM to gain more expressive power for a probability distribution.
  
  Let's say I have only one image, and I transform this image into a binary vector and feed this vector into an RBM with random variables (all the weights and biases are chosen randomly). Then, by turning on the machine, the first hidden vector would be constructed. But this hidden vector does not tell us anything, it would be only used to reconstruct a visible vector.
  **(My understanding for this reconstructed visible vector is this vector is a vector encoded in the defined RBM in the first place, we are not really construct something new, but we just happen to sample this vector from the defined RBM).**
  And we just run this loop of construction and reconstruction for infinitely many times. Finally, what we will get is just the probability distribution encoded in this RBM with random variables.
  
  My second understanding goes like this.
  
  **RBM can be used to perform dimensionality reduction, and those hidden vectors are some abstract representations of the raw inputs**:
  
  Given an RBM, each hidden unit of the RBM would be a classifier, and what it does is to check the input vector lies in which side of the hyperplane defined by these hidden units (by its weights and bias). So, if we input an image into this RBM, the RBM will project this input vector onto many hyperplanes defined by all the hidden units, and thus for an input vector, the corresponding hidden vector is very important. It is some abstract representation. And we can further feed this representation into other learning models.
  
  *What is the role of the hidden vectors in restricted Boltzmann machines?*
  
  If you can answer this question by explaining how RBM is used for MNIST that would be extremely helpful for me.
  
  **Tags:** neural-networks, machine-learning, boltzmann-machine, restricted-boltzmann-machine
  > **Comments:**
  > **username_1**: This question was also asked on Data Science SE: [https://datascience.stackexchange.com/q/23194/10640](https://datascience.stackexchange.com/q/23194/10640).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2018-05-04. Score: 4 | Accepted answer
  
  ### Generative models
  
  > The hidden units are just structural support and we don't care about what those hidden vectors really are.
  
  Generative modeling is concerned about $P(X)$, to be able to compute it, we use **representation learning** (aka deep learning) to identify and disentangle the underlying explanatory factors hidden in the observed data,
  because if we could separate them out like this
  
  $$P(X) = P(X \mid Z) P(Z),$$
  
  where $Z$ could encode not just the digit identity but the angle that the digit is drawn, the stroke width, and also abstract stylistic properties.
  
  Any classifier for some task will decide which of those $z$ elements could help him to make a better decision, i.e. just the part that holds the digit identity and ignore the rest, this kind of modeling is called distributed representation, where each element is a simple indicator about some feature.
  
  ### No conditional dependency within a layer
  
  > (my understanding for this reconstructed visible vector is this vector is a vector encoded in the defined RBM in the first place, we are not really construct something new, but we just happen to sample this vector from the defined RBM)
  
  RBM is an undirected graphical model and no conditional dependency within a layer. Thus, the conditional distribution over the hidden units $h$ given the input image $v$ factorizes, that is
  
  $$p(h \mid v) = \prod\_i p(h\_i \mid v) $$
  
  And the conditional distribution over the visible units $v$ given the hidden units $h$ also factorize.
  
  $$p(v \mid h) = \prod\_j p(v\_j \mid h) $$
  
  ### Inference is tractable
  
  Inference in RBM is tractable and we could compute $p(z|x)$, but the tractability of the RBM does not extend to its partition function.
  
  To minimize the negative log-likelihood of the data, consider the following equation.
  
  $$-\frac{\partial \log p(x)}{\partial \theta} = \frac{\partial \mathcal{F}(x)}{\partial \theta}-\sum\_{\tilde{x}} p(\tilde{x}) \frac{\partial \mathcal{F}(\tilde{x})}{\partial \theta}$$
  
  The first term $\frac{\partial \mathcal{F}(x)}{\partial \theta}$ increases the probability of training data. The second term $\sum\_{\tilde{x}} p(\tilde{x}) \frac{\partial \mathcal{F}(\tilde{x})}{\partial \theta}$ decreases the probability of samples generated by the model.
  
  The second term is nothing less than an expectation over all possible configurations of the input $x$ (under the distribution $P$ formed by the model).
  
  ### Monte Carlo-based algorithms
  
  So, we use Monte Carlo based algorithms to replace the expectation with an average sum. Samples of $p(x)$ from the model can be obtained by running a Markov chain to convergence, using Gibbs sampling as the transition operator.
  
  For RBMs, the set of visible and hidden units, since they are conditionally independent, one can perform block Gibbs sampling. In this setting, visible units are sampled simultaneously given fixed values of the hidden units. Similarly, hidden units are sampled simultaneously given the visible units.
  
  A step in the Markov chain is thus taken as follows:
  
  * $v$: input values
  * $h$: hidden features, latent space, a.k.a $Z$
  * $W$: the weight parameters; note that I am using the transposed $W$,
    since it's an undirected graphical model.
  
  So, we have the formulas.
  
  $$h^{(n+1)} \sim \operatorname{sigm}\left(W^{\prime} v^{(n)}+c\right)$$
  
  and
  
  $$v^{(n+1)} \sim \operatorname{sigm}\left(W h^{(n+1)}+b\right)$$
  
  where $h^{n+1}$ refers to the set of all hidden units at the $n$th step of the Markov chain. What it means is that, for example, $h^(n+1)\_i$ is randomly chosen to be $1$ (versus $0$) with the probability of the sigmoid function and similarly for $v^(n+1)\_j$.
  
  In theory, each parameter update in the learning process would require running one such chain to convergence. It is needless to say that doing so would be prohibitively expensive. As such, several algorithms have been devised for RBMs, see Contrastive Divergence.
  
  We initialize the Markov chain with a training example (i.e., from a distribution that is expected to be close to $p$, so that the chain will be already close to having converged to its final distribution $p$).  
  CD does not wait for the chain to converge. Samples are obtained after only $k$-steps of Gibbs sampling. In practice, $k=1$ has been shown to work surprisingly well.
  
  ### RBM for dimensionality reduction
  
  > RBM can be used to perform dimensionality reduction, and those hidden vectors are some abstract representations of the raw inputs:
  
  According to the paper [Reducing the Dimensionality of Data with Neural Networks](https://www.cs.toronto.edu/~hinton/science.pdf)
  
  > In 2006, a breakthrough in feature learning and deep learning took place (Hinton et al., 2006; Bengio et al., 2007; Ranzato et al., 2007), A central idea, referred to as greedy layerwise unsupervised pre-training, was to learn a hierarchy of features one level at a time, using unsupervised feature learning to learn a new transformation at each level
  > to be composed with the previously learned transformations; essentially, each iteration of unsupervised feature learning adds one layer of weights to a deep neural network. Finally, the set of layers could be combined to initialize a deep supervised predictor, such as a neural network classifier.
  
  [![enter image description here](https://i.sstatic.net/8wyos.png)](https://i.sstatic.net/8wyos.png)
  
  ### Summary
  
  1. Train single layer RBM to reconstruct to input and then keep adding layer after layer until we reach some good low-level representation as depicted in the figure below, W1, W2, W3 are the matrices that have some lower dimensionality of the input space.
  2. Transpose those matrices $W\_1, W\_2, W\_3$ to be the decoder part.
  3. Fine-tune the whole encoder-decoder stack to reconstruct the input. image
  
  [![enter image description here](https://i.sstatic.net/KLakD.png)](https://i.sstatic.net/KLakD.png)
  
  ### RBM vs VAE and GAN
  
  Side note: RBM is now obsolete, see VAE, GAN they are way better at modeling the manifold in which the data is concentrated, yet you could sample from the manifold directly with a single inference step.
  
  
  ---
  
  