Answers.Body: <p>First of all, when you add hidden layers, or stack RBMs, you get a Deep Belief Network (DBN). Your question then deals with the comparison of DBNs and RBMs.</p>
  
  <p>There are some elements to answer this question in the article <a href="http://www.iro.umontreal.ca/~lisa/pointeurs/rbm_universality_techreport.pdf" rel="nofollow noreferrer">Representational Power of Restricted Boltzmann Machines and Deep Belief Networks</a> by Nicolas Le Roux, which can be found summarized in these <a href="http://swoh.web.engr.illinois.edu/courses/IE598/handout/fall2016_slide1.pdf" rel="nofollow noreferrer">course slides</a>. The main results are:</p>
  
  <blockquote>
    <p>Restricted Boltzmann Machines:</p>
    
    <ul>
    <li><p>Increasing the number of hidden units improves representational ability.</p></li>
    <li><p>With an unbounded number of units, any distribution over <span class="math-container">$\{0,1\}^n$</span> can be approximated arbitrarily well.</p></li>
    </ul>
    
    <p>Deep Belief Networks:</p>
    
    <ul>
    <li><p><strong>Adding additional layers using greedy contrastive divergence training does not provide additional benefit</strong>.</p></li>
    <li><p>There remain open questions about the benefits additional layers add.</p></li>
    </ul>
  </blockquote>
  
  <p>I emphasized the <span class="math-container">$4^{th}$</span> point that covers the most your question. It does not mean that additional layers are useless, but since RBM are universal approximators (<span class="math-container">$2^{nd}$</span> point), the benefits of adding layers seem less straightforward. They seem dependent on the first layer, the training procedure... You can see the article for more details and the open questions raised about DBNs. Note that this answer is an entry point on the topic, there might be more recent results following this article which I'm not familiar with...</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-29T07:48:24.080Z
Answers.LastActivityDate: 2019-10-29T13:38:04.060Z
Answers.LastEditDate: 2019-10-29T13:38:04.060Z
Answers.UserDisplayName: TheCG
Body: <p>I'm using a restricted Boltzmann machine (RBM) as an autoencoder. For now, I use a simple architecture of two layers, the input (~100 nodes) and the output (3 nodes) layers.  I'm thinking to add more hidden layers. </p>
  
  <p>Are there some improvements in encoding by adding multiple hidden layers? If yes, how can multiple layers improve the encoding?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-23T09:29:53.383Z
Comments.Text: It's been too long since I've looked at them for me to feel comfortable writing a proper answer... but you're going to want to look into "Deep Belief Networks"
Comments.UserDisplayName: Dennis Soemers
Tags: <hidden-layers><boltzmann-machine><restricted-boltzmann-machine>
Title: Does the encoding of a restricted Boltzmann machine improve with more layers?
UserDisplayName: Simone Morettini
ThreadText: # Does the encoding of a restricted Boltzmann machine improve with more layers?
  
  (Asked by: username_0 on 2019-06-22)
  
  I'm using a restricted Boltzmann machine (RBM) as an autoencoder. For now, I use a simple architecture of two layers, the input (~100 nodes) and the output (3 nodes) layers. I'm thinking to add more hidden layers.
  
  Are there some improvements in encoding by adding multiple hidden layers? If yes, how can multiple layers improve the encoding?
  
  **Tags:** hidden-layers, boltzmann-machine, restricted-boltzmann-machine
  > **Comments:**
  > **username_1**: It's been too long since I've looked at them for me to feel comfortable writing a proper answer... but you're going to want to look into "Deep Belief Networks"  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-10-29. Score: 1
  
  First of all, when you add hidden layers, or stack RBMs, you get a Deep Belief Network (DBN). Your question then deals with the comparison of DBNs and RBMs.
  
  There are some elements to answer this question in the article [Representational Power of Restricted Boltzmann Machines and Deep Belief Networks](http://www.iro.umontreal.ca/~lisa/pointeurs/rbm_universality_techreport.pdf) by Nicolas Le Roux, which can be found summarized in these [course slides](http://swoh.web.engr.illinois.edu/courses/IE598/handout/fall2016_slide1.pdf). The main results are:
  
  > Restricted Boltzmann Machines:
  >
  > * Increasing the number of hidden units improves representational ability.
  > * With an unbounded number of units, any distribution over $\{0,1\}^n$ can be approximated arbitrarily well.
  >
  > Deep Belief Networks:
  >
  > * **Adding additional layers using greedy contrastive divergence training does not provide additional benefit**.
  > * There remain open questions about the benefits additional layers add.
  
  I emphasized the $4^{th}$ point that covers the most your question. It does not mean that additional layers are useless, but since RBM are universal approximators ($2^{nd}$ point), the benefits of adding layers seem less straightforward. They seem dependent on the first layer, the training procedure... You can see the article for more details and the open questions raised about DBNs. Note that this answer is an entry point on the topic, there might be more recent results following this article which I'm not familiar with...
  
  
  ---
  
  