Body: <p>I am currently working with a categorical-binary RBM, where there are 50 categorical visible units and 25 binary hidden units. The categorical visible units are expressed in one-hot encoding format, such that if there is 5 categories, then the visible units are expressed as a <span class="math-container">$50 \times 5$</span> array, where each row is the one-hot encoding of a category from 1 to 5.</p>
  <p>Ideally, the RBM should be able to reconstruct the visible units. However, since the visible units are in one-hot encoding, then the visible units array contains a lot of zeros. This means the RBM quickly learns to guess all zeros for the entire array to minimize the reconstruction loss. How can I force the RBM to not do this and to instead guess 1's where the category occurs and 0's otherwise?</p>
  <p>Note that I would still have this problem with a regular autoencoder.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-10T15:48:36.017Z
Comments.Text: I don't think the problem you are experiencing is related to overfitting (as the title suggests), but due to sparsity of output in the target population for the generator. Perhaps you could [edit] the title? Interesting problem though
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-10T16:23:22.490Z
Comments.Text: Yes you are correct. [Here](https://towardsdatascience.com/building-autoencoders-on-sparse-one-hot-encoded-data-53eefdfdbcc7) is an article discussing this exact problem. However, this article discusses autoencoders, not RBMs. Got any ideas?
Comments.UserDisplayName: mhdadk
Tags: <overfitting><variational-autoencoder><restricted-boltzmann-machine>
Title: How can I reconstruct sparse one-hot encodings using an RBM?
UserDisplayName: mhdadk
ThreadText: # How can I reconstruct sparse one-hot encodings using an RBM?
  
  (Asked by: username_0 on 2020-11-10)
  
  I am currently working with a categorical-binary RBM, where there are 50 categorical visible units and 25 binary hidden units. The categorical visible units are expressed in one-hot encoding format, such that if there is 5 categories, then the visible units are expressed as a $50 \times 5$ array, where each row is the one-hot encoding of a category from 1 to 5.
  
  Ideally, the RBM should be able to reconstruct the visible units. However, since the visible units are in one-hot encoding, then the visible units array contains a lot of zeros. This means the RBM quickly learns to guess all zeros for the entire array to minimize the reconstruction loss. How can I force the RBM to not do this and to instead guess 1's where the category occurs and 0's otherwise?
  
  Note that I would still have this problem with a regular autoencoder.
  
  **Tags:** overfitting, variational-autoencoder, restricted-boltzmann-machine
  > **Comments:**
  > **username_1**: I don't think the problem you are experiencing is related to overfitting (as the title suggests), but due to sparsity of output in the target population for the generator. Perhaps you could [edit] the title? Interesting problem though  
  > **username_0**: Yes you are correct. [Here](https://towardsdatascience.com/building-autoencoders-on-sparse-one-hot-encoded-data-53eefdfdbcc7) is an article discussing this exact problem. However, this article discusses autoencoders, not RBMs. Got any ideas?  
  
  ---
  
  ## 0 answers
  
  