<p>I am currently working with a categorical-binary RBM, where there are 50 categorical visible units and 25 binary hidden units. The categorical visible units are expressed in one-hot encoding format, such that if there is 5 categories, then the visible units are expressed as a <span class="math-container">$50 \times 5$</span> array, where each row is the one-hot encoding of a category from 1 to 5.</p>
<p>Ideally, the RBM should be able to reconstruct the visible units. However, since the visible units are in one-hot encoding, then the visible units array contains a lot of zeros. This means the RBM quickly learns to guess all zeros for the entire array to minimize the reconstruction loss. How can I force the RBM to not do this and to instead guess 1's where the category occurs and 0's otherwise?</p>
<p>Note that I would still have this problem with a regular autoencoder.</p>

CC BY-SA 4.0
2020-11-10T15:48:36.017Z
I don't think the problem you are experiencing is related to overfitting (as the title suggests), but due to sparsity of output in the target population for the generator. Perhaps you could [edit] the title? Interesting problem though
Neil Slater
CC BY-SA 4.0
2020-11-10T16:23:22.490Z
Yes you are correct. [Here](https://towardsdatascience.com/building-autoencoders-on-sparse-one-hot-encoded-data-53eefdfdbcc7) is an article discussing this exact problem. However, this article discusses autoencoders, not RBMs. Got any ideas?
mhdadk
<overfitting><variational-autoencoder><restricted-boltzmann-machine>
How can I reconstruct sparse one-hot encodings using an RBM?
mhdadk
# How can I reconstruct sparse one-hot encodings using an RBM?

(Asked by: username_0 on 2020-11-10)

I am currently working with a categorical-binary RBM, where there are 50 categorical visible units and 25 binary hidden units. The categorical visible units are expressed in one-hot encoding format, such that if there is 5 categories, then the visible units are expressed as a $50 \times 5$ array, where each row is the one-hot encoding of a category from 1 to 5.

Ideally, the RBM should be able to reconstruct the visible units. However, since the visible units are in one-hot encoding, then the visible units array contains a lot of zeros. This means the RBM quickly learns to guess all zeros for the entire array to minimize the reconstruction loss. How can I force the RBM to not do this and to instead guess 1's where the category occurs and 0's otherwise?

Note that I would still have this problem with a regular autoencoder.

**Tags:** overfitting, variational-autoencoder, restricted-boltzmann-machine

> **Comments:**
> **username_1**: I don't think the problem you are experiencing is related to overfitting (as the title suggests), but due to sparsity of output in the target population for the generator. Perhaps you could [edit] the title? Interesting problem though  
> **username_0**: Yes you are correct. [Here](https://towardsdatascience.com/building-autoencoders-on-sparse-one-hot-encoded-data-53eefdfdbcc7) is an article discussing this exact problem. However, this article discusses autoencoders, not RBMs. Got any ideas?  

---

## 0 answers

