Well, removing entries is easy. You can simply index the elements you want to keep. And the gradients will flow properly back to these elements.
Be careful though when you generate these one-hot encodings that you do so in a way that returns non-zero gradients.