self.cat_emb is created via an operation and is thus not a leaf variable anymore.
The optimizer would thus raise an error, as it cannot optimizer non-leaf tensors, if you try to pass self.cat_emb to it.