We propose a method, called Label Embedding Network, which can learn label representation (label embedding) during the training process of deep networks. With the proposed method, the label embedding is adaptively and automatically learned through back propagation. The original one-hot represented loss function is converted into a new loss function with soft distributions, such that the originally unrelated labels have continuous interactions with each other during the training process. As a result, the trained model can achieve substantially higher accuracy and with faster convergence speed. Experimental results based on competitive tasks demonstrate the effectiveness of the proposed method, and the learned label embedding is reasonable and interpretable. The proposed method achieves comparable or even better results than the state-of-the-art systems.
The paper proposes to add an embedding layer for labels that constrains normal classifiers in order to find label representations that are semantically consistent. The approach is then experimented on various image and text tasks.
The description of the model is laborious and hard to follow. Figure 1 helps but is only referred to at the end of the description (at the end of section 2.1), which instead explains each step without the big picture and loses the reader with confusing notation. For instance, it only became clear at the end of the section that E was learned.
One of the motivations behing the model is to force label representations to be in a semantic space (where two labels with similar meanings would be nearby). The assumption given in the introduction is that softmax would not yield such a representation, but nowhere in the paper this assumption is verified. I believe that using cross-entropy with softmax should also push semantically similar labels to be nearby in the weight space entering the softmax. This should at least be verified and compared appropriately.
Another motivation of the paper is that targets are given as 1s or 0s while soft targets should work better. I believe this is true, but there is a lot of prior work on these, such as adding a temperature to the softmax, or using distillation, etc. None of these are discussed appropriately in the paper.
Section 2.2 describes a way to compress the label embedding representation, but it is not clear if this is actually used in the experiments. h is never discussed after section 2.2.
Experiments on known datasets are interesting, but none of the results are competitive with current state-of-the-art results (SOTA), despite what is said in Appending D. For instance, one can find SOTA results for CIFAR100 around 16% and for CIFAR10 around 3%. Similarly, one can find SOTA results for IWSLT2015 around 28 BLEU. It can be fine to not be SOTA as long as it is acknowledged and discussed appropriately.