REPRESENTATION DEGENERATION PROBLEM IN TRAINING NATURAL LANGUAGE GENERATION MOD- ELS
We study an interesting problem in training neural network-based models for natural language generation tasks, which we call the representation degeneration problem. We observe that when training a model for natural language generation tasks through likelihood maximization with the weight tying trick, especially with big training datasets, most of the learnt word embeddings tend to degenerate and be distributed into a narrow cone, which largely limits the representation power of word embeddings. We analyze the conditions and causes of this problem and propose a novel regularization method to address it. Experiments on language modeling and machine translation show that our method can largely mitigate the representation degeneration problem and achieve better performance than baseline algorithms.
The paper presents and discusses a new phenomenon that infrequent words tend to learn degenerate embeddings. A cosine regularization term is proposed to address this issue.
Pros
1. The degenerate embedding problem is novel and interesting.
2. Some positive empirical results.
Cons and questions
1. The theory in Section 4 suggests that the degeneration problem originates from underfitting; i.e., there's not enough data to fit the embeddings of the infrequent words, when epsilon is small. However, the solution in Section 5 is based on a regularization term. This seems contradictory to me because adding regularization to an underfit model would not make it better. In other words, if there's not enough data to fit the word embeddings, one should feed more data. It seems that a cosine regularization term could only make the embeddings different from each other, but not better.
2. Since this is an underfitting problem (as described in Section 4), I'm wondering what would happen on larger datasets. The claims in the paper could be better substantiated if there are results on larger datasets like WT103 for LM and en-fr for MT. Intuitively, by increasing the amount of total data, the same word gets more data to fit, and thus epsilon gets large enough so that degeneration might not happen.
3. "Discussion on whether the condition happens in real practice" below Theorem 2 seems not correct to me. Even when layer normalization is employed and bias is not zero, the convex hull can still contain the origin as long as the length of the bias vector is less than 1. In fact, this condition seems fairly strong, and surely it will not hold "almost for sure in practice".
4. The cosine regularization term seems expensive, especially when the vocab size is large. Any results in terms of computational costs? Did you employ tricks to speed it up?
5. What would happen if we only apply the cosine term on infrequent words? An ablation study might make it clear why it improves performance.
UPDATE:
I think the rebuttal addresses some of my concerns. I am especially glad to see improvement on en-fr, too. Thus I raised my score from 5 to 7.