Decoding Decoders: Finding Optimal Representation Spaces for Unsupervised Similarity Tasks
Experimental evidence indicates that simple models outperform complex deep networks on many unsupervised similarity tasks. Introducing the concept of an optimal representation space, we provide a simple theoretical resolution to this apparent paradox. In addition, we present a straightforward procedure that, without any retraining or architectural modifications, allows deep recurrent models to perform equally well (and sometimes better) when compared to shallow models. To validate our analysis, we conduct a set of consistent empirical evaluations and introduce several new sentence embedding models in the process. Even though this work is presented within the context of natural language processing, the insights are readily applicable to other domains that rely on distributed representations for transfer tasks.
 This paper proposes the concept of optimal representation space and suggests that a model should be evaluated in its optimal representation space to get good performance. It could be a good idea if this paper could suggest some ways to find the optimal representation space in general, instead of just showing two cases. It is disappointing, because this paper is named as "finding optimal representation spaces ...". In addition, one of the contributions claimed in this paper is about introducing the "formalism" of an optimal representation space. However, I didn*t see any formal definition of this concept or theoretical justification. About FastSent or any other log-linear model, the reason that dot product (or cosine similarity) is a good metric is because the model is trained to optimize the dot product, as shown in equation 5 --- I think this simple fact is missed in this paper. The experimental results are not convincing, because I didn*t find any consistent pattern that shows the performance is getting better once we evaluated the model in its optimal representation space. There are statements in this paper that I didn*t agree with 1) Distributional hypothesis from Harris (1954) is about words not sentences. 2) Not sure the following line makes sense: "However, these unsupervised tasks are more interesting from a general AI point of view, as they test whether the machine truly understands the human notion of similarity, without being explicitly told what is similar"
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Decoding Decoders: Finding Optimal Representation Spaces for Unsupervised Similarity Tasks
Experimental evidence indicates that simple models outperform complex deep networks on many unsupervised similarity tasks. Introducing the concept of an optimal representation space, we provide a simple theoretical resolution to this apparent paradox. In addition, we present a straightforward procedure that, without any retraining or architectural modifications, allows deep recurrent models to perform equally well (and sometimes better) when compared to shallow models. To validate our analysis, we conduct a set of consistent empirical evaluations and introduce several new sentence embedding models in the process. Even though this work is presented within the context of natural language processing, the insights are readily applicable to other domains that rely on distributed representations for transfer tasks.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper proposes the concept of optimal representation space and suggests that a model should be evaluated in its optimal representation space to get good performance. It could be a good idea if this paper could suggest some ways to find the optimal representation space in general, instead of just showing two cases. It is disappointing, because this paper is named as "finding optimal representation spaces ...". In addition, one of the contributions claimed in this paper is about introducing the "formalism" of an optimal representation space. However, I didn*t see any formal definition of this concept or theoretical justification. About FastSent or any other log-linear model, the reason that dot product (or cosine similarity) is a good metric is because the model is trained to optimize the dot product, as shown in equation 5 --- I think this simple fact is missed in this paper. The experimental results are not convincing, because I didn*t find any consistent pattern that shows the performance is getting better once we evaluated the model in its optimal representation space. There are statements in this paper that I didn*t agree with 1) Distributional hypothesis from Harris (1954) is about words not sentences. 2) Not sure the following line makes sense: "However, these unsupervised tasks are more interesting from a general AI point of view, as they test whether the machine truly understands the human notion of similarity, without being explicitly told what is similar"
assistant
 This paper proposes the concept of optimal representation space and suggests that a model should be evaluated in its optimal representation space to get good performance. It could be a good idea if this paper could suggest some ways to find the optimal representation space in general, instead of just showing two cases. It is disappointing, because this paper is named as "finding optimal representation spaces ...". In addition, one of the contributions claimed in this paper is about introducing the "formalism" of an optimal representation space. However, I didn*t see any formal definition of this concept or theoretical justification. About FastSent or any other log-linear model, the reason that dot product (or cosine similarity) is a good metric is because the model is trained to optimize the dot product, as shown in equation 5 --- I think this simple fact is missed in this paper. The experimental results are not convincing, because I didn*t find any consistent pattern that shows the performance is getting better once we evaluated the model in its optimal representation space. There are statements in this paper that I didn*t agree with 1) Distributional hypothesis from Harris (1954) is about words not sentences. 2) Not sure the following line makes sense: "However, these unsupervised tasks are more interesting from a general AI point of view, as they test whether the machine truly understands the human notion of similarity, without being explicitly told what is similar"