Regularly varying representation for sentence embedding
The dominant approaches to sentence representation in natural language rely on learning embeddings on massive corpuses. The obtained embeddings have desirable properties such as compositionality and distance preservation (sentences with similar meanings have similar representations). In this paper, we develop a novel method for learning an embedding enjoying a dilation invariance property. We propose two algorithms: Orthrus, a classification algorithm, constrains the distribution of the embedded variable to be regularly varying, i.e. multivariate heavy-tail. and uses Extreme Value Theory (EVT) to tackle the classification task on two separate regions: the tail and the bulk. Hydra, a text generation algorithm for dataset augmentation, leverages the invariance property of the embedding learnt by Orthrus to generate coherent sentences with controllable attribute, e.g. positive or negative sentiment. Numerical experiments on synthetic and real text data demonstrate the relevance of the proposed framework.
This paper proposes a new embedding method for sentences that aims to preserve dilation invariance. Much of the methodology is justified by results for extremal point classification under particular assumptions, and then the authors try and encourage these assumptions to be met via penalty terms introduced in their embeddings/augmentation models. However, while the proposed methodology seems interesting/novel, it remains conceptually unclear why it should be superior to standard text classification methods (ie. exactly what assumptions are being exploited to improve performance and how exactly do those assumptions help should be made more explicit). In particular, why is dilation invariance even a good idea? Overall, I find the paper a bit mathematically dense in Secs 2.1-2.2, which would not be a bad thing if the math were necessary to justify why the proposed methodology works well, but it in this case seems mainly presented as background material (as if it were a prerequisite to understand the method itself, which it is certainly not). Why not instead present an explicit theorem providing some statistical guarantees for the proposed methodology in Sec 3.1 based on the constant-along-rays result (which would be nice to have regardless), and then follow up the theorem with background math from 2.1-2.2 which is necessary to understand the proof? As it is currently written the paper is a bit too dense in terminology, and opaque names like Hydra and Orthrus used to describe straightforward concepts that are essentially a neural classifier (of a particular form) and a seq2seq-based data augmentation procedure (which would be good to describe in language more familiar to the ML audience). In particular, the goals of Hydra and Orthrus should first be intuitively described before delving into their various components. - Why did the authors never evaluate the overall sentiment prediction performance of Orthrus + Hydra used together vs other classifiers + data augmentation strategies? - If the goal of dilation invariance is to help the classifiers better generalize to out-of-distribution test sentences, then why not verify this happened, eg. by training on Yelp and testing on Amazon? - The authors should better justify the assumption of Jalalzai et al, and why this is appropriate for the MLP classifier used later in the paper. - This statement needs to be clarified and have citation: *Such classifier whose output solely depends on the angle ?(x) of the considered input, with provable guarantees concerning the classification risk in out-of-sample regions scaling as the square root of the number of extreme points used at the training step* - The authors should explain Equation (1) in English rather than referring to it so early on the paper (pg 1: *satisfying Equation 1*). I had no idea what this was supposed to mean as a first time reader. - The way figure 4 is presented is a bit opaque and took me a while to understand (have to look closely at Fig 4a to see the columns are not monochromatic). - *We also compare Hydra to a Vanilla Sequence to Sequence to demonstrate the validity of our approach* How *Vanilla Sequence to Sequence* (word *model* is missing) is used for dataset augmentation needs to be clarified here. - A figure demonstrating an example of the phenomenon explained in Sec 2.2 would be helpful to aid reader*s intuition. - Typo: *eugmentation*
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Regularly varying representation for sentence embedding
The dominant approaches to sentence representation in natural language rely on learning embeddings on massive corpuses. The obtained embeddings have desirable properties such as compositionality and distance preservation (sentences with similar meanings have similar representations). In this paper, we develop a novel method for learning an embedding enjoying a dilation invariance property. We propose two algorithms: Orthrus, a classification algorithm, constrains the distribution of the embedded variable to be regularly varying, i.e. multivariate heavy-tail. and uses Extreme Value Theory (EVT) to tackle the classification task on two separate regions: the tail and the bulk. Hydra, a text generation algorithm for dataset augmentation, leverages the invariance property of the embedding learnt by Orthrus to generate coherent sentences with controllable attribute, e.g. positive or negative sentiment. Numerical experiments on synthetic and real text data demonstrate the relevance of the proposed framework.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper proposes a new embedding method for sentences that aims to preserve dilation invariance. Much of the methodology is justified by results for extremal point classification under particular assumptions, and then the authors try and encourage these assumptions to be met via penalty terms introduced in their embeddings/augmentation models. However, while the proposed methodology seems interesting/novel, it remains conceptually unclear why it should be superior to standard text classification methods (ie. exactly what assumptions are being exploited to improve performance and how exactly do those assumptions help should be made more explicit). In particular, why is dilation invariance even a good idea? Overall, I find the paper a bit mathematically dense in Secs 2.1-2.2, which would not be a bad thing if the math were necessary to justify why the proposed methodology works well, but it in this case seems mainly presented as background material (as if it were a prerequisite to understand the method itself, which it is certainly not). Why not instead present an explicit theorem providing some statistical guarantees for the proposed methodology in Sec 3.1 based on the constant-along-rays result (which would be nice to have regardless), and then follow up the theorem with background math from 2.1-2.2 which is necessary to understand the proof? As it is currently written the paper is a bit too dense in terminology, and opaque names like Hydra and Orthrus used to describe straightforward concepts that are essentially a neural classifier (of a particular form) and a seq2seq-based data augmentation procedure (which would be good to describe in language more familiar to the ML audience). In particular, the goals of Hydra and Orthrus should first be intuitively described before delving into their various components. - Why did the authors never evaluate the overall sentiment prediction performance of Orthrus + Hydra used together vs other classifiers + data augmentation strategies? - If the goal of dilation invariance is to help the classifiers better generalize to out-of-distribution test sentences, then why not verify this happened, eg. by training on Yelp and testing on Amazon? - The authors should better justify the assumption of Jalalzai et al, and why this is appropriate for the MLP classifier used later in the paper. - This statement needs to be clarified and have citation: *Such classifier whose output solely depends on the angle ?(x) of the considered input, with provable guarantees concerning the classification risk in out-of-sample regions scaling as the square root of the number of extreme points used at the training step* - The authors should explain Equation (1) in English rather than referring to it so early on the paper (pg 1: *satisfying Equation 1*). I had no idea what this was supposed to mean as a first time reader. - The way figure 4 is presented is a bit opaque and took me a while to understand (have to look closely at Fig 4a to see the columns are not monochromatic). - *We also compare Hydra to a Vanilla Sequence to Sequence to demonstrate the validity of our approach* How *Vanilla Sequence to Sequence* (word *model* is missing) is used for dataset augmentation needs to be clarified here. - A figure demonstrating an example of the phenomenon explained in Sec 2.2 would be helpful to aid reader*s intuition. - Typo: *eugmentation*
assistant
This paper proposes a new embedding method for sentences that aims to preserve dilation invariance. Much of the methodology is justified by results for extremal point classification under particular assumptions, and then the authors try and encourage these assumptions to be met via penalty terms introduced in their embeddings/augmentation models. However, while the proposed methodology seems interesting/novel, it remains conceptually unclear why it should be superior to standard text classification methods (ie. exactly what assumptions are being exploited to improve performance and how exactly do those assumptions help should be made more explicit). In particular, why is dilation invariance even a good idea? Overall, I find the paper a bit mathematically dense in Secs 2.1-2.2, which would not be a bad thing if the math were necessary to justify why the proposed methodology works well, but it in this case seems mainly presented as background material (as if it were a prerequisite to understand the method itself, which it is certainly not). Why not instead present an explicit theorem providing some statistical guarantees for the proposed methodology in Sec 3.1 based on the constant-along-rays result (which would be nice to have regardless), and then follow up the theorem with background math from 2.1-2.2 which is necessary to understand the proof? As it is currently written the paper is a bit too dense in terminology, and opaque names like Hydra and Orthrus used to describe straightforward concepts that are essentially a neural classifier (of a particular form) and a seq2seq-based data augmentation procedure (which would be good to describe in language more familiar to the ML audience). In particular, the goals of Hydra and Orthrus should first be intuitively described before delving into their various components. - Why did the authors never evaluate the overall sentiment prediction performance of Orthrus + Hydra used together vs other classifiers + data augmentation strategies? - If the goal of dilation invariance is to help the classifiers better generalize to out-of-distribution test sentences, then why not verify this happened, eg. by training on Yelp and testing on Amazon? - The authors should better justify the assumption of Jalalzai et al, and why this is appropriate for the MLP classifier used later in the paper. - This statement needs to be clarified and have citation: *Such classifier whose output solely depends on the angle ? (x) of the considered input, with provable guarantees concerning the classification risk in out-of-sample regions scaling as the square root of the number of extreme points used at the training step* - The authors should explain Equation (1) in English rather than referring to it so early on the paper (pg 1: *satisfying Equation 1*). I had no idea what this was supposed to mean as a first time reader. - The way figure 4 is presented is a bit opaque and took me a while to understand (have to look closely at Fig 4a to see the columns are not monochromatic). - *We also compare Hydra to a Vanilla Sequence to Sequence to demonstrate the validity of our approach* How *Vanilla Sequence to Sequence* (word *model* is missing) is used for dataset augmentation needs to be clarified here. - A figure demonstrating an example of the phenomenon explained in Sec 2.2 would be helpful to aid reader*s intuition. - Typo: *eugmentation*