The dominant approaches to sentence representation in natural language rely on learning embeddings on massive corpuses. The obtained embeddings have desirable properties such as compositionality and distance preservation (sentences with similar meanings have similar representations). In this paper, we develop a novel method for learning an embedding enjoying a dilation invariance property. We propose two algorithms: Orthrus, a classification algorithm, constrains the distribution of the embedded variable to be regularly varying, i.e. multivariate heavy-tail. and uses Extreme Value Theory (EVT) to tackle the classification task on two separate regions: the tail and the bulk. Hydra, a text generation algorithm for dataset augmentation, leverages the invariance property of the embedding learnt by Orthrus to generate coherent sentences with controllable attribute, e.g. positive or negative sentiment. Numerical experiments on synthetic and real text data demonstrate the relevance of the proposed framework.
The paper presented two methods for augmenting sentiment classification from the perspective of applying the Extreme Value Theory (EVT), including:
1) A classification algorithm which has an adversarial classifier to enforce the intermediate representations of a neural network to be similar to one EVT distribution, logistic distribution;
2) An encoder-decoder model that is able to generate grammatically coherent sentences with the same sentiment as the given input sentence.
Questions:
--
1) the Fisher–Tippett–Gnedenko theorem states that it is possible that the maximum value of a set of iid samples converges to one of three plausible distributions, and the chosen logistic distribution falls into the Weibull distribution category. I have a couple concerns about this choice:
1.1) In order to show that the EVT indeed helps empirically in the way that an adversarial classifier enforces the inf-norm of vectors follow the Generalised Extreme Value (GEV) distribution, at least three plausible distributions from each form of the GEV distribution needs to be checked. The logistic distribution is interesting, but the marginal improvement gained by enforcing the lengths of the produced vectors to follow the logistic distribution could be a result of hyper-param tuning, which shouldn't be a piece of supporting evidence.
1.2) From the perspective of applying the EVT, recent successful work from the best of my knowledge is on Anomaly Detection [1], where the EVT enables the system to learn from samples in only one class and also adjust the threshold for detecting the abnormal behaviour of samples. It is also theoretically grounded as the error variable of a logistic regression follows a Gumbel distribution which is one form of the GEV distribution, therefore, applying EVT for binary classification case makes sense.
1.3) From the perspective of learning representations with structured priors, there exists an interesting work on decomposing vector representations into lengths and directions and enforcing lengths to follow a uniform distribution and directions a Von Mises–Fisher (vMF) distribution as in [2]. It would be interesting to see if the proposed method is indeed better than the way that structured priors are enforced in [2].
1.4) Linguistically, given the distributional hypothesis, the length of learnt vectors tends to be highly correlated with the frequency information of available concepts and the direction of them matters more. The argument is also presented by the paper. However, in sentiment analysis, the length could contain the information about how strong the sentiment of the input sentence is, so I am not convinced that the proposed method would be applicable in fine-grained sentiment analysis, such as Stanford Sentiment Treebank [3]. 
--
2) A soft approximation over the inf-norm of a set of iid samples is log-sum-exp function, and it is the cdf of softmax function, which is also theoretically grounded in EVT for classifications. It could be a nicer story than the current one as the choice of the logistic distribution seems to be too intend. 
--
3) The construction of the two datasets seems to be very arbitrary given that there exists a large number of sentiment analysis datasets and many with lots of samples, I am not sure that the results on the chosen constructed two datasets are sufficient enough to support the claim.
3.1) The size of the datasets is too small. Given that, the marginal improvement against the NN baseline could be a result of a specific initialisation, which doesn't generalise to other random initialisations.
3.2) The dimension of vector representations is also too small. Normally, commonly used word embeddings are of 300 dimensions, and contextualised ones are of higher than 1200 dimensions. The chosen 50 dimension could prevent the NN baseline model to perform well and IMO, it is helpful for picking a suitable logistic prior than it is in a very high dimensional space.
3.3) There are many straightforward distributions that could be applied as a prior on the lengths of vector representations, e.g. the Rayleigh distribution in 2D and the Chi-squared distribution in higher-dimension. Then again, the distribution gets flatter and becomes similar to a uniform distribution when the dim goes higher, which is a common issue. It goes back to my concern or doubt on the usability of a prior on the norm of high dimensional vectors.
--
4) I am still interested in seeing EVT being applied in various domains, but I'd be in favor of more justifiable approaches. 
[1] Siffer, Alban, et al. "Anomaly detection in streams with extreme value theory." Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2017.
[2] Guu, Kelvin, et al. "Generating sentences by editing prototypes." Transactions of the Association for Computational Linguistics 6 (2018): 437-450.
[3] Socher, Richard, et al. "Recursive deep models for semantic compositionality over a sentiment treebank." Proceedings of the 2013 conference on empirical methods in natural language processing. 2013.