Sequential word order is important when processing text. Currently, neural networks (NNs) address this by modeling word position using position embeddings. The problem is that position embeddings capture the position of individual words, but not the ordered relationship (e.g., adjacency or precedence) between individual word positions. We present a novel and principled solution for modeling both the global absolute positions of words and their order relationships. Our solution generalizes word embeddings, previously defined as independent vectors, to continuous word functions over a variable (position). The benefit of continuous functions over variable positions is that word representations shift smoothly with increasing positions. Hence, word representations in different positions can correlate with each other in a continuous function. The general solution of these functions is extended to complex-valued domain due to richer representations. We extend CNN, RNN and Transformer NNs to complex-valued versions to incorporate our complex embedding (we make all code available). Experiments on text classification, machine translation and language modeling show gains over both classical word embeddings and position-enriched word embeddings. To our knowledge, this is the first work in NLP to link imaginary numbers in complex-valued representations to concrete meanings (i.e., word order).
### Summary
The authors present a "natural" way of encoding position information into word embeddings and present extensive empirical evidence to support their method. I believe that paper meets the bar for acceptance.
### Details
The paper "Encoding word order in complex embeddings" presents a method for making word embeddings position dependent. The idea in a nutshell is to map each discrete position , n, to a value  `   A exp( freq_{word, dim} × n )` . So a word embedding is a collection of complex valued signals sampled at discrete points.
The frequency is dependent on each word and each dimension in general. The authors motivate/justify this particular formulation via their Claim 1, which argues that their particular formulation uniquely satisfies two intuitive constraints. Although one of those constraints (i.e. linearly witnessed Position-free offset) almost completely specifies the solution.
The experiments in the paper are fairly thorough and cover text classification, machine translation and language modeling. Through the comparative experiments the complex embeddings we can see that the formulation in this paper outperforms existing SOTA methods, sometimes with significantly difference such as a difference of 1.3 BLEU point on the MT task. 
I would have liked to say that the ablation are similarly conclusive but there seem to be a problem in the table, which eroded my confidence:
1. The number of parameters in rows 5 and 8  (w/t encoding positions, share / not-share respectively) are reported to be 9.38M and 8.33M which has to be wrong. Similar problem happens with other pairs. And now I am not sure whether the results were also swapped or not. Still the results in general trend in the right direction.
### Possible improvements to the paper
1. The main weakness of the paper is that the authors repeatedly mention that encoding the position as a multiplicative factor which is multiplied to the frequency gives leads to a more decoupled/interpretable embedding but their experiments are solely focused on accuracy measurement. I would have liked to see the authors carry out more experiments to see whether the frequency parameters really are interpretable? For example, 
      -  What is the histogram of the frequencies ? Are some of them negative? 
      - Which word has the highest frequencies (pooled over all dimensions) in absolute term? Does it make sense that that word's meaning is so position dependent? For example, positions can capture subjects versus objects in english, but they will more reliably reflect the subject versus verb distinction in hindi. 
      - Are the word frequencies by themselves predictive of anything? For example, what happens if the word embedding amplitudes are tied across words or dimensions? We expect the performance to be bad but how bad? 
These kinds of ablations  / qualitative analysis will really make the paper more informative and interesting. Right now it just seems like yet another paper where the capacity of the model is increased and the accuracy increases. Specially because the delta improvement over the fixed positional embeddings of (Transformer-TPE Vaswani et al. 2017) is so limited.
### Edit after the author response
The authors have made the required corrections and added the necessary analysis. One interesting outcome was that the "word-sharing amplitude schema" seems to drop so little in performance, it's almost like all the information can be coded in just the phase vectors alone. It will be nice if the authors release their trained phase embeddings as well, for the words.