UNIVERSAL TRANSFORMERS
Recurrent neural networks (RNNs) sequentially process data by updating their state with each new data point, and have long been the de facto choice for sequence modeling tasks. However, their inherently sequential computation makes them slow to train. Feed-forward and convolutional architectures have recently been shown to achieve superior results on some sequence modeling tasks such as machine translation, with the added advantage that they concurrently process all inputs in the sequence, leading to easy parallelization and faster training times. Despite these successes, however, popular feed-forward sequence models like the Transformer fail to generalize in many simple tasks that recurrent models handle with ease, e.g. copying strings or even simple logical inference when the string or formula lengths exceed those observed at training time. We propose the Universal Transformer (UT), a parallel-in-time self-attentive recurrent sequence model which can be cast as a generalization of the Transformer model and which addresses these issues. UTs combine the parallelizability and global receptive field of feed-forward sequence models like the Transformer with the recurrent inductive bias of RNNs. We also add a dynamic per-position halting mechanism and find that it improves accuracy on several tasks. In contrast to the standard Transformer, under certain assumptions UTs can be shown to be Turing-complete. Our experiments show that UTs outperform standard Transformers on a wide range of algorithmic and language understanding tasks, including the challenging LAMBADA language modeling task where UTs achieve a new state of the art, and machine translation where UTs achieve a 0.9 BLEU improvement over Transformers on the WMT14 En-De dataset.
My summary: A new model, the UT, is based on the Transformer model, with added recurrence and dynamic halting of the recurrence. The UT should unite the computational universality properties of Neural Turing Machines and Neural GPU with good performance on disparate language and algorithmic tasks.
(I have read your author feedback and have modified my rating according to my understanding.)
Review:
The paper is well written and proofread, concrete and clear. The model is quite clearly explained, especially with the additional space of the supplementary material, appendices A and B (note fig 4 is less good quality than fig 2 for some reason) -- I’m fine with the use of the Supp Mat for this purpose.
 
The experiments have been conducted well, and demonstrate a wide range of tasks, which seems to suggest that the UT has pretty general purpose. The range of algorithmic tasks is limited, e.g. compared to the NTM paper.
I miss any experimental details at all on training.
I miss a comparison to Neural GPU and Stack RNN in 3.1, 3.2.
I miss a proof that the UT is computationally equivalent to a Turing machine. It does not have externally addressable, shared memory like a tape, and I’m not sure how to transpose read/write heads either.
The argument that the UT offers a good balance between inductive bias and expressivity is weak, though it may be the best one can hope for of a statistical model in a way. I note that in 3.1, the Transformer overfits, while it seems to underfit in 3.3 (lower LM and RC accuracy, higher LM perplexity), while the UT fare well, which suggests that the UT hits the balance better than the Transformer, at least.
From the point of view of network structure, it seems natural to lift further constraints on the model: 
why should width of intermediate layers be exactly equal to sequence length?
why should all hidden state vectors be size $d$, the size of the embeddings chosen at the first layer, which might be chosen out of purely practical reasons like the availability of pre-trained word embeddings?
What is the contribution of this work? It starts from the Transformer, the ACT idea for dynamic halting in recurrent nets, the need for models fit for algorithmic tasks. 
The UT’s building blocks are near-identical to the Transformers (and the paper is upfront and does a good job of explaining these similarities, fortunately)
- cf eq1-5: residuals, multi-headed self attention, and layer norm around all this. 
- shared weights among all such units
- encoder-decoder architecture
- autoregressive decoder with teacher forcing
- decoder units like the encoder’s but with extra layer of attention to final output of encoder
- coordinate embeddings
The authors may correct me, but I believe that the UT with FC layers is exactly identical to the Transformer described in Vaswani 2017 for T=6. 
So this paper introduces the idea of varying T, interprets it as a form of recurrence, and adds dynamic halting with ACT to that. Interestingly, the recurrence is not over sequence positions here.
This contribution is not major, on the other hand the experimental validation suggests the model is promising.
Typos and writing suggestions
above eq 8: masked such that -> masked so that
eq 8: dimensions of O and H^T are incompatible: d*V, m*d; to evacuate the notation issue for transposition, cf footnote 1, here and elsewhere, you could use either ${^t A}$ or $A^\top$ or $A^\intercal$. You could also write $t=T$ instead of just $T$.
sec3.3 line -1: designed such that -> designed so that
Towards the beginning of the paper, it may be useful to stabilise terminology for $t$: depth (as opposed to width for $m$), time steps, recurrence dimension, revisions, refinements