With interest in program synthesis and similarly flavored problems rapidly increasing, neural models optimized for tree-domain problems are of great value. In the sequence domain, transformers can learn relationships across arbitrary pairs of positions with less bias than recurrent models. Under the intuition that a similar property would be beneficial in the tree domain, we propose a method to extend transformers to tree-structured inputs and/or outputs. Our approach abstracts transformer's default sinusoidal positional encodings, allowing us to substitute in a novel custom positional encoding scheme that represents node positions within a tree. We evaluated our model in tree-to-tree program translation and sequenceto-tree semantic parsing settings, achieving superior performance over the vanilla transformer model on several tasks.
The paper describes an interesting idea for using Vashwani's transformer with tree-structured data, where nodes' positions in the tree are encoded using unique affine transformations. They test the idea in several program translation tasks, and find small-to-medium improvements in performance. 
Overall the idea is promising, but the work isn't ready for publication. The implementation details weren't easy to follow, the experiments were narrow, and there are key citations missing. I would recommend trying some more diverse tasks, and putting this approach against other graph neural network techniques.
REVISED:
I've revised by review upwards by 1, though I still recommend rejection. The authors improved the scholarship by adding many more citations and related work. They also made the model details and implementation more clear. 
The remaining problem I see is that the results are just not that compelling, and the experiments do not test any other graph neural network architectures.
Specifically, in Table 1 (synthetic experiments) the key result is that their tree-transformer outperforms seq-transformer on structured input. But seq-transformer is best on raw programs. I'm not sure what to make of this. But I wouldn't use tree-transformer in this problem. I'd use seq-transformer.
In Table 2 (CoffeeScript-JavaScript experiments), no seq-transformer results are presented. That seems... suspicious. Did the authors try those experiments? What were the results? I'd definitely like to see them, or an explanation of why they're not shown. This paper tests whether tree-transformers are better than seq-transformer and other seq/tree models, but this experiment's results do not address that fully. Of the 8 tasks tested, tree-transformer is best on 5/8 while tree2tree is best on 3/8. 
In Table 3, there's definitely a moderate advantage to using tree-transformer over seq-transformer, but in 5/6 of the tasks tree-transformer is worse than other approaches. The authors write, "Transformer architectures in general, however, do not yet compete with state-of-the-art results.". 
Finally, no other graph neural network/message-passing/graph attention architectures are tested (eg. Li et al 2016 was cited but not tested, and Gilmer et al 2017 and Veličković et al 2017 weren't cited or tested), but there's a reasonable chance they'd outperform the tree-transformer.
So overall the results are intriguing, and I believe there's something potentially valuable here. But I'm not sure there's sufficient reason presented in the paper to use tree-transformer over seq-transformer or other seq/tree models. Also, while the basic idea is nice, as I understand it is restricted to trees, so other graphical structures wouldn't be handled.