Word inputs tend to be represented as single continuous vectors in deep neural networks. It is left to the subsequent layers of the network to extract relevant aspects of a word's meaning based on the context in which it appears. In this paper, we investigate whether word representations can be improved by explicitly incorporating the idea of latent roles. That is, we propose a role interaction layer (RIL) that consists of context-dependent (latent) role assignments and rolespecific transformations. We evaluate the RIL on machine translation using two language pairs (En-De and En-Fi) and three datasets of varying size. We find that the proposed mechanism improves translation quality over strong baselines with limited amounts of data, but that the improvement diminishes as the size of data grows, indicating that powerful neural MT systems are capable of implicitly modeling role-word interaction by themselves. Our qualitative analysis reveals that the RIL extracts meaningful context-dependent roles and that it allows us to inspect more deeply the internal mechanisms of state-of-the-art neural machine translation systems.
[Summary]
This paper proposes “a role interaction layer” (briefly, RIL) that consists of context-dependent (latent) role assignments and role-specific transformations: Given an RIL layer, different dimensions of an embedding vector are “interacted” based on Eqn. (5), Eqn. (7), etc. The authors work on IWSLT De->En and WMT En->De, En->Fi to verify their proposed algorithm with case study included. 
[Pros]
(+) I think the idea/thought of using a “role interaction layer” is interesting.  The case study in Section 5.3 demonstrates different “roles”. Also, different RIL architectures are designed.
(+) The paper is easy to follow.
[Cons & Details]
(1) As stated in the abstract, “…, but that the improvement diminishes as the size of data grows, indicating that powerful neural MT systems are capable of implicitly modeling role-word interaction by themselves…” (1) The main concern is that, considering RIL does not obtain significant gain on large datasets, then we cannot say that the proposed algorithm is better than the baseline. (2) Why the NMT systems trained on large dataset can “implicitly modeling role-word interaction”, while small dataset cannot? Any analysis?
(2) For the “matched baseline”, page 5, you increase the dimensionality of the models. But an RIL is an additional layer, which makes the network deeper. Therefore, a baseline with an additional layer should be implemented.