With a view to bridging the gap between deep learning and symbolic AI, we present a novel end-to-end neural network architecture that learns to form propositional representations with an explicitly relational structure from raw pixel data. In order to evaluate and analyse the architecture, we introduce a family of simple visual relational reasoning tasks of varying complexity. We show that the proposed architecture, when pre-trained on a curriculum of such tasks, learns to generate reusable representations that better facilitate subsequent learning on previously unseen tasks when compared to a number of baseline architectures. The workings of a successfully trained model are visualised to shed some light on how the architecture functions.
This paper presents PrediNet: an architecture explicitly designed to extract representations in the form of three-place predicates (relations). They evaluate the architecture on a visual relational task called the "Relations Game" which involves comparing Tetris-like shapes according to their appearance, relative positions, etc.. They show that their architecture leads to useful generalizable representations in the sense that they can be used for new tasks without retraining.
I think this paper contains a number of unusual and interesting ideas but is let down by its presentation. The writing is good, but provides very little intuition for why we should expect this approach to work (aside from its connection to equation 1) - I discuss this in more depth below. The experimental task is interesting (I'm okay with synthetic tasks of this form for unusual new architectures like this), but I'm not sure what it tests that isn't tested in the CLEVR and sort-of -CLEVR datasets which rely on similar relational reasoning to solve. The advantage of those datasets is they are well-established with strong baselines so we can be more certain that a fair comparison is been made. I've voted to reject this paper because I feel its premature in its current form.
Expanding on this - The description of PrediNet covers the basics, but missing detail and intuition for why certain choices are made. For example, 
 - why is L flattened for the queries (I think it’s because the query is independent of pixel location, but flattening seems of when L also includes co-ordinates) but not the key, K? 
 - Why is the key space shared between heads (this seems more intuitive - keys should have consistent meaning… but if that’s the case, make that intention explicit)? 
 - Also, writing the dimensionality of the matrices, would help (e.g. is W_S in R^{m x j} or R^{m x 1} or something else?). 
 - What is the meaning of the position features in E_1 and E_2? From the softmax product it seems they should be a weighted sum of the pixels that are addressed - implying that it is the weighted average location?
 - The final representation mostly consists of an h x (j) vector (ignoring positions) containing the output of the comparators. Could you provide some intuition for why we would expect such a representation to be useful for the downstream task? This representation seems to differ substantially from what is used in the baseline methods: i.e. attention-weighted sums of the input features.