The robust and efficient recognition of visual relations in images is a hallmark of biological vision. Here, we argue that, despite recent progress in visual recognition, modern machine vision algorithms are severely limited in their ability to learn visual relations. Through controlled experiments, we demonstrate that visual-relation problems strain convolutional neural networks (CNNs). The networks eventually break altogether when rote memorization becomes impossible such as when the intra-class variability exceeds their capacity. We further show that another type of feedforward network, called a relational network (RN), which was shown to successfully solve seemingly difficult visual question answering (VQA) problems on the CLEVR datasets, suffers similar limitations. Motivated by the comparable success of biological vision, we argue that feedback mechanisms including working memory and attention are the key computational components underlying abstract visual reasoning.
The authors introduce a set of very simple tasks that are meant to illustrate the challenges of learning visual relations.  They then evaluate several existing network architectures on these tasks, and show that results are not as impressive as others might have assumed they would be.   They show that while recent approaches (e.g. relational networks) can generalize reasonably well on some tasks, these results do not generalize as well to held-out-object scenarios as might have been assumed. 
Clarity:  The paper is fairly clearly written.   I think I mostly followed it.   
Quality:  I'm intrigued by but a little uncomfortable with the generalization metrics that the authors use.   The authors estimate the performance of algorithms by how well they generalize to new image scenarios when trained on other image conditions.   The authors state that ". . . the effectiveness of an architecture to learn visual-relation problems should be measured in terms of generalization over multiple variants of the same problem, not over multiple splits of the same dataset."  Taken literally, this would rule out a lot of modern machine learning, even obviously very good work. On the other hand, it's clear that at some point, generalization needs to occur in testing ability to understand relationships.  I'm a little worried that it's "in the eye of the beholder" whether a given generalization should be expected to work or not.     
There are essentially three scenarios of generalization discussed in the paper:
        (a) various generalizations of image parameters in the PSVRT dataset
        (b) various hold-outs of the image parameters in the sort-of-CLEVR dataset
        (c) from sort-of-CLEVR "objects" to PSVRT bit patterns
The result that existing architectures didn't do very well at these generalizations (especially b and c) *may* be important -- or it may not.    Perhaps if CNN+RN were trained on a quite rich real-world training set with a variety of real-world three-D objects beyond those shown in sort-of-CLEVR, it would generalize to most other situations that might be encountered.    After all, when we humans generalize to understanding relationships, exactly what variability is present in our "training sets" as compared to our "testing" situations?   How do the authors know that humans are effectively generalizing rather than just "interpolating" within their (very rich) training set?  It's not totally clear to me that if totally naive humans (who had never seen spatial relationships before) were evaluated on exactly the training/testing scenarios described above, that they would generalize particularly well either.   I don't think it can just be assumed a priori that humans would be super good this form of generalization.  
So how should authors handle this criticism?  What would be useful would either be some form of positive control.  Either human training data showing very effective generalization (if one could somehow make "novel" relationships unfamiliar to humans), or a different network architecture that was obviously superior in generalization to CNN+RN. If such were present, I'd rate this paper significantly higher.    
Also, I can't tell if I really fully believe the results of this paper.  I don't doubt that the authors saw the results they report.  However, I think there's some chance that if the same tasks were in the hands of people who *wanted* CNNs or CNN+RN to work well, the results might have been different.   I can't point to exactly what would have to be different to make things "work", because it's really hard to do that ahead of actually trying to do the work.   However, this suspicion on my part is actually a reason I think it might be *good* for this paper to be published at ICLR.  This will give the people working on (e.g.) CNN+RN somewhat more incentive to try out the current paper's benchmarks and either improve their architecture or show that the the existing one would have totally worked if only tried correctly.  I myself am very curious about what would happen and would love to see this exchange catalyzed. 
Originality and Significance:  The area of relation extraction seems to me to be very important and probably a bit less intensively worked on that it should be.  However, as the authors here note, there's been some recent work (e.g. Santoro 2017) in the area.   I think that the introduction of baselines  benchmark challenge datasets such as the ones the authors describe here is very useful, and is a somewhat novel contribution.