Human world knowledge is both structured and flexible. When people see an object, they represent it not as a pixel array but as a meaningful arrangement of semantic parts. Moreover, when people refer to an object, they provide descriptions that are not merely true but also relevant in the current context. Here, we combine these two observations in order to learn fine-grained correspondences between language and contextually relevant geometric properties of 3D objects. To do this, we employed an interactive communication task with human participants to construct a large dataset containing natural utterances referring to 3D objects from ShapeNet in a wide variety of contexts. Using this dataset, we developed neural listener and speaker models with strong capacity for generalization. By performing targeted lesions of visual and linguistic input, we discovered that the neural listener depends heavily on part-related words and associates these words correctly with the corresponding geometric properties of objects, suggesting that it has learned task-relevant structure linking the two input modalities. We further show that a neural speaker that is 'listener-aware' -that plans its utterances according to how an imagined listener would interpret its words in context -produces more discriminative referring expressions than an 'listener-unaware' speaker, as measured by human performance in identifying the correct object.
#update: I've read the authors comments but unfortunately my main concerns about the contributions and novelty of this work are not answered. As such, I cannot increase my score.
------------------ 
The authors provide a study on learning to refer to 3D objects. The authors collect a dataset of referential expressions and train several models by experimenting with a number of architectural choices.
This is an interesting study reporting results on the effect that several architectural choices have generating referential expressions. Overall, while I appreciate all the experiments and results, I don't really feel I've learned something from this paper. 
First and foremost, the paper, from the title already starts to build up expectations about the 3d nature of the study, however this is pretty much ignored at the rest of the paper. I would expect the paper to provide  some results and insights regarding the 3D nature of the dataset and how this affects referential expressions, however, there is no experiment that has used this 3d-ness in any way. Even the representations of the objects are stripped down to essentially 2D (a single-view of a 3D object used to derived VGG features is as 3D as any image dataset used for similar studies, right?). 
My major question is then: why should all this research take place in a 3D dataset? Is it to validate that research like this is at all possible with 3D objects? 
Moreover, all interesting aspects of referential expressions are stripped out since the authors experiment only with this geometric visual property (which has again nothing to do with 3d-ness, you could totally get that out of images). An interesting study would be to have all objects in the same image and have referential expressions that have to do with spatial expressions, something that the depth or a different view of the of the object could play a role.
Given the fact that there are no technical innovations, I can't vouch for accepting this paper, since there has been quite a lot of research on generating  referential expressions on image datasets (e.g., Kazemzadeh., 2014 and related papers). However, learning to refer to 3D objects is a very interesting topic, and of great importance given the growing interest of training agents in 3D virtual environments, and I would really encourage the authors to embrace the 3d-ness of objects and design studies that highlight the challenges and opportunities that the third dimension brings.
Kazemzadeh et al.: ReferIt Game: Referring to Objects in Photographs of Natural Scenes