Visual relationship recognition models are limited in the ability to generalize from finite seen predicates to unseen ones. We propose a new problem setting named predicate zero-shot learning (PZSL): learning to recognize the predicates without training data. It is unlike the previous zero-shot learning problem on visual relationship recognition which learns to recognize the unseen relationship triplets (<subject, predicate, object>) but requires all components (subject, predicate, and object) to be seen in the training set. For the PZSL problem, however, the models are expected to recognize the diverse even unseen predicates, which is meaningful for many downstream high-level tasks, like visual question answering, to handle complex scenes and open questions. The PZSL is a very challenging task since the predicates are very abstract and follow an extreme long-tail distribution. To address the PZSL problem, we present a model that performs compatibility learning leveraging the linguistic priors from the corpus and knowledge base. An unbalanced sampled-softmax is further developed to tackle the extreme long-tail distribution of predicates. Finally, the experiments are conducted to analyze the problem and verify the effectiveness of our methods. The dataset and source code will be released for further study.
Title: Good work, requires some edits.
1. Summarize:
This paper proposes a new problem setting in visual relation detection which is called “Predicate Zero-shot Learning (PZSL)”. They provide a clear motivation and description of this setting. They propose a solution to this problem which leverages linguistic priors and knowledge bases. Furthermore they propose an unbalanced sampled-softmax to tackle the long tail distribution of predicates.
2. Clearly state your decision. One or two key reasons for this choice.
I will go for a weak accept for the paper at this stage. (+) I think the proposed problem setting is well-motivated and useful. Also, (+) the proposed initial solution to this problem is interesting. However, (-) they propose a “fast graph convolution network” which seems to be precisely equivalent to a PinSage.  Also, (-) the paper requires to be polished as it lacks clarity.
3. Main discussion
My first argument is: I’m not sure why the authors have changed the name of PinSage and just mentioned that “their” “Fast Graph Convolution Network” is “inspired” from PinSage. To me it looks exactly the same. If there are any differences, it should be stated clearly. In fact, I would not be against using PinSage as a part of their approach. However, trying to rename it without clear reasons is not a good idea.
My second argument is that the paper lacks clarity in writing (for detailed suggestions please refer to comments and feedbacks). Specially the evaluation section lacks details and clarity: a) In the beginning of this section (page 7), the authors talk about “generalized” and “traditional” settings without properly defining them. b) The descriptions for Table 1 and Table 2 fail to provide enough details to help understand the difference between the results in these two tables (one of them states “Accuracy of unseen predicate recognition” and the other one “Accuracy of recognition of triplets with unseen predicates”). 
4. Comments and feedback.
Introduction: 
Paragraph one in the: 
1. The relationship recognition methods are mainly supervised “that” → “to”.
2. last line: …. and do not study “on generalizing” → “the generalization of”.
Paragraph two:
1. no manual annotations or “real samples” → “image samples”. (a real sample is ill-defined)
2. For example, no instance of chew → For example “given” no instance of chew.
Paragraph three:
1. … is difficult since predicates are often abstract not as specific →   is difficult since predicates are often abstract “and” not as specific.
2. Furthermore, unlike many object ZSL methods … → This line to the end is very complicated and hard to understand.
Related Works:
1. Visual Relationships: I would cite “Graph R-CNN for scene graph generation” since it is the most relevant work regarding the similarity of pipeline (using GCNs).
2. External Knowledge bases  (KB): I would cite “Improving Visual Relationship Detection using
Semantic Modeling of Scene Descriptions” since it is one of the most relevant works using knowledge graph modelings to improve visual relation detection.
Problem Setup:
Do you plan to provide the proposed dataset splits so others can work on this setting? I consider this very important given your paper’s contribution. Maybe it is better if it is also mentioned in the paper.
Pipeline:
1. Paragraph 2: … the output of which is fused with …: Given Figure 2, it does not seem like $V_p$ is being created by fusing $V_s$ and $V_o$. It looks more like it is extracted directly from the image (union of bounding boxes).
2. In Figure 2: In the representation of Pipeline (A), the graph is colored by dark blue for objects and light blue for predicates. The represented graphs show Object to Object and Predicate to Predicate connections which I’m not sure if it is correct. Shouldn’t we always have a light blue between every pair of dark blue connections?
Evaluation:
1. Please consider the mentioned points in the Main Discussions.
2. In Table 1, I suggest re-naming “embedding” to “initial embedding”.
3. In Table 1, Hit@k should be Hits@k.
4. Please define the metrics clearly (Hits@k).
Extra: I have a question regarding the ablation studies with GloVe, Normal and InferSent initialization. The question is whether this initialization is necessary? It seems like in the setting “W/O KG”, even though the embeddings are initialized with GloVe, there is no gain (all of the Hits@k values are 0.0). So GloVe embedding without KG bring no external semantic knowledge? Then why use them? Regarding that, I can see that a GCN, initialized with normally distributed embeddings (row 14 in Table 1) has given 0.0 accuracies, but I find this very counter intuitive, as graph convolution layers already have trainable weights capable of compensating for the lack of ‘proper’ initialized embedding and getting 0.0 does not make sense to me.
Conclusions and Future Work:
1. two lines before the last: please use “\citep”.