VISUAL SEMANTIC NAVIGATION USING SCENE PRIORS
Coffee machine
Figure 1: Our goal is to use scene priors to improve navigation in unseen scenes and towards novel objects. (a) There is no mug in the field of view of the agent, but the likely location for finding a mug is the cabinet near the coffee machine. (b) The agent has not seen a mango before, but it infers that the most likely location for finding a mango is the fridge since similar objects such as apple appear there as well. The most likely locations are shown with the orange box.
How do humans navigate to target objects in novel scenes? Do we use the semantic/functional priors we have built over years to efficiently search and navigate? For example, to search for mugs, we search cabinets near the coffee machine and for fruits we try the fridge. In this work, we focus on incorporating semantic priors in the task of semantic navigation. We propose to use Graph Convolutional Networks for incorporating the prior knowledge into a deep reinforcement learning framework. The agent uses the features from the knowledge graph to predict the actions. For evaluation, we use the AI2-THOR framework. Our experiments show how semantic knowledge improves performance significantly. More importantly, we show improvement in generalization to unseen scenes and/or objects.
This paper tackles the problem of navigating scenes to find objects which are potentially not included in the training phase. To find an unseen object from a scene, the proposed model incorporates an external knowledge graph as an augmented input of the actor-critic model. To construct a knowledge graph, entities in a scene are identified by ResNet and then the link structure between entities are extracted from VIsual Genome dataset. Through the ablation study, it is shown that using the knowledge graph helps to track and identify unseen objects during training.
- The original knowledge graph (KG) has relation labels (such as next to, on in figure 3) between different objects, however, GCN does not take into account the relations between objects. Only co-occurrence patterns will be encoded into the KG constructed from an image. There are more complex graph convolutional models modelling relations between nodes such as [1]. Have you considered adding explicit relations between entities? will it increase the navigation performance? if not why?
- It is unclear how many objects are used to construct a KG from an image. For example, are top-k objects identified by ResNet used to construct a KG?
- Description of the reward is a bit unclear as well, especially when the model is trained without stop action. From the text, the agents receive a positive reward when it is close to the target (within a certain number of steps). Does this mean that the agent gets a positive reward on every step near the target while it's not in the final state?
- This might be a trivial question, but I couldn't find it from the text. Can you find all object from AI2-THOR in the categories of ImageNet and of Visual Genome? is there any information loss while constructing a KG from the classification result? What is the average number of nodes of a KG? and is there any correlation between the size of KG and the result?
- Why are the performances of the models is unstable with Bedroom dataset (in terms of variance)? 
- The input feature of GCN is a combination of word feature and image feature. It is clear that there is a corresponding word embedding for each of the identified objects, but it is unclear what is the corresponding image feature. If two objects are identified in the same frame, do input features of these two objects share the same image features from Resnet?
[1] Schlichtkrull, Michael, et al. "Modeling relational data with graph convolutional networks." European Semantic Web Conference. Springer, Cham, 2018.