INTERACTIVE GROUNDED LANGUAGE ACQUISITION AND GENERALIZATION IN A 2D WORLD
We build a virtual agent for learning language in a 2D maze-like world. The agent sees images of the surrounding environment, listens to a virtual teacher, and takes actions to receive rewards. It interactively learns the teacher's language from scratch based on two language use cases: sentence-directed navigation and question answering. It learns simultaneously the visual representations of the world, the language, and the action control. By disentangling language grounding from other computational routines and sharing a concept detection function between language grounding and prediction, the agent reliably interpolates and extrapolates to interpret sentences that contain new word combinations or new words missing from training sentences. The new words are transferred from the answers of language prediction. Such a language ability is trained and evaluated on a population of over 1.6 million distinct sentences consisting of 119 object words, 8 color words, 9 spatial-relation words, and 50 grammatical words. The proposed model significantly outperforms five comparison methods for interpreting zero-shot sentences. In addition, we demonstrate human-interpretable intermediate outputs of the model in the appendix.
[Overview]
In this paper, the authors proposed a unified model for combining vision, language, and action. It is aimed at controlling an agent in a virtual environment to move to a specified location in a 2D map, and answer user's questions as well. To address this problem, the authors proposed an explicit grounding way to connect the words in a sentence and spatial regions in the images. Specifically, By this way, the model could exploit the outputs of concept detection module to perform the actions and question answering as well jointly. In the experiments, the authors compared with several previous attention methods to show the effectiveness of the proposed concept detection module and demonstrated its superiority on several configurations, including in-domain and out-of-domain cases.
[Strengths]
1. I think this paper proposed interesting tasks to combine the vision, language, and actions. As we know, in a realistic environment, all three components are necessary to complete a complex tasks which need the interactions with the physical environments. The authors should release the dataset to prompt the research in this area.
2. The authors proposed a simple method to ground the language on visual input. Specifically, the authors grounded each word in a sentence to all locations of the visual map, and then perform a simple concept detection upon it. Then, the model used this intermediate representation to guide the navigation of agent in the 2D map and visual question answering as well.
3. From the experiments, it is shown that the proposed model outperforms several baseline methods in both normal tasks and out-of-domain ones. According to the visualizations, the interpreter could generate meaningful attention map given a textual query.
[Weakness]
1. The definition of explicit grounding is a bit misleading. Though the grounding or attention is performed for each word at each location of the visual map. It is a still kind of soft-attention, except that is performed for each word in a sentence. As far as I know, this has been done in several previous works, such as: (a). Hierarchical question-image co-attention for visual question answering (https://scholar.google.com/scholar?oi=bibs&cluster=15146345852176060026&btnI=1&hl=en). Lu et al. NIPS 2016. (b). Graph-Structured Representations for Visual Question Answering. Teney et al. arXiv 2016. At most recent, we have seen some more explicit way for visual grounding like: (c). Bottom-up and top-down attention for image captioning and VQA (https://arxiv.org/abs/1707.07998). Anderson et al. arXiv 2017.
2. Since the model is aimed at grounding the language on the vision based on interactions, it is worth to show how well the final model could ground the text words to each of the visual objects. Say, show the affinity matrix between the words and the objects to indicate the correlations.
[Summary]
I think this is a good paper which integrates vision, language, and actions in a virtual environment. I would foresee more and more works will be devoted to this area, considering its close connection to our daily life. To address this problem, the authors proposed a simple model to ground words on visual signals, which prove to outperform previous methods, such as CA, SAN, etc. According to the visualization, the model could attend the right region of the image for finishing a navigation and QA task. As I said, the authors should rephrase the definition of explicit grounding, to make it clearly distinguished with the previous work I listed above. Also, the authors should definitely show the grounding attention results of words and visual signal jointly, i.e., showing them together in one figure instead of separately in Figure 9 and Figure 10.