A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input
We propose a method for automatically answering questions about images by bringing together recent advances from natural language processing and computer vision. We combine discrete reasoning with uncertain predictions by a multiworld approach that represents uncertainty about the perceived world in a bayesian framework. Our approach can handle human questions of high complexity about realistic scenes and replies with range of answer like counts, object classes, instances and lists of them. The system is directly trained from question-answer pairs. We establish a first benchmark for this task that can be seen as a modern attempt at a visual turing test.
This paper describes a dataset and system for answering natural language questions about a depth/RGB image, which uses multiple segmentations to capture uncertainty in the scene understanding when computing answers.Quality=======This is an extremely difficult task which requires understanding natural language forms of queries, segmenting and categorising the depth image, understanding the scene layout in three dimensions (including some understanding of support) and bringing all these pieces together.  The state of the art in each part individually is not currently close to having good accuracy - one concern is that it is premature to try and tie these systems together to solve an even harder problem.  The worry is that errors will just accumulate from system to system - although it certainly helps that some uncertainty in the segmentation is communicated though samples (multiple worlds).  This concern seems borne out by the fact that the overall system is giving results only slightly better than always picking the same label all the time (7.86% vs 4.4%).  I wonder how much better the system is doing than a colour/word histogram-based straw man?  With just 4 examples (Fig 6) It's hard to tell the kinds of mistakes the system is making and which piece of the system is the cause, but the fact that it is getting the type wrong (reporting a colour in answer to "what is behind the television?") suggests that there's room for improvement in the question-answering part alone.  The WUPS evaluation metric seems to have some issues - the fact that humans doing so badly (61% at 0.9) seems more likely to be an issue with the metric than with the humans.  I'm not quite sure what the sentence "Despite our instructions the answers are subject to... human judgement" is implying! It might be interesting to explore an evaluation metric that says whether a human thinks the automatically-given answer is correct or not - or at least use this to validate the WUPS metric.Clarity=======Overall acceptably clear, but with a lot of missing details which require reading [1] or are just not present e.g. the definition of the 'sampling strategy' used to produce segmentation samples.  The figures could use refinement e.g. the segmentation images lack a key, the text in Fig 6 is hard to read.  The authors should also say how the examples in Fig 6 were selected (at random? are these typical?).  The introduction could use some references for the methods being referred to as could the second sentence of section 3.Originality & significance==========================The question/answer dataset which augments the existing NYU depth/RGB image dataset is a useful contribution.  Given the extreme difficulty of the task and the issues with the individual components, the contribution of combining them is mainly to get a straw man accuracy number.Typos: Pg 2 "[4] objects" - words missingBottom of pg 4 "words"->"worlds"Refs: many capitalization issues.