We introduce FigureQA, a visual reasoning corpus of over one million questionanswer pairs grounded in over 100, 000 images. The images are synthetic, scientific-style figures from five classes: line plots, dot-line plots, vertical and horizontal bar graphs, and pie charts. We formulate our reasoning task by generating questions from 15 templates; questions concern various relationships between plot elements and examine characteristics like the maximum, the minimum, area-under-the-curve, smoothness, and intersection. To resolve, such questions often require reference to multiple plot elements and synthesis of information distributed spatially throughout a figure. To facilitate the training of machine learning systems, the corpus also includes side data that can be used to formulate auxiliary objectives. In particular, we provide the numerical data used to generate each figure as well as bounding-box annotations for all plot elements. We study the proposed visual reasoning task by training several models, including the recently proposed Relation Network as strong baseline. Preliminary results indicate that the task poses a significant machine learning challenge. We envision FigureQA as a first step towards developing models that can intuitively recognize patterns from visual representations of data.
Summary:
The paper introduces a new visual reasoning dataset called Figure-QA which consists of 140K figure images and 1.55M QA pairs. The images are generated synthetically by plotting perturbed sampled data using a visualization tool. The questions are also generated synthetically using 15 templates. Performance of baseline models and humans show that it is a challenging task and more advanced models are required to solve this task.
Strengths:
— FigureQA can help in developing models that can extract useful information from visual representations of data.
— Since performance on CLEVR dataset is already close to 100%, more challenging visual reasoning datasets would encourage the community to develop more advanced reasoning models. One of such datasets can be FigureQA.
— The paper is well written and easy to follow.
Weaknesses:
— Since the dataset is created synthetically, it is not clear if it is actually visual reasoning which is needed to solve this task, or the models can exploit biases (not necessarily language biases) to perform well on this dataset. In short, how do we know if the models trained on this dataset are actually learning something useful? One way to ensure this would be to show that models trained on this dataset can perform well on some other task. The first thing to try to show the usefulness of FigureQA is to show that the models trained on FigureQA dataset perform well on a real (figure, QA) dataset.
— The only advantages mentioned in the paper of using a synthetic dataset for this task are having greater control over task’s complexity and enabling auxiliary supervision signals, but none of them are shown in this paper, so it’s not clear if they are needed or useful.
— The paper should discuss what type of abilities are required in the models to perform well on this task, and how these abilities are currently not studied in the research community. Or in short, what new challenges are being introduced by FigureQA and how should researchers go about solving them on a high level?
— With what goal were these 15 types of questions chosen? Are these the most useful questions analysts want to extract out of plots? I am especially concerned about finding the roughest/smoothest and low/high median. Even humans are relatively bad at these tasks. Why do we expect models to do well on them?
— Why only binary questions? It is probably more difficult for analysts to ask a binary question than to ask non-binary ones such as “What is the highest in this plot?” — Why these 5 types of plots? Can the authors justify that these 5 types of plots are the most frequent ones dealt by analysts?
— Are the model accuracies in Table 3 on the same subset as humans or on the complete test set? Can the authors please report both separately?
Overall: 
The proposed dataset seems reasonable but neither the dataset seems properly motivated (something where analysts actually struggle and models can help) nor it is clear if it will actually be useful for the research community (models performing well on this dataset will need to focus on specific abilities which have not been studied in the research community).