Current evaluation metrics for image description may be too coarse. We therefore propose a series of binary forced-choice tasks that each focus on a different aspect of the captions. We evaluate a number of different off-the-shelf image description systems. Our results indicate strengths and shortcomings of both generation and ranking based approaches.
['Micah Hodosh', 'Julia Hockenmaier']
Focused Evaluation for Image Description with Binary Forced-Choice Tasks