  Neural Conversational QA tasks like ShARC require systems to answer questions
based on the contents of a given passage. On studying recent state-of-the-art
models on the ShARCQA task, we found indications that the models learn spurious
clues/patterns in the dataset. Furthermore, we show that a heuristic-based
program designed to exploit these patterns can have performance comparable to
that of the neural models. In this paper we share our findings about four types
of patterns found in the ShARC corpus and describe how neural models exploit
them. Motivated by the aforementioned findings, we create and share a modified
dataset that has fewer spurious patterns, consequently allowing models to learn
better.

Computation and Language;Artificial Intelligence