  State-of-the-art natural language processing models have been shown to
achieve remarkable performance in 'closed-world' settings where all the labels
in the evaluation set are known at training time. However, in real-world
settings, 'novel' instances that do not belong to any known class are often
observed. This renders the ability to deal with novelties crucial. To initiate
a systematic research in this important area of 'dealing with novelties', we
introduce 'NoveltyTask', a multi-stage task to evaluate a system's performance
on pipelined novelty 'detection' and 'accommodation' tasks. We provide
mathematical formulation of NoveltyTask and instantiate it with the authorship
attribution task that pertains to identifying the correct author of a given
text. We use Amazon reviews corpus and compile a large dataset (consisting of
250k instances across 200 authors/labels) for NoveltyTask. We conduct
comprehensive experiments and explore several baseline methods for the task.
Our results show that the methods achieve considerably low performance making
the task challenging and leaving sufficient room for improvement. Finally, we
believe our work will encourage research in this underexplored area of dealing
with novelties, an important step en route to developing robust systems.

Computation and Language