  When developing deep learning models, we usually decide what task we want to
solve then search for a model that generalizes well on the task. An intriguing
question would be: what if, instead of fixing the task and searching in the
model space, we fix the model and search in the task space? Can we find tasks
that the model generalizes on? How do they look, or do they indicate anything?
These are the questions we address in this paper.
  We propose a task discovery framework that automatically finds examples of
such tasks via optimizing a generalization-based quantity called agreement
score. We demonstrate that one set of images can give rise to many tasks on
which neural networks generalize well. These tasks are a reflection of the
inductive biases of the learning framework and the statistical patterns present
in the data, thus they can make a useful tool for analysing the neural networks
and their biases. As an example, we show that the discovered tasks can be used
to automatically create adversarial train-test splits which make a model fail
at test time, without changing the pixels or labels, but by only selecting how
the datapoints should be split between the train and test sets. We end with a
discussion on human-interpretability of the discovered tasks.

Machine Learning