  Majority of research in learning based methods has been towards designing and
training networks for specific tasks. However, many of the learning based
tasks, across modalities, share commonalities and could be potentially tackled
in a joint framework. We present an approach in such direction, to learn
multiple tasks, in multiple modalities, with a unified architecture. The
proposed network is composed of task specific encoders, a common trunk in the
middle, followed by task specific prediction heads. We first pre-train it by
self-supervised masked training, followed by sequential training for the
different tasks. We train the network on all major modalities, e.g.\ visual,
audio, text and 3D, and report results on $22$ diverse and challenging public
benchmarks. We demonstrate empirically that, using a joint network to train
across modalities leads to meaningful information sharing and this allows us to
achieve state-of-the-art results on most of the benchmarks. We also show
generalization of the trained network on cross-modal tasks as well as unseen
datasets and tasks.

Computer Vision and Pattern Recognition;Machine Learning