Transformer is a popularly used neural network architecture, especially for language understanding. We introduce an extended and unified architecture that can be used for tasks involving a variety of modalities like image, text, videos, etc. We propose a spatio-temporal cache mechanism that enables learning spatial dimension of the input in addition to the hidden states corresponding to the temporal input sequence. The proposed architecture further enables a single model to support tasks with multiple input modalities as well as asynchronous multi-task learning, thus we refer to it as OmniNet. For example, a single instance of OmniNet can concurrently learn to perform the tasks of part-of-speech tagging, image captioning, visual question answering and video activity recognition. We demonstrate that training these four tasks together results in about three times compressed model while retaining the performance in comparison to training them individually. We also show that using this neural network pre-trained on some modalities assists in learning unseen tasks such as video captioning and video question answering. This illustrates the generalization capacity of the self-attention mechanism on the spatio-temporal cache present in OmniNet.
This paper describes the "OmniNet" architecture, which is essentially a transformer to convert any 2-dimensional (time and spatial) input into a sequence of output tokens. The idea is that a single model (the "Central Neural Processor") would learn to perform multiple tasks on multiple inputs at the same time. This seems like a reasonable approach, and would allow a single model to be able to process text, image, as well as video or potentially speech input just the same. Dedicated modality-specific "input peripherals" would "normalize" the data appropriately, and a "start token" seems to provide the model with information on what type of input data is to follow.
I am a bit confused by the problem formulation and implementation:
- Multi-task learning works best if there are clear dependencies and shared characteristics between the task - I am not sure if the tasks used here (POS tagging, image captioning, VQA, video activity recognition) share sufficient characteristics 
- Successful multi-task learning often leads to shared representations developing across the different tasks. Is there any indication of this happening in the proposed work? Would it help to make the modality specific encoders (BPE and ResNet) trainable, so that the transformer model has an easier time learning multi- and cross-modal representations? Or is this entirely the CNP's role?
- It is probably possible to use HogWild as a way to train a multi-modal and multi-task system in the described manner, but will this lead to shared representations developing? Is there any way to include regularization or other tricks to encourage the formation of shared cross-modal representations?
Here are a few ideas:
- Would it be possible to show how well the model works when trained and optimized (including hyper parameter optimization) on a single task only? it would then be possible to show that the multi-task model can outperform single-modality single-task models 
- What about tasks like speech input? There is other, published work on multi-task training for CTC models w.g. by Ramon Sanabria ("Hierarchical Multi Task Learning With CTC", SLT 2018), in addition to the reference provided
- Would it be possible to show how the model learns to develop cross-modal representations during training? this would be a convincing visualization and rationalization of the proposed approach