summaries: This paper proposes a unified architecture in the context of multi-task learning where they demonstrate that training four tasks (with a variety of modalities like image, text, and videos) together results in about three times compressed model, while maintaining the performance similar to their respective individually trained models. The major components of this archtiecutre are (1) peripheral networks: used to encode the domain specific input into feature representations. (2) central neural processor: a fully attention based encoder-decoder model similar to the Transformer networks which encodes the spatio-temporal information. Further, this paper suggests that their unified architecture enables to perform decent on unseen tasks during its training. In this paper they test such scenarios on video captioning and video question answering.  This paper describes the "OmniNet" architecture, which is essentially a transformer to convert any 2-dimensional (time and spatial) input into a sequence of output tokens. The idea is that a single model (the "Central Neural Processor") would learn to perform multiple tasks on multiple inputs at the same time. This seems like a reasonable approach, and would allow a single model to be able to process text, image, as well as video or potentially speech input just the same. Dedicated modality-specific "input peripherals" would "normalize" the data appropriately, and a "start token" seems to provide the model with information on what type of input data is to follow. The authors propose an extended and unifying learning architecture – OmniNet- based on transformer, which tackles tasks with various modalities such as images, text and videos. This is attained with a spatio-temporal cache mechanism, that can both capture and store temporal information and spatial information. In addition, this proposed framework supports asynchronous multi-task learning with pre-trained neural networks on different modalities. OmiNet’s generalization capability is illustrated and demonstrated with experiments. The proposed model has multiple peripheral networks each majoring on one unique modality of data. These peripheral networks project input data into the same shared format/space that can be uniformly processed in a central neural processor working like a CPU. The central neural processor uses self-attention and RNN for temporal encoding and spatial encoding. The output of the encoding components are stored in temporal and spatial caches respectively. The two caches are then used as input to the spatial temporal decoder for different tasks. 
abstractText: Transformer is a popularly used neural network architecture, especially for language understanding. We introduce an extended and unified architecture that can be used for tasks involving a variety of modalities like image, text, videos, etc. We propose a spatio-temporal cache mechanism that enables learning spatial dimension of the input in addition to the hidden states corresponding to the temporal input sequence. The proposed architecture further enables a single model to support tasks with multiple input modalities as well as asynchronous multi-task learning, thus we refer to it as OmniNet. For example, a single instance of OmniNet can concurrently learn to perform the tasks of part-of-speech tagging, image captioning, visual question answering and video activity recognition. We demonstrate that training these four tasks together results in about three times compressed model while retaining the performance in comparison to training them individually. We also show that using this neural network pre-trained on some modalities assists in learning unseen tasks such as video captioning and video question answering. This illustrates the generalization capacity of the self-attention mechanism on the spatio-temporal cache present in OmniNet.
references.authors: P. Anderson X. He C. Buehler D. Teney M. Johnson S. Gould L. Zhang
references.title: Bottom-up and top-down attention for image captioning and visual question answering
references.venue: IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Dzmitry Bahdanau Kyunghyun Cho Yoshua Bengio
references.title: Neural machine translation by jointly learning to align and translate
references.venue: In 3rd International Conference on Learning Representations,
references.authors: Eric Battenberg Jitong Chen Rewon Child Adam Coates Yashesh Gaur Yi Li Hairong Liu Sanjeev Satheesh David Seetapun Anuroop Sriram Zhenyao Zhu
references.title: Exploring neural transducers for end-to-end speech recognition
references.venue: CoRR, abs/1707.07413,
references.authors: Yaran Chen Dongbin Zhao Le Lv Qichao Zhang
references.title: Multi-task learning for dangerous object detection in autonomous driving
references.venue: Information Sciences,
references.authors: Kyunghyun Cho Bart van Merrienboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio
references.title: Learning phrase representations using RNN encoder– decoder for statistical machine translation
references.venue: In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP),
references.authors: Ronan Collobert Jason Weston
references.title: A unified architecture for natural language processing: Deep neural networks with multitask learning
references.venue: In Proceedings of the 25th International Conference on Machine Learning,
references.authors: Daxiang Dong Hua Wu Wei He Dianhai Yu Haifeng Wang
references.title: Multi-task learning for multiple language translation
references.venue: In ACL,
references.authors: Yash Goyal Tejas Khot Douglas Summers-Stay Dhruv Batra Devi Parikh
references.title: Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering
references.venue: In Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: K. He X. Zhang S. Ren J. Sun
references.title: Deep residual learning for image recognition
references.venue: IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Lanqing Hu Meina Kan Shiguang Shan Xilin Chen
references.title: Duplex generative adversarial network for unsupervised domain adaptation
references.venue: In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Melvin Johnson Mike Schuster Quoc V. Le Maxim Krikun Yonghui Wu Zhifeng Chen Nikhil Thorat Fernanda Viégas Martin Wattenberg Greg Corrado Macduff Hughes Jeffrey Dean
references.title: Google’s multilingual neural machine translation system: Enabling zero-shot translation
references.venue: Transactions of the Association for Computational Linguistics,
references.authors: Lukasz Kaiser Aidan N. Gomez Noam Shazeer Ashish Vaswani Niki Parmar Llion Jones Jakob Uszkoreit
references.title: One model to learn them all
references.venue: CoRR, abs/1706.05137,
references.authors: Jin-Hwa Kim Sang-Woo Lee Donghyun Kwak Min-Oh Heo Jeonghee Kim Jung-Woo Ha Byoung-Tak Zhang
references.title: Multimodal residual learning for visual qa
references.venue: Advances in Neural Information Processing Systems
references.authors: Jin-Hwa Kim Jaehyun Jun Byoung-Tak Zhang
references.title: Bilinear attention networks
references.venue: Advances in Neural Information Processing Systems
references.authors: Diederik P. Kingma Jimmy Ba
references.title: Adam: A method for stochastic optimization
references.venue: In 3rd International Conference on Learning Representations,
references.authors: Kalpesh Krishna Shubham Toshniwal Karen Livescu
references.title: Hierarchical multitask learning for ctcbased speech recognition
references.venue: CoRR, abs/1807.06234,
references.authors: H. Kuehne H. Jhuang E. Garrote T. Poggio T. Serre
references.title: HMDB: a large video database for human motion recognition
references.venue: In Proceedings of the International Conference on Computer Vision (ICCV),
references.authors: Jie Lei Licheng Yu Mohit Bansal Tamara L. Berg
references.title: TVQA: localized, compositional video question answering
references.venue: CoRR, abs/1809.01696,
references.authors: Tsung-Yi Lin Michael Maire Serge Belongie James Hays Pietro Perona Deva Ramanan Piotr Dollár C. Lawrence Zitnick
references.title: Microsoft coco: Common objects in context
references.venue: Computer Vision – ECCV
references.authors: Thang Luong Quoc V. Le Ilya Sutskever Oriol Vinyals Lukasz Kaiser
references.title: Multi-task sequence to sequence learning
references.venue: In International Conference on Learning Representations,
references.authors: Mitchell Marcus Grace Kim Mary Ann Marcinkiewicz Robert MacIntyre Ann Bies Mark Ferguson Karen Katz Britta Schasberger
references.title: The penn treebank: Annotating predicate argument structure
references.venue: In Proceedings of the Workshop on Human Language Technology,
references.authors: Volodymyr Mnih Adrià Puigdomènech Badia Mehdi Mirza Alex Graves Tim Harley Timothy P. Lillicrap David Silver Koray Kavukcuoglu
references.title: Asynchronous methods for deep reinforcement learning
references.venue: In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48,
references.authors: Jiquan Ngiam Aditya Khosla Mingyu Kim Juhan Nam Honglak Lee Andrew Y. Ng
references.title: Multimodal deep learning
references.venue: In Proceedings of the 28th International Conference on International Conference on Machine Learning,
references.authors: Adam Paszke Sam Gross Soumith Chintala Gregory Chanan Edward Yang Zachary DeVito Zeming Lin Alban Desmaison Luca Antiga Adam Lerer
references.title: Automatic differentiation in PyTorch
references.venue: In NIPS Autodiff Workshop,
references.authors: Benjamin Recht Christopher Re Stephen Wright Feng Niu
references.title: Hogwild: A lock-free approach to parallelizing stochastic gradient descent
references.venue: Advances in Neural Information Processing Systems
references.authors: M.L. Seltzer J. Droppo
references.title: Multi-task learning in deep neural networks for improved phoneme recognition
references.venue: IEEE International Conference on Acoustics, Speech and Signal Processing,
references.authors: Rico Sennrich Barry Haddow Alexandra Birch
references.title: Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1715–1725, Berlin, Germany, August 2016
references.venue: Association for Computational Linguistics. doi: 10.18653/v1/P16-1162. URL https://www.aclweb. org/anthology/P16-1162
references.authors: Noam Shazeer Mitchell Stern
references.title: Adafactor: Adaptive learning rates with sublinear memory
references.venue: cost. CoRR,
references.authors: Rui Shu Hung H. Bui Hirokazu Narui Stefano Ermon
references.title: A dirt-t approach to unsupervised domain adaptation
references.authors: Karen Simonyan Andrew Zisserman
references.title: Two-stream convolutional networks for action recognition in videos
references.venue: Advances in Neural Information Processing Systems
references.authors: Drahomı́ra “johanka” Spoustová Jan Hajič Jan Raab Miroslav Spousta
references.title: Semi-supervised training for the averaged perceptron POS tagger
references.venue: In Proceedings of the 12th Conference of the European Chapter of the ACL (EACL
references.authors: Ilya Sutskever Oriol Vinyals Quoc V Le
references.title: Sequence to sequence learning with neural networks
references.venue: Advances in Neural Information Processing Systems
references.authors: Christian Szegedy Sergey Ioffe Vincent Vanhoucke
references.title: Inception-v4, inception-resnet and the impact of residual connections on learning
references.venue: CoRR, abs/1602.07261,
references.authors: Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Ł ukasz Kaiser Illia Polosukhin
references.title: Attention is all you need
references.venue: Advances in Neural Information Processing Systems
references.authors: Zhanpeng Zhang Ping Luo Chen Change Loy Xiaoou Tang
references.title: Facial landmark detection by deep multi-task learning
references.venue: In In ECCV
references.authors: Wei Zhao Benyou Wang Jianbo Ye Min Yang Zhou Zhao Ruotian Luo Yu Qiao
references.title: A multi-task learning approach for image captioning
references.venue: In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence,
sections.heading: 1 INTRODUCTION
sections.text: Transformer (Vaswani et al., 2017) is currently one of the best performing models for any sequence transduction tasks, especially those involving natural language. It is originally designed for a single task at a time. In fact, most of the generic deep learning architectures (Wu et al., 2016; Szegedy et al., 2016; Battenberg et al., 2017) that have been designed and developed are able to learn, albeit very well, a single task and handle one task specific input domain like image, text or audio. Furthermore with these models, we often rely on the generalization capability of the trained network to guarantee performance on unseen examples. Transfer learning (Shu et al., 2018; Hu et al., 2018) is another popular paradigm used to adapt the model to learn a related task with similar input domain. The success of neural networks across these challenges is known to be due to their ability in learning effective representations of the data. For example, the self-attention mechanism in Transformers can capture the global temporal dependence in sequential data very well. Naturally, the question arises whether we can extend these architectures, like the Transformer, to be able to learn shared representations from multiple input domains and to be able attend on these representations to perform a multitude of tasks concurrently.
  The research into multi-task models that learn to solve varied tasks across a multitude of input domains is not new. Work done in Ngiam et al. (2011) demonstrates an architecture capable of learning a shared representation across audio and video modalities. Similarly in Collobert & Weston (2008) a convolutional architecture has been designed to support a variety of NLP tasks. However, most of these architectures are designed to learn specific set of tasks with known input domains. To the best of our knowledge, there does not exist a single unified architecture that works out of the box for any combination of multi-modal inputs.
  To address this gap, we extend Transformer towards a unified architecture, namely OmniNet, which enables a single model to support tasks with multiple input modalities and asynchronous multi-task learning. We consider that most real-life data like image, text, speech, video, etc. is a direct conjunction of spatial and temporal components. Therefore, we employ a spatio-temporal cache mechanism to learn a shared representation of the input data across the spatial (space) and temporal (time) di-
  mension. Using a generalized encode() function, OmniNet can process and store spatio-temporal representation for each of the input domains and then decode() predictions across a multitude of tasks. In our experiments, we train a single instance of the OmniNet to solve a number of tasks spanning multiple multi-domains such as part-of-speech tagging, image captioning, visual question answering and video activity recognition. To make our work reproducible, open to scrutiny and further development, we will open source a demonstration of our system implemented using Pytorch (Paszke et al., 2017).
sections.heading: 2 RELATED WORK
sections.text: Multi-task learning has been extensively studied in the literature, with applications to a wide set of problems ranging from natural language processing (NLP) (Collobert & Weston, 2008; Johnson et al., 2017; Dong et al., 2015) to speech recognition (Seltzer & Droppo, 2013; Krishna et al., 2018) to vision (Zhang et al., 2014; Chen et al., 2018; Anderson et al., 2018). It has also found its use in a combination of diverse tasks like image captioning and text translation and parsing (Luong et al., 2016; Zhao et al., 2018). However, most of these architectures assume the set of tasks to be known in advance. Similarly, multi-modal learning has been essential for solving a broad range of interesting problems such as Visual Question Answering (Kim et al., 2016; 2018) and Video Question Answering (Lei et al., 2018). Again, the state-of-the-art models are highly specific to the objective in hand and not easily adaptable to different tasks or domains. (Kaiser et al., 2017) proposed MultiModel architecture for learning multiple tasks but lacks support for multi-modal tasks with more than one input domains such as visual question answering.
sections.heading: 3 PROPOSED MODEL
sections.text: We propose a unified architecture, namely OmniNet, to enable learning multi-modal tasks with multiple input domains and support generic multi-tasking for any set of tasks. The OmniNet architecture consists of multiple sub-networks, called peripheral networks, connected to a common central neural network called the Central Neural Processor (CNP) (Figure 1). Each peripheral network is used to encode the domain specific input into feature representations. In this work, we describe image, text and video peripherals (Section 3.1). One can add more, say speech peripheral, depending on the task. The output representation of a peripheral network is always a spatio-temporal tensor x ∈ Rt×s×dmodel , where t & s are the temporal and spatial dimensions of the input respectively, and dmodel is the model dimension input to the CNP.
  The spatio-temporal representations generated by the peripheral networks corresponding to each input domain are then processed by the CNP. The CNP uses fully attention based encoder-decoder (Bahdanau et al., 2015; Sutskever et al., 2014; Cho et al., 2014) model for sequence transduction similar to the Transformer architecture (Vaswani et al., 2017), which is the state-of-the-art for multiple language modeling tasks (Section 3.2). During the encoding stage, the CNP implements a
  generic encode(x, D) function to first process and store the spatio-temporal representations of the input, where x ∈ Rt×s×dmodel is the spatio-temporal tensor produced by the peripheral networks and D ∈ Z : 0 ≤ D < Dlen is the domain id and Dlen is the max number of domains supported by the CNP. The encode() function is called multiple times, once for each multi-modal input from respective peripheral. During the decoding stage, a decode(yshifted, τ ) function is used to decode predictions as softmax probabilities, where yshifted ∈ ZN−1 are the target outputs shifted one timestep to the right, N is the length of the output sequence; τ ∈ Z : 0 ≤ τ < τlen is task id and τlen is the total number of supported tasks. The decoding step is similar to Vaswani et al. (2017), modified to incorporate a two-step attention mechanism over spatial and temporal cache.
sections.heading: 3.1 PERIPHERAL NETWORKS
sections.text: First, we elaborate on how we support multiple input domains using peripheral networks. A peripheral network can use a pre-trained model from existing literature to ultimately encode a given domain input to a standardized feature representation x ∈ Rt×s×dmodel , where t & s are the temporal and spatial dimensions of the input respectively, and dmodel is the model dimension input to the Central Neural Processor. Here we detail text and vision peripherals and one can add more peripherals or alter the peripheral design depending on the task.
  Vision peripheral: This peripheral uses a convolutional neural network to encode image and video inputs in the tasks. For an image of dimension h × w × nc, this peripheral down-samples in to h′ ×w′ × n′c, where h,w, nc are the height, width and number of input channels respectively. For a video, each frame is input to the peripheral to produce F ×h′×w′×n′c, where F is the total number of frames in the video. The encoding vectors are then projected to dimension dmodel using a fully connected layer. The output is then reshaped into a spatio-temporal tensor of x ∈ Rt×h′w′×dmodel , where t = 1 for an image and t = F for a video. In our experiments, we use the pre-trained ResNet-152 model, a variant of ResNet (He et al., 2016) consisting of 152 convolutional layers. We remove the final fully connected and avg-pooling layers to generate spatial feature representations for a given image/video.
  Language peripheral: The Language peripheral uses byte-pair encoding (Sennrich et al., 2016) to generate subwords for a given input sentence. The subwords are passed to an embedding layer to generate subword embeddings of dimension demb and projected to dimension dmodel using a fully connected layer. The output is then reshaped into a spatio-temporal tensor x ∈ Rt×1×dmodel , where t equal to number of subwords in the input sentence. As we do not have any spatial dimension in textual data, the spatial dimension of x from a Language peripheral is always 1. In our experiments, We used pre-trained subword embeddings with demb = 300 and vocab size = 25000 from Heinzerling & Strube (2018), which includes pre-trained subword embeddings of over 275 languages, to initialize the weights of the embedding matrix.
sections.heading: 3.2 CENTRAL NEURAL PROCESSOR (CNP)
sections.text: To process the spatio-temporal information in the input data, the CNP implements a spatial cache Cs, temporal cache Ct and a link array L. The spatial and temporal cache and the link array are a list of elements, initialized as empty before the encoding process. During the encoding stage, an encode() routine takes as input, the tensor x generated from the peripheral and corresponding domain/peripheral id D. This function processes the spatial and temporal information in the input x and stores them into the spatial cache Cs and the temporal cache Ct , respectively and stores their dimensions t& s in the link array. For a given task, this encode() routine is calledK times, whereK is the number of inputs in the task. Note that these inputs can belong to same or different domains.
  Encode (x, D): For a given input x ∈ Rt×s×dmodel and domain identifier D, the encode() routine is described in Algorithm 1. Since inputs can come from multiple peripherals, the algorithm first concatenates the input with the domain embedding to ensure a domain-aware encoding of the input (Steps 2 to 3). Steps 4 to 7 process the spatial information in x by unrolling the time dimension and adding these unrolled vectors into the spatial cache. Steps 8 to 10 process the temporal information in x by averaging the spatial dimension of x and then passing the averaged tensor to a self-attention based TemporalEncoder. This TemporalEncoder is similar to the encoder used in Vaswani et al. (2017) as shown in Figure 2 is used to calculate temporal embeddings of the input sequence. The output from the TemporalEncoder is appended to the temporal cache.
  Algorithm 1 encode(): Encodes spatial and temporal representations into spatial and temporal cache Require: x ∈ Rt×s×dmodel , D, Cs, L, Ct
  1: L← L ∪ (t→ s) 2: Demb ← EmbedLayer(D) 3: x← FC(Concat(x,Demb), dmodel) 4: if s > 1 then 5: S ← Reshape(x, (ts, dmodel)) {where, output S = [S1, . . . , Sts] s.t. Si ∈ Rdmodel is a spatial feature vector.} 6: Cs ← Cs ∪ [S1, . . . , Sts] {Append spatial representations to spatial cache} 7: end if 8: T ← ( ∑s i=1 x[:, i, :])/s 9: T ← TemporalEncoder(T ) {where, output T = [T1, . . . , Tt] s.t. Tj ∈ Rdmodel is the encoding of temporal dimension in x.}
  10: Ct ← Ct ∪ [T1, . . . , Tt] {Append temporal representations to temporal cache}
  The above encoding routine keeps appending spatio-temporal information to Ct & Cs for each input xk ∈ Rtk×sk×dmodel . Note the superscript k to denote correspondence to k-th input of the task, where k ∈ 1, . . . ,K. After K calls, we have the temporal cache Ct = [T1, . . . , TR], where R =∑K r=1 tr; the spatial cache Cs = [S1, . . . , SP ], where P = { ∑ p tp ∗ sp : p ∈ 1, . . . ,K ∧ sp > 1} and the link array L = [(t1 → s1), . . . , (tK → sK)]. Note that Cs can also be empty in case the encode() is only called with inputs with sk = 1∀k. Next, we use the decode() routine to generate predictions as softmax probabilities.
  Decode (yshifted, τ ): The architecture of the decode() function is shown in Figure 2. The decode() takes as argument the output labels yshifted shifted one time step to the right, a task id τ and gen-
  erates predictions by attending from the spatial and temporal cache. The decode() function is structured similar to the decoder used in the Transformer architecture Vaswani et al. (2017) and jointly attends on the vectors stored in the temporal and spatial cache. Similar to Vaswani et al. (2017), the decoding first starts by attending over the output embeddings using masked multi-head scaled dot product attention. The attention layer for the temporal cache uses scaled dot-product attention with multiple heads as specified in Vaswani et al. (2017). Attention layer for the spatial cache, uses gated multi-head attention to attend over the elements of the spatial cache. For inputs with both time and space dimension (e.g. video), we want the spatial attention layer to attend more on frames which have relatively high attention scores in the temporal cache attention layer. Therefore, the attention score output from the temporal cache multi-head attention layerA ∈ Rnh×N×R, is used to calculate the tensor G ∈ Rnh×N×P used for gating the attention score output in the spatial attention layer, where nh is the number of heads in multi-head attention as described in Vaswani et al. (2017). The tensor G is calculated using A & L as detailed in Algorithm 2. Given1 Q: the matrix of queries, K: keys of dimension dk and V : values of dimension dv , the scaled dot-product attention for the spatial layer is modified as:
  Attention(Q,K, V,G) = ( Softmax ( QKT√ dv ) G ) V (1)
  In order to use the same CNP for multiple tasks with varying output vocabularies, we use multiple output embedding layers OutputEmbedLayer1, . . . , OutputEmbedLayerτlen , to generate the output embeddings for each task. At the final layer, we use multiple (FC + Softmax)1, . . . , (FC + Softmax)τlen classification layers for each task. We also calculate a task embedding vector using τ and always start decoding using the task embedding vector.
  Algorithm 2 Calculate G using output scores from temporal attention and link array Require: L, A
  1: idx← 0 2: for each t, s in L do 3: G← [] 4: if s > 0 then 5: A′ ← A[:, :, idx : idx+ t] 6: A′ ← Expand(A′, (nh, N, t, s)) {where Expand(tensor, dimension) expands tensor according to a given dimension} 7: A′ ← Reshape(A′, (nh, N, ts)) 8: G← G ∪A′ {Append the respective temporal attention scores to G} 9: end if
  10: idx← idx+ t 11: end for 12: G← Stack(G) {Stack the list of tensors to construct tensor G of dimension (nh, N, P )}
sections.heading: 3.3 MULTI-TASK LEARNING
sections.text: In order to train a single a model simultaneously on mutiple tasks we used the HogWild training approach as described in Recht et al. (2011). Similar to the approach described in Mnih et al. (2016), the main process holds a global copy of the model. We create separate worker processes for each task, where each process maintains a local copy of a model. At each training iteration, each process starts by synchronizing its local model with the global copy. This is done through forward and backward propagation on its local copy and then copying the locally computed gradients to the global model asynchronously. Each process then calls the global model optimizer asynchronously to update the weights of the global model. Instead of storing the model in CPU as in Mnih et al. (2016) we always store the local copies across multiple GPUs.
sections.heading: 4 TASKS AND SETUP
sections.text: To evaluate the effectiveness of our proposed framework for tasks spanning diverse modalities, we choose a set covering all possible spatio-temporal data archetypes: Image Captioning, Part-of-
  1For brevity, we reuse the notations of Vaswani et al. (2017) in this description
  Speech (POS) tagging, Visual Question Answering (VQA) and Video-activity Recognition. Each of these tasks explores a unique potential spatio-temporal configuration of the input containing different values of t and s, where t and s are the temporal and spatial dimensions of the input respectively. This enables us to perform a comprehensive study of the multi-modal and multi-input capabilities of the system. We further elaborate on the properties of these tasks below.
  For training, we always use cross-entropy loss with Adam optimizer (Kingma & Ba, 2015) and schedule the learning rate using Noam scheduler (Shazeer & Stern, 2018) similar to (Vaswani et al., 2017) 2. In the vision peripheral, we freeze the layers of pretrained ResNet model. Remaining parts of the architecture (peripherals and CNP) are kept trainable. In this section, we provide details on the datasets used and the model setup for each of these tasks.
  Part-of-speech (POS)Tagging: To illustrate the task with only temporal modality (t > 1 & s = 1, where t and s are the temporal and spatial dimensions of the input respectively), we consider POS tagging problem. Given an input sequence of words, the model should produce a sequence of POS tags corresponding to each word. We use Penn Tree-bank3 (Marcus et al., 1994) which contains gold annotations on English WSJ articles by experts. During the encoding stage, each input sentence is processed by the language peripheral to generate a spatio-temporal tensor x ∈ Rt×1×dmodel , where t is the sequence length of subwords in the input. The CNP encode() function is then used to encode x into the temporal cache. Note that spatial cache is empty for text inputs as s = 1. Therefore, the decoding stage is same as that of Transformers to predict the sequence of POS tags.
  Image Captioning: This task represents the ones with inputs containing only spatial modality (t = 1 & s > 1). The captioning model is required to predict text caption for a given image. We use the MSCOCO 2014 dataset (Lin et al., 2014) for training and present results on the COCO validation set. During the encoding stage, the input image is resized to 224× 224 and processed by the vision peripheral containing pre-trained ResNet-152 to produce image embeddings x ∈ R1×49×dmodel . x is then input to the encode() function which populates corresponding spatial and temporal cache. The decoding stage uses decode() function with output vocabulary size 25000 to generate the captions.
  Visual Question Answering: For the task with inputs from multiple domain, such that each contains either spatial or temporal modality (either t > 1 & s = 1, or t = 1 & s > 1 for each input), we choose the task of visual question answering. Given a question over an image as inputs, the model is supposed to predict the correct answer label. We use the recently introduced VQA v2.0 dataset (Goyal et al., 2017) for this purpose and perform evaluation on the VQA test-dev set. All the images are resized to dimension 224× 224 before training. The encoding stage of this task utilizes two peripherals: the vision peripheral is used to generate a tensor x1 ∈ R1×49×dmodel for the input image. The language peripheral is used to encode the questions into x2 ∈ Rt×1×dmodel , where t is equal to the length of the subwords in the question. The encode() function is the called two times, first with x1 and second with x2 as input. Finally, the decode() with output vocabulary size 3500 is to generate the answers as softmax probabilities in a single decoding step.
  Video Activity Recognition: For tasks which contain both spatial and temporal modality in a single input (t > 1 & s > 1), we consider the action recognition task on videos. For this purpose, we use the HMDB dataset (Kuehne et al., 2011). The dataset consists of over 5000 short length clips of real life actions with 51 classes. We present our results on train-test split 1. We use 16 frames per video and resize each of them to 224× 224. During the encoding stage, each frame of the video is passed through the vision peripheral to cumulatively generate a video encoding x ∈ R16×49×dmodel which is then used as input to the encode() function. Finally, the decode() with output vocabulary size 51 is to predict the action as softmax probabilities in a single decoding step.
sections.heading: 5 RESULTS AND DISCUSSION
sections.text: We present the evaluation on (a) Tasks of various modalities illustrated in Section 4 (b) Multi-tasking setup for these tasks (Table 1) (c) Reuse of the multi-task model for an unseen task (Figure 3). In addition, we also provide some ablation studies on the architecture (Table 2)
  2The hyperparameter values used for nh, dmodel, N Layers, dk, dv are same as that specified in Transformer base model (Vaswani et al., 2017).
  3https://catalog.ldc.upenn.edu/LDC99T42; We use splits 0-18 as training, 19-21 as development and 22-24 as test sets
  Performance of proposed architecture on individual tasks: We choose a set of four tasks with diverse input modalities and combinations as described in previous Section 4. We train the OmniNet model independently across each of the above tasks. Each of the tasks demonstrates unique capabilities of this generic architecture. More specifically, in Table 1 we compare our results with the following state-of-the-art4:- POS tagging: Spoustová et al. (2009); image captioning & VQA: Anderson et al. (2018) and HMDB: Simonyan & Zisserman (2014). It is important to note that we do not perform any hyper-parameter optimization. We believe that, with more computational power, fine tuning the hyperparameters towards these tasks should result in comparable or even improved performance to the state-of-the-art. These results can indeed be used as a baseline for any future work which aims at using a single architecture across various possible spatio-temporal archetypes. It is interesting to note that the model is extensible to a new domain without any modification to the CNP as long as one can add a specific peripheral to convert domain inputs into spatio-temporal tensors. This aspect of the architecture makes it applicable to several popular multi-modal tasks.
  Effect of training a diverse set of tasks together: We trained two multi-task models: (1) MULT-3 (POS+VQA+Captioning) and (2) MULT-4 (POS+VQA+Captioning+HMDB), using hogwild approach. While the MULT-3 model attains similar and sometimes better performance, the final MULT-4 model attains slightly reduced performance, when compared to the independent task scores. We believe this is due to the skewness in the size of the HMDB dataset containing only 5000 training samples. However as a tradeoff, adding the HMDB task shows interesting zero-shot results demonstrated below. Using a multi-task model also results in three times reduction in the total number of parameters. That is, when a separate model is used for each task, we have a total of over 450× 106 parameters. Whereas during multi-tasking since a single model is shared, we have a total of over 149 × 106 parameters, while achieving similar performance. Interestingly, the model is able to attend on spatio-temporal components of the inputs from different tasks and concurrently generate predictions across them, thus demonstrating the generalization capability of our architecture.
  Towards zero-shot learning: reuse of pre-trained network for unseen task: Sharing representations across multiple tasks provides the benefit to transfer of useful knowledge across multiple domains. Since, image and video are processed by the same vision peripheral, we conducted an experiment to see whether our model pre-trained on all the four tasks (MULT-4) can perform video
  4Since most of these tasks are popular challenges, we compare with state-of-the-art which are generically applicable for the respective task instead of the challenge dataset.
  captioning and video question-answering without any explicit training on these tasks i.e. zero-shot learning. The results of the evaluation on randomly picked instances from the HMDB test split 1 are shown in Figure 3. Interestingly, the model performs quite well on related actions that were present in the COCO and VQA training set; such as captions related to horse riding and baseball; or questions related to concepts present in VQA. Without training on any video captioning & video QA instance, the model could use the trained information from image captioning, image QA (VQA) and video action recognition (HMDB) apply them on videos to generate meaningful predictions, hence demonstrating the capability of the model to transfer knowledge across related multi-modal tasks. However, on concepts that are not present in the trained datasets, the model either describes the environment in the video or replaces with alternate known concepts. This case study, although not comprehensive, shows the capability of the model to learn shared representations and ability to transfer knowledge across domains. We believe that adding more tasks and domains will lead to more interesting zero-shot learning results in future across a wide range of problems.
  Impact of individual architectural components: In order to support different input domains, our architecture introduces spatial cache and link array components to the original Transformer architecture (which only consists of mechanisms to handle temporal data). We conducted an ablation study on each of these components to verify their importance across various tasks as shown in Table 2. The ablation was conducted on the independent (IND) as well the multi-tasking model (MULT-4). The second row ablates the link array from our architecture i.e. removing the multiplication of G in Equation 1. The link array was designed to assist in tasks with inputs such as video, containing both spatial as well as temporal modality in a single input. The total number of spatial components becomes very large as number of frames in the video increases, thereby making it difficult to attend on various spatial regions throughout the video. Using link array the spatial attention layer can attend more on specific important frames in the video. Therefore, removal of link array leads to a huge reduction in performance in HMDB compared to other tasks as they do not have both spatio-temporal modalities for any single input. Removal of spatial cache, on the other hand, has significant effect on performance across all tasks containing spatial modality. Since, image captioning contains primarily spatial modality and hence the BLEU drops significantly after ablation. As other tasks utilize the temporal cache for prediction, in the multi-task setting the captioning task learns to utilize the spatial average of the image stored in the temporal cache for prediction and hence retains some performance after ablation of the spatial cache. On the other hand when trained independently on captioning, the network learns to utilize the information in the spatial cache only, and hence the score drops to zero after ablation. VQA leverages both, spatial information from image and temporal information from question, retains some performance from the use of temporal cache. Note that, POS tagging task is not affected by ablation of any of the components since it only has temporal modality in the input.
sections.heading: 6 CONCLUSIONS AND FUTURE WORK
sections.text: We present a unified neural network architecture OmniNet capable of learning tasks with multiple inputs of varying modalities. The architecture can be further adopted for multi-task learning across any set of tasks containing spatio-temporal data. Sharing one model across multiple tasks also results in a significant reduction in the total number of parameters. We further demonstrate that this shared model can learn robust representations from various spatio-temporal inputs which are reusable for unseen tasks. We believe that this proposed architecture has wide applicability to any task with spatio-temporal inputs. To extend its usability, we would like to introduce new peripherals supporting more domains such as speech. We are also keen on exploring other aspects to the data beyond temporal and spatial dimensions such as graphs and relational data. Further, it would be interesting to investigate scheduling mechanisms for optimizing the multi-tasking framework.