LEARNING FACTORIZED MULTIMODAL REPRESENTATIONS
Learning multimodal representations is a fundamentally complex research problem due to the presence of multiple heterogeneous sources of information. Although the presence of multiple modalities provides additional valuable information, there are two key challenges to address when learning from multimodal data: 1) models must learn the complex intra-modal and cross-modal interactions for prediction and 2) models must be robust to unexpected missing or noisy modalities during testing. In this paper, we propose to optimize for a joint generative-discriminative objective across multimodal data and labels. We introduce a model that factorizes representations into two sets of independent factors: multimodal discriminative and modality-specific generative factors. Multimodal discriminative factors are shared across all modalities and contain joint multimodal features required for discriminative tasks such as sentiment prediction. Modality-specific generative factors are unique for each modality and contain the information required for generating data. Experimental results show that our model is able to learn meaningful multimodal representations that achieve state-of-the-art or competitive performance on six multimodal datasets. Our model demonstrates flexible generative capabilities by conditioning on independent factors and can reconstruct missing modalities without significantly impacting performance. Lastly, we interpret our factorized representations to understand the interactions that influence multimodal learning.
This paper presents 'Multimodal Factorization model' that factorizes representations into shared multimodal discriminative factors and modality specific generative factors. This work applies 'Wassertein Auto-Encoders' by Tolstikhin et al (with proofs that this setup works in the multimodal case) for handling factorized joint distributions over the multimodal space. Can this method be considered as a generalization of the wasserstein autoencoder based method with a broader application? - the authors should discuss this more broadly in the paper.
Pros:
- There has been many recent work in the area of disentangling joint representations for improving generative auto-encoding architectures using VAEs, GANs, WAE and some variants of these. This work falls in this category with many interesting experiments showing SOTA generation and discrimination results on several tasks.
- This work is practical due to its robustness to noisy and/or missing data for one or more of the modalities in a multimodal machine learning classification (or generation) problem. Application of this technique for continuous multimodal time series data modeling and prediction for high accuracy requirement applications is very promising.
- The methods seems to be easily portable to other tasks. The authors say that they will make the code available to other researchers.
Cons:
- Some more comparison to other disentangling approaches such as beta-VAE, InfoGAN and partitioned VAE methods would have been useful for understanding the advantages and disadvantages of this techniques. (The authors do add a note about comparison with partitioned VAE method in the Appendix)
- For generation and classification tasks, the authors have chosen the tasks for digit recognition and sentiment analysis - I wonder if the results would hold for other types of multimodal tasks.
Overall the paper is very well-written with many experiments to support the claims.