Improved Multimodal Deep Learning with Variation of Information
Deep learning has been successfully applied to multimodal representation learning problems, with a common strategy to learning joint representations that are shared across multiple modalities on top of layers of modality-specific networks. Nonetheless, there still remains a question how to learn a good association between data modalities; in particular, a good generative model of multimodal data should be able to reason about missing data modality given the rest of data modalities. In this paper, we propose a novel multimodal representation learning framework that explicitly aims this goal. Rather than learning with maximum likelihood, we train the model to minimize the variation of information. We provide a theoretical insight why the proposed learning objective is sufficient to estimate the data-generating joint distribution of multimodal data. We apply our method to restricted Boltzmann machines and introduce learning methods based on contrastive divergence and multi-prediction training. In addition, we extend to deep networks with recurrent encoding structure to finetune the whole network. In experiments, we demonstrate the state-of-the-art visual recognition performance on MIR-Flickr database and PASCAL VOC 2007 database with and without text features.
This paper proposes a modality reconstruction criterion (given the othermodalities) for training the top level of a multi-modal architecture (withthe lower levels pre-trained by unsupervised learning on a per-modalitybasis, and a separate path for each modality, like in previous multi-modalmodeling with the DBM).The mathematical results are pretty straightforward and unsurprising, andone would think that this methodology is less "powerful" than the maximumlikelihood training, of say DBMs (which have a similar architecture). Yetthe comparative experimental results are impressive and beat the DBMconsistently across many benchmarks. The fact that the whole thing isfine-tuned like in the MP-DBM may be part of the explanation (and it wouldbe good to test it, i.e., what happens without the fine-tuning part?).One thing I don't like, besides the fact that the training criterion is nottrying to model the marginals, is that the conditionals are approximated bya factorial distribution, which seems like a very strong assumption. Isuspect that the reason it seems to work so well is that the tasks involvechoosing one value of a modality given the value observed for the other. Soin a sense we are not even interested in the posterior distribution butmore in the MAP, and a unimodal factorial posterior is then a reasonableway to focus on an estimated MAP. I bet that sampling from one modalitygiven the other would not give so nice results (unlike the case ofthe DBM). But maybe that's ok, and if we really want to solve thekinds of problems in the benchmark, one is really better off trainingin this MAP-like manner than in the full probabilistic mode that DBMs enjoy. So I think that although the ideas in this paper are simple on thesurface, the very surprisingly good results and the questions theseraise are very interesting and worth publishing. I would modify thepaper accordingly, trying to focus the attention, the discussion,and maybe extra experiments, in trying to understand *why* theproposed approach works better than the multi-modal DBM.Toy experiments on MNIST: Note that CD is not recognized to be aparticularly good way of training RBMs in a generative modeling perspective, soyou should really try to train them by PCD/SML instead. And you can'treally call CD-training "maximum likelihood" because it is quite far fromit. PCD/SML is closer to maximum likelihood.Spell out 'GT' in Figure 2.