  The ability to jointly learn from multiple modalities, such as text, audio,
and visual data, is a defining feature of intelligent systems. While there have
been promising advances in designing neural networks to harness multimodal
data, the enormous success of data augmentation currently remains limited to
single-modality tasks like image classification. Indeed, it is particularly
difficult to augment each modality while preserving the overall semantic
structure of the data; for example, a caption may no longer be a good
description of an image after standard augmentations have been applied, such as
translation. Moreover, it is challenging to specify reasonable transformations
that are not tailored to a particular modality. In this paper, we introduce
LeMDA, Learning Multimodal Data Augmentation, an easy-to-use method that
automatically learns to jointly augment multimodal data in feature space, with
no constraints on the identities of the modalities or the relationship between
modalities. We show that LeMDA can (1) profoundly improve the performance of
multimodal deep learning architectures, (2) apply to combinations of modalities
that have not been previously considered, and (3) achieve state-of-the-art
results on a wide range of applications comprised of image, text, and tabular
data.

Machine Learning;Computation and Language;Computer Vision and Pattern Recognition