  In multimedia applications, the text and image components in a web document
form a pairwise constraint that potentially indicates the same semantic
concept. This paper studies cross-modal learning via the pairwise constraint,
and aims to find the common structure hidden in different modalities. We first
propose a compound regularization framework to deal with the pairwise
constraint, which can be used as a general platform for developing cross-modal
algorithms. For unsupervised learning, we propose a cross-modal subspace
clustering method to learn a common structure for different modalities. For
supervised learning, to reduce the semantic gap and the outliers in pairwise
constraints, we propose a cross-modal matching method based on compound ?21
regularization along with an iteratively reweighted algorithm to find the
global optimum. Extensive experiments demonstrate the benefits of joint text
and image modeling with semantically induced pairwise constraints, and show
that the proposed cross-modal methods can further reduce the semantic gap
between different modalities and improve the clustering/retrieval accuracy.

Computer Vision and Pattern Recognition