A system (200) for cross-modal data retrieval is provided. The system (200) includes a database (205) for storing training sets of two different modalities of time series and free-form text comments as pairs of mixed modality data. The computer processing system further includes a neural network having a time series encoder (210) and text encoder (215) which are jointly trained using a canonical correlation analysis that finds transformations of feature vectors from among the pairs of mixed modality data such that corelated mixed modality data is emphasized in the two different modalities and uncorrelated mixed modality data is minimized. The feature vectors are obtained by encoding a training set of the time series using the time series encoder and encoding a training set of the free-form text comments using the text encoder.
UNSUPERVISED CONCEPT DISCOVERY AND CROSS-MODAL RETRIEVAL IN TIME SERIES AND TEXT COMMENTS BASED ON CANONICAL CORRELATION ANALYSIS