In contrast to recent advances focusing on high-level representation learning across modalities, in this work we present a self-supervised learning framework that is able to learn a representation that captures finer levels of granularity across different modalities such as concepts or events represented by visual objects or spoken words. Our framework relies on a discretized embedding space created via vector quantization that is shared across different modalities. Beyond the shared embedding space, we propose a Cross-Modal Code Matching objective that forces the representations from different views (modalities) to have a similar distribution over the discrete embedding space such that cross-modal objects/actions localization can be performed without direct supervision. We show that the proposed discretized multi-modal fine-grained representation (e.g., pixel/word/frame) can complement high-level summary representations (e.g., video/sentence/waveform) for improved performance on cross-modal retrieval tasks. We also observe that the discretized representation uses individual clusters to represent the same semantic concept across modalities. Toddlers acquire much of their knowledge through grounded learning  visual concepts can be acquired through language, and language acquisition emerges through visual interaction. Inspired by this type of grounded learning, a rich body of representation learning research (Harwath et al., 2018; Miech et al., 2020; Alayrac et al., 2020; Monfort et al., 2021; Luo et al., 2021) has been exploring the potential to learn from multi-modal data such as video-text, video-audio, and image-audio pairs. These works typically focus on learning a joint embedding space between different modalities, in which high-level summary representations are extracted as embedding vectors. These embedding vectors often represent entire video clips, spoken utterances, or sentences as single vectors, and can be useful on tasks such as cross-modal data retrieval, e.g., finding the most similar visual scene according to a spoken language description. The predominant approach to learning these embedding vectors is to use modality-independent encoders, and while this has been successful for downstream retrieval tasks, it makes it difficult to compare the activations of the encoders from different modalities. Further, the space of continuous embedding vectors is unbounded, which makes interpreting the learned representations challenging. To this end, we propose to jointly learn high-level embedding vector representations with a fine-grained discrete embedding space that is shared across different modalities. The discrete embedding space enables model interpretability since there are a finite number of embedding vectors which are shared across modalities. Besides the shared embedding space, we propose a CrossModal Code Matching (CMCM) objective that guides the embedding space to capture cross-modal correspondences of concepts, actions, and words. This not only improves downstream performance on retrieval, but also allows us to better interpret what the model recognized through cross-modal grounded learning. To verify the effectiveness of our proposed learning framework, we conducted experiments in several cross-modal domains, including video-text, video-audio, and image-audio. We found consistent improvements over baseline models, verifying that the gain was not restricted to the particular choice of network architecture, input modalities, or dataset. We also demonstrate the interpretability of the fine-grained discrete representations by showing the cross-modal relations between the embedding vectors and semantic concepts appearing in the input modalities. Our approach also enables cross-modal concept localization without requiring any labels during training. We begin by describing the two-branch cross-modal representation learning paradigm in Section 2.1 (the blue and yellow regions). Next, we introduce our shared discrete embedding space in Section 2.2 (the green region). Finally, in Section 2.3 and Figure 2, we introduce the CrossModal Code Matching objective which guides the model to learn semantically meaningful representations through the shared discrete embedding space. Given a set of data X = { ( x Ai , x Bi ) } Ni =1 of size N where each instance x i is instantiated in different modalities A and B (e.g. video and its corresponding caption), the goal is to derive high-level representative vectors ( z Ai , z Bi ) for each instance ( x Ai , x Bi ) that capture the cross-modal relation measured by a choice of similarity function S (  ,  ) . For a specific modality M  { A, B } , a common first step is to encode raw data x Mi into a sequence of fine-grained latent features HM i with a modality-specific neural network f M fine , i.e. H Mi = f M fine ( x Mi ) . The fine-grained representations H Mi can express different kinds of raw data, such as video, audio, or sentences, as a sequence of vectors { h Mi, 1 , ..., h Mi,L } of length L . In the second step, a high-level representation z Mi can be derived by summarizing the fine-grained latent features H Mi with another encoding function f M high that reduces the sequence into a single vector, i.e. z Mi = f M high ( H Mi ) . For example, with modality A being video, raw data x Ai can be treated as a sequence along time and space and encoded into fine-grained representations H Ai = { h Ai,l } Ll =1 by choosing f A fine to be a Residual Network (He et al., 2016). For the second step, a natural choice for f A high to derive the high-level representation z Ai would be a mean pooling function over the time and spatial axes (arranged along l ). With the sets of high-level representations { z Ai } Ni =1 and { z Bj } Nj =1 from different modalities, we can measure the cross-modal relation between any pair of representations ( z Ai , z Bj ) with some similarity function 1 S (  ,  ) . The final step in this paradigm is to adopt an objective function that maximizes the similarity score between positive pairs (where i = j , and thus the true pairs) and minimizes the similarity score between negative pairs (where i  = j , and thus imposter pairs). While different objective functions, such as Semi-Hard Negative Mining (Schroff et al., 2015) (SHN) and Noise Constrastive Estimation (Gut-mann and Hyvrinen, 2010) (NCE), have been studied in prior work, we focused on the Masked Margin Softmax (Ilharco et al., 2019) (MMS) loss LMMS =  1 NN (cid:88) i =1 log e S ( z Ai ,z Bi )  M e S ( z Ai ,z Bi )  M + (cid:80) Nj =1 I i  = j e S ( z Ai ,z Bj ) , (1) where the margin M is a hyperparameter to encourage a higher similarity for positive pairs. The MMS loss LMMS can be seen as an application of the InfoNCE (Oord et al., 2018) loss with a margin. The effectiveness of the described cross-modal learning paradigm has been shown by recent works that achieved state-of-the-art results on benchmark 1 While we used dot product throughtout this work, we also found euclidean distance works well in practice. datasets in different cross-modal scenarios such as video-text (Luo et al., 2021), video-audio (Monfort et al., 2021; Rouditchenko et al., 2020), and image-text (Radford et al., 2021). While the high-level representations ( z Ai , z Bi ) given by the cross-modal learning paradigm benefit end tasks such as data retrieval, the representations cannot be easily interpreted by humans. To obtain fine-grained representations that are more interpretable, we introduce a Vector Quantization (Oord et al., 2017) (VQ) mechanism after obtaining the H Mi representations. Formally, with an auxiliary embedding table E = { e 1 , e 2 , ..., e V } of size V , which we refer to as the codebook , vector quantization is performed on each fine-grained representation h Mi,l  H Mi of modality M  { A, B } with  h Mi,l = f M ( h Mi,l ) + sg ( e v  f M ( h Mi,l )) , where f M is a modality specific projection network to project the input to the shared embedding space, v = arg min k  V  h Mi,l  e k  2 , and sg (  ) is the stop-gradient operator proposed in straight-through gradient estimation (Bengio et al., 2013) that treats the input as constant during backpropagation. In other words, each vector h Mi,l will be replaced by its nearest neighbor e v , which we refer to as the codeword , in the codebook E . The codebook is randomly initialized and updated with the exponential moving average (Oord et al., 2017) given the fine-grained representations (more details in Section A of the Appendix). We trained the shared embedding space jointly with the rest of the framework by modifying the high-level representations z Mi to include the discretized fine-grained representations as z Mi = f M high ( H Mi ) + f M code (  H Mi ) , where f M code is, similar to f M high , the encoding function for summarizing the sequence of quantized fine-grained representations (e.g., an average pooling function over l ). Having such a discrete embedding space allows humans to better interpret the learned embeddings since they are shared across modalities and there are a finite number of them. Ideally, the codebook should be shared across different modalities since the quantization method is independent to the input modality. However, as we demonstrate in Section F of the Appendix, the model will learn to partition the codebook into modality-specific subspaces due to the significant difference between fine-grained representations from different modalities. To learn a shared embedding space that is invariant to input modality, we propose the Cross-Modal Code Matching objective which encourages the model to focus more on the semantic aspect of the input, as illustrated in Figure 2. For each vector h Mi,l in the fine-grained representation sequence H Mi encoded from an instance x Mi of modality M , we first define the probability of h M i,l belonging to the codeword e v as the Softmin function of their Euclidean distance, P ( e v | h Mi,l ) = exp(  f M ( h Mi,l )  e v  2 ) (cid:80) k  V exp(  f M ( h Mi,l )  e k  2 ) . Note that this definition assigns higher a probability to codewords that are closer to the fine-grained representation, where the closest codeword is used to perform vector quantization. We can then define the sequence-level probability distribution over the codebook as the average of the fine-grained distribution, P ( e v | H Mi ) = 1 L (cid:80) l P ( e v | h M i,l ) , which is the normalized frequency of codeword usage for a given sequence of fine-grained representations. Next, for a pair of cross-modal data ( x Ai , x Bj ) , we define their code similar-3015 ity as the negative symmetric cross entropy of probability distribution over the codebook S code ( x Ai , x Bj ) = (cid:80) v P ( e v | H Ai ) log P ( e v | H Bj ) + (cid:80) v P ( e v | H Bj ) log P ( e v | H Ai ) . Intuitively, the proposed objective encourages the model to represent the input ( x Ai , x Bj ) with similar codewords for positive pairs ( i = j ) and nonmatching codewords for negative pairs ( i  = j ). As a consequence, each codeword is expected to be a modality invariant representation of a more fine-grained concept, action, or word that can be discovered from cross-modal data. For example, a codeword could correspond to both the visual scene of a man juggling, and also the spoken word juggling, as we demonstrate in our experimental results in Table 2 and Figure 4. The full objective of our proposed cross-modal representation learning framework is the combination of objectives at different levels L = LMMS +  LCMCM , where  controls the weight between the two terms. Empirically, we found  = 0 . 1 worked well across different settings. Please refer to Section C and D in Appendix for ablation study and comparison to possible alternatives to our method. Examples of the cross-modal learning paradigm. As described in Section 2.1, many of the existing methods for cross-modal learning fit into the paradigm where encoders are modality-independent. This paradigm has been shown to be effective by achieving state-of-the-art retrieval performance on benchmark datasets with the modality pairs that we considered in this work: videotext (Bain et al., 2021; Luo et al., 2021), video-audio (Monfort et al., 2021; Rouditchenko et al., 2020), and image-audio (Harwath et al., 2018, 2020). While these prior works relied on different pre-training datasets, model architectures, and objective functions, they all leverage modality-independent encoders. One of the most important features of this paradigm is the fixed inference time for retrieval. Since the encoders are modality-independent, embedding vectors for samples in a given modality can be computed without using any samples from the other modality. Thus retrieval only involves computing the dot product between embedding vectors from two different modalities. As a consequence, these models are more flexible for large-scale retrieval, and the embedding vectors from each modality can be used independently for other downstream tasks. Other cross-modal learning frameworks. In contrast to the aforementioned works, some methods leverage cross-modal relations within the encoders instead of using modality-independent encoders. This has been done with both cross-modal encoders (Lei et al., 2021; Luo et al., 2021) and cross-modal attention mechanisms (Miech et al., 2018; Liu et al., 2019b,a; Gabeur et al., 2020). However, the cross-modal interactions increase the complexity for retrieval since every instance of a specific modality must be used as input with every instance of another modality to obtain the embedding vectors. With m and n samples in the modalities respectively, this increases the complexity from the modality-independent approach from O ( m + n ) to O ( mn ) . Further, it also makes analysis of the embedding vectors from any individual modality challenging and inhibits single-modality downstream tasks. Our proposed framework builds on the modality-independent approach to enable light-weight retrieval, but it also enables crossmodal interaction through our proposed codebook and Cross-Modal Code Matching objective. Uncovering semantic-level correspondences. Image-audio models have been shown to discover spoken words and visual objects without supervision through retrieval tasks (Synnaeve et al., 2014; Harwath and Glass, 2015; Harwath et al., 2017; Kamper et al., 2018), and the audio embedding vectors have been shown to cluster into word-like speech units (Harwath and Glass, 2017; Wang and Hasegawa-Johnson, 2019; Harwath et al., 2020). Some work has studied the ability of video-audio models to relate spoken words to visual objects and actions in videos (Boggust et al., 2019; Rouditchenko et al., 2020). However, none of these models incorporated a shared embedding space that enabled modality-invariant representations. VQ units have been used in the audio encoder of an image-audio model (Harwath et al., 2020), which allowed it to capture the hierarchical structure of spoken language. While our proposed framework is similar in that it also discretizes the audio sequence 3016 Table 1: Cross-Modal retrieval results on S-MiT, Places, and MSR-VTT. Existing model reproduced with LMMS for fair comparison, see Table 3 in the Appendix for more detail. * Results obtained by running the official code and pre-trained models, see Appendix for more details. with VQ units, our work differs significantly by capturing the cross-modal interactions between visual and audio inputs in the shared embedding space rather than solely capturing the tree structure of speech. Further, besides image-audio data, our proposed framework can handle video-audio and video-text data. To demonstrate the generalizability of the proposed method, we tested our framework on different cross-modal datasets and baseline models that fit into the cross-modal learning paradigm. All setups are listed below and summarized in Table 3 of the Appendix. For training the proposed model, we randomly initialized all the modules related to the discrete shared embedding space and trained them jointly with the rest of the framework (see Figure 1). Unless otherwise specified, (1) we warm-started our proposed framework by initializing it with the modality-specific encoders (namely, f M fine and f M high ) from the baseline models; (2) both the projection network f M and the encoder network f M code are single linear layers; (3) the codebook size is set to 1024. Please refer to Section B in the Appendix for more implementation details. corresponding spoken audio captions averaging 8 seconds. We followed the official protocol to train on the training set of 500k pairs, use the validation set of 10k pairs for development and analysis, and report the retrieval result on a 1k search space over 5 runs randomly sampled from a held-out test set. We selected the same baseline model used on the dataset (Monfort et al., 2021), which contains a visual encoder composed of a ResNet-152 pre-trained on ImageNet (Deng et al., 2009) and TSM ResNet-50 (Lin et al., 2019) pre-trained on M-MiT (Monfort et al., 2019). The audio encoder is a randomly initialized 1D-ResNet (Harwath et al., 2018) designed specifically for spectrograms. The shared embedding space has the dimension of 4096, matching the encoders in the baseline model. Image-Audio: Places (Harwath et al., 2017) contains over 400k pairs of images from the Places 205 dataset (Zhou et al., 2014) and corresponding spoken audio captions averaging 10 seconds. We followed the previous works (Harwath et al., 2018, 2020) to use the training set of 400k pairs and report results on the validation set of 1k pairs. We select ResDAVEnet (Harwath et al., 2018) as the baseline model where the visual encoder is a ResNet-50 pre-trained on ImageNet (Deng et al., 2009) and the audio encoder is a randomly initialized 1D-ResNet (Harwath et al., 2018) designed specifically for spectrograms. The shared embedding space has the dimension of 1024. Video-Text: MSR-VTT (Xu et al., 2016) contains 10k video clips with length varying from 10 to 32 seconds. While each video is provided with 20 related captions for training, we followed the evaluation protocol from previous works (Luo et al., 2021; Gabeur et al., 2020; Yu et al., 2018) to use the training-9k / test 1k-A splits for training and testing respectively. CLIP4Clip (Luo et al., 2021), the current state-of-the-art on MSR-VTT, is selected as the baseline model. Following the crossmodal learning paradigm described in Section 2.1, CLIP4Clip is composed of a pair of encoders: a Visual Transformer (Dosovitskiy et al., 2020) and a Text Transformer (Vaswani et al., 2017). Both encoders are initialized from the CLIP model (Rad-ford et al., 2021), which is pre-trained on the text-image dataset WIT (Radford et al., 2021) and optimized in the end-to-end manner from pixel/text input. For training the proposed framework on top of CLIP4Clip, we freeze the transformers from CLIP4Clip and update only the modules related to the discrete shared embedding space. Both the projection network f M and the encoder network f M code are 4D-Convolutions for video with a depth of 3 and BiLSTMs for text, also with a depth of 3. While CLIP4Clip provided different options for the high-level visual encoder f M high , we adopted the vanilla mean-pooling model. Following CLIP4Clip, the shared embedding space has a dimension of 512. Data retrieval is one of the most common evaluations for cross-modal representation learning. For example, in video retrieval with input query text, videos in the search space will be ranked by the similarity between the representation of each video and the query. We report the standard retrieval metrics recall at rank K (R@K) and median rank (MdR) in Table 1. We show the performance on both visual retrieval, where input language queries are used to retrieve videos or images, and language retrieval, where input visual queries are used to retrieve spoken or text captions. Video-Audio Retrieval. Video-Audio retrieval on S-MiT (Monfort et al., 2021) is a challenging task since videos are paired with raw speech audio, which is untranscribed, unsegmented, and can contain background noise and speaker variation. However, our proposed framework that leverages cross-modal connections between visual actions and spoken words is able to improve the baseline model by a margin. We further analyze our frame-work's ability to relate visual actions and spoken words in Section 4.3. Image-Audio Retrieval. Comparing the baseline model, ResDAVEnet (Harwath et al., 2018), and the current state-of-the-art ResDAVEnet-VQ (Har-wath et al., 2020), the latter model introduces VQ units into the audio encoder, allowing it to model the hierarchical structure of speech and achieve better retrieval results. With our framework, we introduce our shared VQ embedding space into the ResDAVEnet model to capture cross-modal interactions. This improves the performance over both ResDAVEnet and ResDAVEnet-VQ. method against recent works achieving state-of-the-art (Bain et al., 2021; Liu et al., 2021; Luo et al., 2021) and provide a full comparison against more prior work (Liu et al., 2019b; Rouditchenko et al., 2020; Gabeur et al., 2020; Patrick et al., 2020; Dzabraev et al., 2021; Croitoru et al., 2021) in Section E of the Appendix. Frozen-in-Time (Bain et al., 2021) and CLIP4Clip (Luo et al., 2021) are similar methods that employ a Visual Transformer (Dosovitskiy et al., 2020) to encode video as sequence of images. The key differences between them is the choice of summarizing function (i.e. f M high ) for video and the pre-training procedure. We also note that the CLIP4Clip with tight transformer encoder (Luo et al., 2021) (CLIP4Clip-tightT) relied on cross-modal reference via self-attention encoders to derive representations, which has a higher time complexity as mentioned in Section 3. With the shared codebook and Cross-Modal Code Matching objection, our proposed framework also enables cross-modal reference and gives an improvement over the baseline model without increasing the time complexity. Overall, our proposed method enables consistent improvements regardless of the data modalities and baseline architectures, demonstrating its effectiveness and generalizability. One of the important motivations of introducing the discrete cross-modal embedding space is better model interpretability. In this section, we take a closer look into the codewords learned through our proposed framework. For the evaluation, we chose the video-audio setup on S-MiT (Monfort et al., 2021). We used video-audio pairs from the development set, where each pair is labeled with an action out of 332 categories. Note that we only 3019 used labels for analysis, labels are never used for training. Conditional Probability of Action Labels Given Codeword. First, we compute the conditional probability distributions of action labels given the codewords over the video inputs. Each video input is fixed-length and represented by 27 codewords (3 frames each represented by 3  3 code-words), and we labeled all these codewords with the video's action label. By accumulating codeword labels through the whole development set, we can compute the conditional probability of each action given any codeword, i.e. P ( action | codeword ) . Results are visualized in the upper part of Figure 3. Similarly, we computed the conditional probabilities based on the audio input where each utterance is represented by up to 32 codewords depending on the utterance length. We selected the most frequent codewords used by the video inputs and plot the conditional probabilities based on the audio input in the lower part of Figure 3. We can observe that both matrices have similar patterns, i.e., when a codeword is activated, there is a high chance of a specific action appearing in the input regardless if it is video or audio. This suggests that our model is able to learn cross-modal representations for actions grounded by either visual or spoken language input. The codewords are not only modality invariant, but more importantly, they also capture the semantic relations of the labels. e.g., codewords with the highest chance to represent autographing typically have the second highest chance of representing signing; codewords for surfing are less likely to represent other actions as all of them are very different from surfing. We also note that without the Cross-Modal Code Matching objective, semantically related video and audio inputs no longer use the same codewords, which we illustrate in Section F of the Appendix. Cross-Modal Correspondences. Next, we analyze the connections captured by the codewords between action labels and spoken words. With the same label accumulation method described previously, we compute the precision of action prediction with codewords (i.e. code-action co-occurrence code occurrence ). For the audio, we used word-level transcriptions (from Google's speech-to-text API) to assign a spoken word to each codeword when it is activated by the input utterance. This results in a hypothesis set including around 7k words for each codeword, and we listed the top 2 hypotheses for each codeword with the highest F1 score (instead of precision to avoid domination of high-frequency words). Results are listed in Table 2. For the codewords that have the highest precision on predicting the action label, we found the top hypotheses for spoken words are often the action label itself. E.g., the codeword (rank 1st) for the visual action juggling maps to the spoken word juggling perfectly. As precision on visual action prediction decreases, we observed fewer perfect mappings, but the spoken word hypotheses remained semantically related to the visual action hypotheses. E.g., the codeword (rank 35th) for the visual action dunking with lower precision now maps to the spoken word bas-ketball. Surprisingly, even the codewords with the lowest precision capture relationships between visual actions and spoken words to some extent. E.g., codeword (rank 743th) that is most related to the action baking has the top and second word hypotheses cupcake and peanut. Codeword Localization. Finally, to visualize the relation between codewords and the input data, we localize the segments of both the video and audio input that are assigned to certain codewords. This is possible because quantization in our shared embedding space is done at the fine-grained level, so that the time and spatial axes are preserved. Examples are shown in Figure 4, where the regions assigned to the given code are highlighted. Interestingly, we see the codewords being aligned to both the visual actions and the corresponding spoken words. This supports our claim of having a more interpretable representation at the fine-grained level. In this paper, we proposed a framework for crossmodal representation learning with a discrete embedding space that is shared amongst different modalities and enables model interpretability. We also propose a Cross-Modal Code Matching objective that encourages models to represent cross-model semantic concepts in the embedding space. Combining our discrete embedding space and objective with existing cross-modal representation learning models improves retrieval performance on video-text, video-audio, and image-audio datasets. We also analyze the shared embedding space and find that semantically related video and audio inputs tend to use the same codewords. This research was supported in part by the MIT-IBM Watson AI Lab and its member companies, Nexplore and Woodside, and by MIT Lincoln Laboratory.
objective method objective objective result abstain abstain abstain abstain abstain abstain objective abstain objective result objective result objective abstain abstain method objective other abstain abstain abstain abstain abstain method abstain other abstain abstain abstain abstain abstain other method abstain other abstain abstain abstain other abstain method method abstain abstain abstain abstain abstain abstain other other other other other other other other other other other other other other objective other other other other other objective other other objective objective method abstain method method abstain abstain method method abstain abstain abstain method method abstain abstain method abstain abstain abstain method method abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain result result abstain result abstain abstain abstain abstain abstain abstain abstain result abstain objective objective objective result other