Given a video and a sentence, the goal of weakly-supervised video moment retrieval is to locate the video segment which is described by the sentence without having access to temporal annotations during training. Instead, a model must learn how to identify the correct segment (i.e. moment) when only being provided with video-sentence pairs. Thus, an inherent challenge is automatically inferring the latent correspondence between visual and language representations. To facilitate this alignment, we propose our Weakly-supervised Moment Alignment Network (wMAN) which exploits a multi-level co-attention mechanism to learn richer multimodal representations. The aforementioned mechanism is comprised of a Frame-By-Word interaction module as well as a novel Word-Conditioned Visual Graph (WCVG). Our approach also incorporates a novel application of positional encodings, commonly used in Transformers, to learn visual-semantic representations that contain contextual information of their relative positions in the temporal sequence through iterative message-passing. Comprehensive experiments on the DiDeMo and Charades-STA datasets demonstrate the effectiveness of our learned representations: our combined wMAN model not only outperforms the state-of-the-art weakly-supervised method by a significant margin but also obtains an improvement of 10% for the Recall@1 accuracy metric over stronglysupervised state-of-the-art methods on the DiDeMo dataset.
Summary:
This paper proposes a method for aligning an input text with the frames in a video that correspond to what the text describes in a weakly supervised way. The authors propose a combination of a “Frame-By-Word” (FBW) representation and a Word-Conditioned Visual Graph (WCVG). The proposed method outperforms the weakly supervised baseline presented in the paper in experiments by a large margin. In addition, it quantitatively performs close to previous strongly supervised methods.
Pros:
+ New Word-Conditioned Visual Graph representation
+ Outperforms weakly supervised baseline
+ Ablation study of the moving parts
+ Interesting use of positional embeddings for multi-modal learning
Weaknesses / comments:
- What is the processing speed of the method compared to the baseline?
The proposed method makes multiple comparisons while computing the attention weights over all words and frames. Does this cause the method to be slower than the baseline? If so, how much slower is it?
Answers to these questions can help readers to keep in mind the trade-off of the proposed method for achieving the accuracy presented in the paper.
- Number of parameters comparison with baseline:
Did the authors make sure to have similar number of model parameters for the baselines and the proposed method? Maybe I missed it, but I couldn’t see a mention of this anywhere. It would be useful to state this so that readers are sure that it’s not the number of parameters that is helping the method.
- Assumption that sentences are only associated with its ground truth video:
The authors mention that they have the same assumption as Mithun et al., 2019. Can this assumption be detrimental if the dataset does not follow it? Say there are sentences in the dataset that could describe segments in multiple videos. Could this assumption lead to suboptimal representation learning / relationship learning for words / video frames?
- Determining the size of the sliding window:
From reading the paper, it looks like the sliding window used for computing the word / frame relationships has to be manually defined. This seems a bit suboptimal for the generalizability of this method. Do the authors have any comments on this?
- Can this model be supervised? If so, how does it compare to the supervised baselines?
The authors point out that their weakly supervised method performs close to the strongly supervised previously proposed. This is a nice finding, however, have the authors try to answer the question of what would happen if the proposed model is supervised? Will the proposed model outperform the strongly supervised baselines? Or at least perform the same?
Conclusion:
In conclusion, the proposed method makes sense and it has been shown to empirically outperforms a previous weakly supervised baseline. The authors also provide an ablation study of the moving parts to show that the entire pipeline is important to achieve the highest performance in the hardest setting. It would be nice if the authors successfully answer / address the questions / concerns mentioned above in the rebuttal.