Enhancing Video-Language Representations with Structural Spatio-Temporal Alignment
<|reference_start|>Enhancing Video-Language Representations with Structural Spatio-Temporal Alignment: While pre-training large-scale video-language models (VLMs) has shown remarkable potential for various downstream video-language tasks, existing VLMs can still suffer from certain commonly seen limitations, e.g., coarse-grained cross-modal aligning , under-modeling of temporal dynamics, detached video-language view. In this work, we target enhancing VLMs with a fine-grained structural spatio-temporal alignment learning method (namely Finsta). First of all, we represent the input texts and videos with fine-grained scene graph (SG) structures, both of which are further unified into a holistic SG (HSG) for bridging two modalities. Then, an SG-based framework is built, where the textual SG (TSG) is encoded with a graph Transformer, while the video dynamic SG (DSG) and the HSG are modeled with a novel recurrent graph Transformer for spatial and temporal feature propagation. A spatial-temporal Gaussian differential graph Transformer is further devised to strengthen the sense of the changes in objects across spatial and temporal dimensions. Next, based on the fine-grained structural features of TSG and DSG, we perform object-centered spatial alignment and predicate-centered temporal alignment respectively, enhancing the video-language grounding in both the spatiality and temporality. We design our method as a plug&play system, which can be integrated into existing well-trained VLMs for further representation augmentation, without training from scratch or relying on SG annotations in downstream applications. On 6 representative VL modeling tasks over 12 datasets in both standard and long-form video scenarios, Finsta consistently improves the existing 13 strong-performing VLMs persistently, and refreshes the current state-of-the-art end task performance significantly in both the fine-tuning and zero-shot settings.<|reference_end|>
@article{fei2024enhancing,
    title={Enhancing Video-Language Representations with Structural Spatio-Temporal
  Alignment},
    author={Hao Fei, Shengqiong Wu, Meishan Zhang, Min Zhang, Tat-Seng Chua,
  Shuicheng Yan},
    journal={[J].IEEE Transactions on Pattern Analysis and Machine
  Intelligence, 2024},
    year={2024},
    doi={10.1109/TPAMI.2024.3393452},
    archivePrefix={arXiv},
    eprint={2406.19255},
    primaryClass={cs.CV cs.CL}
}