Mamba Fusion: Learning Actions Through Questioning
<|reference_start|>Mamba Fusion: Learning Actions Through Questioning: Video Language Models (VLMs) are crucial for generalizing across diverse tasks and using language cues to enhance learning. While transformer-based architectures have been the de facto in vision-language training, they face challenges like quadratic computational complexity, high GPU memory usage, and difficulty with long-term dependencies. To address these limitations, we introduce MambaVL, a novel model that leverages recent advancements in selective state space modality fusion to efficiently capture long-range dependencies and learn joint representations for vision and language data. MambaVL utilizes a shared state transition matrix across both modalities, allowing the model to capture information about actions from multiple perspectives within the scene. Furthermore, we propose a question-answering task that helps guide the model toward relevant cues. These questions provide critical information about actions, objects, and environmental context, leading to enhanced performance. As a result, MambaVL achieves state-of-the-art performance in action recognition on the Epic-Kitchens-100 dataset and outperforms baseline methods in action anticipation.<|reference_end|>
@article{dong2024mamba,
    title={Mamba Fusion: Learning Actions Through Questioning},
    author={Zhikang Dong, Apoorva Beedu, Jason Sheinkopf, Irfan Essa},
    journal={arXiv preprint arXiv:2409.11513},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.11513},
    primaryClass={cs.CV cs.AI}
}