title: OMCAT: Omni Context Aware Transformer
summary: Large Language Models (LLMs) have made significant strides in text generation
  and comprehension, with recent advancements extending into multimodal LLMs that
  integrate visual and audio inputs. However, these models continue to struggle
  with fine-grained, cross-modal temporal understanding, particularly when
  correlating events across audio and video streams. We address these challenges
  with two key contributions: a new dataset and model, called OCTAV and OMCAT
  respectively. OCTAV (Omni Context and Temporal Audio Video) is a novel dataset
  designed to capture event transitions across audio and video. Second, OMCAT
  (Omni Context Aware Transformer) is a powerful model that leverages RoTE
  (Rotary Time Embeddings), an innovative extension of RoPE, to enhance temporal
  grounding and computational efficiency in time-anchored tasks. Through a robust
  three-stage training pipeline-feature alignment, instruction tuning, and
  OCTAV-specific training-OMCAT excels in cross-modal temporal understanding. Our
  model demonstrates state-of-the-art performance on Audio-Visual Question
  Answering (AVQA) tasks and the OCTAV benchmark, showcasing significant gains in
  temporal reasoning and cross-modal alignment, as validated through
  comprehensive experiments and ablation studies. Our dataset and code will be
  made publicly available. The link to our demo page is https://om-cat.github.io.
paper_authors._id: 67107aea2b203d5fcbbe5b5c
paper_authors.name: Arushi Goel
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-10-17T02:48:12.423Z
paper_authors._id: 65c0b5babf441a1953586df7
paper_authors.avatarUrl: /avatars/b767ab08dc096e0aeac29acfcb26d39e.svg
paper_authors.fullname: Arushi Goel
paper_authors.type: user
paper_authors.user: goarushi27
paper_authors._id: 67107aea2b203d5fcbbe5b5d
paper_authors.name: Karan Sapra
paper_authors._id: 67107aea2b203d5fcbbe5b5e
paper_authors.name: Matthieu Le
paper_authors._id: 67107aea2b203d5fcbbe5b5f
paper_authors.name: Rafael Valle
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-02-09T17:18:54.013Z
paper_authors._id: 67a8e3610836885c44bd2ab2
paper_authors.avatarUrl: /avatars/ae8d578a47de3302f07430924db3f105.svg
paper_authors.fullname: Rafael Valle
paper_authors.type: user
paper_authors.user: rafaelvallenvidia
paper_authors._id: 67107aea2b203d5fcbbe5b60
paper_authors.name: Andrew Tao
paper_authors._id: 67107aea2b203d5fcbbe5b61
paper_authors.name: Bryan Catanzaro
paper_title: OMCAT: Omni Context Aware Transformer
paper_summary: Large Language Models (LLMs) have made significant strides in text generation
  and comprehension, with recent advancements extending into multimodal LLMs that
  integrate visual and audio inputs. However, these models continue to struggle
  with fine-grained, cross-modal temporal understanding, particularly when
  correlating events across audio and video streams. We address these challenges
  with two key contributions: a new dataset and model, called OCTAV and OMCAT
  respectively. OCTAV (Omni Context and Temporal Audio Video) is a novel dataset
  designed to capture event transitions across audio and video. Second, OMCAT
  (Omni Context Aware Transformer) is a powerful model that leverages RoTE
  (Rotary Time Embeddings), an innovative extension of RoPE, to enhance temporal
  grounding and computational efficiency in time-anchored tasks. Through a robust
  three-stage training pipeline-feature alignment, instruction tuning, and
  OCTAV-specific training-OMCAT excels in cross-modal temporal understanding. Our
  model demonstrates state-of-the-art performance on Audio-Visual Question
  Answering (AVQA) tasks and the OCTAV benchmark, showcasing significant gains in
  temporal reasoning and cross-modal alignment, as validated through
  comprehensive experiments and ablation studies. Our dataset and code will be
  made publicly available. The link to our demo page is https://om-cat.github.io.
paper_ai_keywords: multimodal LLMs audio and video streams event transitions OCTAV (Omni Context and Temporal Audio Video) OMCAT (Omni Context Aware Transformer) RoTE (Rotary Time Embeddings) RoPE (Rotary Position Embeddings) temporal grounding computational efficiency time-anchored tasks three-stage training pipeline feature tuning instruction tuning OCTAV-specific training Audio-Visual Question Answering (AVQA) FID cross-modal Temporal understanding ablation studies