title: LITA: Language Instructed Temporal-Localization Assistant
summary: There has been tremendous progress in multimodal Large Language Models
  (LLMs). Recent works have extended these models to video input with promising
  instruction following capabilities. However, an important missing piece is
  temporal localization. These models cannot accurately answer the "When?"
  questions. We identify three key aspects that limit their temporal localization
  capabilities: (i) time representation, (ii) architecture, and (iii) data. We
  address these shortcomings by proposing Language Instructed
  Temporal-Localization Assistant (LITA) with the following features: (1) We
  introduce time tokens that encode timestamps relative to the video length to
  better represent time in videos. (2) We introduce SlowFast tokens in the
  architecture to capture temporal information at fine temporal resolution. (3)
  We emphasize temporal localization data for LITA. In addition to leveraging
  existing video datasets with timestamps, we propose a new task, Reasoning
  Temporal Localization (RTL), along with the dataset, ActivityNet-RTL, for
  learning and evaluating this task. Reasoning temporal localization requires
  both the reasoning and temporal localization of Video LLMs. LITA demonstrates
  strong performance on this challenging task, nearly doubling the temporal mean
  intersection-over-union (mIoU) of baselines. In addition, we show that our
  emphasis on temporal localization also substantially improves video-based text
  generation compared to existing Video LLMs, including a 36% relative
  improvement of Temporal Understanding. Code is available at:
  https://github.com/NVlabs/LITA
paper_authors._id: 660645469f5a7bd168fa41df
paper_authors.name: De-An Huang
paper_authors._id: 660645469f5a7bd168fa41e0
paper_authors.name: Shijia Liao
paper_authors._id: 660645469f5a7bd168fa41e1
paper_authors.name: Subhashree Radhakrishnan
paper_authors._id: 660645469f5a7bd168fa41e2
paper_authors.name: Hongxu Yin
paper_authors._id: 660645469f5a7bd168fa41e3
paper_authors.name: Pavlo Molchanov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-24T07:29:25.549Z
paper_authors._id: 646d0c1c534e52f8c30500a6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646d0c1c534e52f8c30500a6/75VH8ClbRaP75BU2ONfXE.png
paper_authors.fullname: Pavlo Molchanov
paper_authors.type: user
paper_authors.user: pmolchanov
paper_authors._id: 660645469f5a7bd168fa41e4
paper_authors.name: Zhiding Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-29T07:25:15.228Z
paper_authors._id: 66c8037c737ba92ae3fe0322
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66c8037c737ba92ae3fe0322/WR_Yh5DWOVVh7IFlF24NM.jpeg
paper_authors.fullname: Zhiding Yu
paper_authors.type: user
paper_authors.user: Zhiding
paper_authors._id: 660645469f5a7bd168fa41e5
paper_authors.name: Jan Kautz
paper_title: LITA: Language Instructed Temporal-Localization Assistant
paper_summary: There has been tremendous progress in multimodal Large Language Models
  (LLMs). Recent works have extended these models to video input with promising
  instruction following capabilities. However, an important missing piece is
  temporal localization. These models cannot accurately answer the "When?"
  questions. We identify three key aspects that limit their temporal localization
  capabilities: (i) time representation, (ii) architecture, and (iii) data. We
  address these shortcomings by proposing Language Instructed
  Temporal-Localization Assistant (LITA) with the following features: (1) We
  introduce time tokens that encode timestamps relative to the video length to
  better represent time in videos. (2) We introduce SlowFast tokens in the
  architecture to capture temporal information at fine temporal resolution. (3)
  We emphasize temporal localization data for LITA. In addition to leveraging
  existing video datasets with timestamps, we propose a new task, Reasoning
  Temporal Localization (RTL), along with the dataset, ActivityNet-RTL, for
  learning and evaluating this task. Reasoning temporal localization requires
  both the reasoning and temporal localization of Video LLMs. LITA demonstrates
  strong performance on this challenging task, nearly doubling the temporal mean
  intersection-over-union (mIoU) of baselines. In addition, we show that our
  emphasis on temporal localization also substantially improves video-based text
  generation compared to existing Video LLMs, including a 36% relative
  improvement of Temporal Understanding. Code is available at:
  https://github.com/NVlabs/LITA
paper_ai_keywords: time tokens SlowFast tokens temporal localization Reasoning Temporal Localization (RTL) ActivityNet-RTL mIoU (mean intersection-over-union) Temporal Understanding