Towards Retrieval Augmented Generation over Large Video Libraries
Video content creators need efficient tools to repurpose content, a task that
often requires complex manual or automated searches. Crafting a new video from
large video libraries remains a challenge. In this paper we introduce the task
of Video Library Question Answering (VLQA) through an interoperable
architecture that applies Retrieval Augmented Generation (RAG) to video
libraries. We propose a system that uses large language models (LLMs) to
generate search queries, retrieving relevant video moments indexed by speech
and visual metadata. An answer generation module then integrates user queries
with this metadata to produce responses with specific video timestamps. This
approach shows promise in multimedia content retrieval, and AI-assisted video
content creation.
667920938c58c5a12a9ca8c9
Yannis Tevissen
claimed_verified
2024-06-24T07:32:54.486Z
62cd4fc3299c0c2e0e42f249
https://cdn-avatars.huggingface.co/v1/production/uploads/62cd4fc3299c0c2e0e42f249/4_frS9S9RXVY4fidEoJSZ.jpeg
Yannis Tevissen
user
YannisTevissen
667920938c58c5a12a9ca8ca
Khalil Guetari
claimed_verified
2024-06-24T13:10:29.694Z
64f70494b23f1414f73b1293
/avatars/86c0bc7365c9cbcf09974f492ebc6fe6.svg
Khalil Guetari
user
KhalilGuetari
667920938c58c5a12a9ca8cb
Frédéric Petitpont
Towards Retrieval Augmented Generation over Large Video Libraries
Video content creators need efficient tools to repurpose content, a task that
often requires complex manual or automated searches. Crafting a new video from
large video libraries remains a challenge. In this paper we introduce the task
of Video Library Question Answering (VLQA) through an interoperable
architecture that applies Retrieval Augmented Generation (RAG) to video
libraries. We propose a system that uses large language models (LLMs) to
generate search queries, retrieving relevant video moments indexed by speech
and visual metadata. An answer generation module then integrates user queries
with this metadata to produce responses with specific video timestamps. This
approach shows promise in multimedia content retrieval, and AI-assisted video
content creation.
The proposed system uses Retrieval Augmented Generation (RAG) and large language models to efficiently answer questions about video libraries by generating relevant search queries, retrieving video moments, and producing timestamped responses.
Retrieval Augmented Generation (RAG) large language models (LLMs) multimedia content retrieval AI-assisted video content creation