Self-Enhancing Video Data Management System for Compositional Events with Large Language Models [Technical Report]
<|reference_start|>Self-Enhancing Video Data Management System for Compositional Events with Large Language Models [Technical Report]: Complex video queries can be answered by decomposing them into modular subtasks. However, existing video data management systems assume the existence of predefined modules for each subtask. We introduce VOCAL-UDF, a novel self-enhancing system that supports compositional queries over videos without the need for predefined modules. VOCAL-UDF automatically identifies and constructs missing modules and encapsulates them as user-defined functions (UDFs), thus expanding its querying capabilities. To achieve this, we formulate a unified UDF model that leverages large language models (LLMs) to aid in new UDF generation. VOCAL-UDF handles a wide range of concepts by supporting both program-based UDFs (i.e., Python functions generated by LLMs) and distilled-model UDFs (lightweight vision models distilled from strong pretrained models). To resolve the inherent ambiguity in user intent, VOCAL-UDF generates multiple candidate UDFs and uses active learning to efficiently select the best one. With the self-enhancing capability, VOCAL-UDF significantly improves query performance across three video datasets.<|reference_end|>
@article{zhang2024self-enhancing,
    title={Self-Enhancing Video Data Management System for Compositional Events
  with Large Language Models [Technical Report]},
    author={Enhao Zhang, Nicole Sullivan, Brandon Haynes, Ranjay Krishna,
  Magdalena Balazinska},
    journal={arXiv preprint arXiv:2408.02243},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.02243},
    primaryClass={cs.DB}
}