In-Context Ensemble Improves Video-Language Models for Low-Level Workflow Understanding from Human Demonstrations
<|reference_start|>In-Context Ensemble Improves Video-Language Models for Low-Level Workflow Understanding from Human Demonstrations: A Standard Operating Procedure (SOP) defines a low-level, step-by-step written guide for a business software workflow based on a video demonstration. SOPs are a crucial step toward automating end-to-end software workflows. Manually creating SOPs can be time-consuming. Recent advancements in large video-language models offer the potential for automating SOP generation by analyzing recordings of human demonstrations. However, current large video-language models face challenges with zero-shot SOP generation. We explore in-context learning with video-language models for SOP generation. We report that in-context learning sometimes helps video-language models at SOP generation. We then propose an in-context ensemble learning to further enhance the capabilities of the models in SOP generation.<|reference_end|>
@article{xu2024in-context,
    title={In-Context Ensemble Learning from Pseudo Labels Improves Video-Language
  Models for Low-Level Workflow Understanding},
    author={Moucheng Xu and Evangelos Chatzaroulas and Luc McCutcheon and Abdul
  Ahad and Hamzah Azeem and Janusz Marecki and Ammar Anwar},
    journal={arXiv preprint arXiv:2409.15867},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.15867},
    primaryClass={cs.AI}
}