HOI-Swap: Swapping Objects in Videos with Hand-Object Interaction Awareness
<|reference_start|>HOI-Swap: Swapping Objects in Videos with Hand-Object Interaction Awareness: We study the problem of precisely swapping objects in videos, with a focus on those interacted with by hands, given one user-provided reference object image. Despite the great advancements that diffusion models have made in video editing recently, these models often fall short in handling the intricacies of hand-object interactions (HOI), failing to produce realistic edits -- especially when object swapping results in object shape or functionality changes. To bridge this gap, we present HOI-Swap, a novel diffusion-based video editing framework trained in a self-supervised manner. Designed in two stages, the first stage focuses on object swapping in a single frame with HOI awareness; the model learns to adjust the interaction patterns, such as the hand grasp, based on changes in the object's properties. The second stage extends the single-frame edit across the entire sequence; we achieve controllable motion alignment with the original video by: (1) warping a new sequence from the stage-I edited frame based on sampled motion points and (2) conditioning video generation on the warped sequence. Comprehensive qualitative and quantitative evaluations demonstrate that HOI-Swap significantly outperforms existing methods, delivering high-quality video edits with realistic HOIs.<|reference_end|>
@article{xue2024hoi-swap:,
    title={HOI-Swap: Swapping Objects in Videos with Hand-Object Interaction
  Awareness},
    author={Zihui Xue, Mi Luo, Changan Chen, Kristen Grauman},
    journal={arXiv preprint arXiv:2406.07754},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.07754},
    primaryClass={cs.CV}
}