Spatiotemporal Graph Guided Multi-modal Network for Livestreaming Product Retrieval
<|reference_start|>Spatiotemporal Graph Guided Multi-modal Network for Livestreaming Product Retrieval: With the rapid expansion of e-commerce, more consumers have become accustomed to making purchases via livestreaming. Accurately identifying the products being sold by salespeople, i.e., livestreaming product retrieval (LPR), poses a fundamental and daunting challenge. The LPR task encompasses three primary dilemmas in real-world scenarios: 1) the recognition of intended products from distractor products present in the background; 2) the video-image heterogeneity that the appearance of products showcased in live streams often deviates substantially from standardized product images in stores; 3) there are numerous confusing products with subtle visual nuances in the shop. To tackle these challenges, we propose the Spatiotemporal Graphing Multi-modal Network (SGMN). First, we employ a text-guided attention mechanism that leverages the spoken content of salespeople to guide the model to focus toward intended products, emphasizing their salience over cluttered background products. Second, a long-range spatiotemporal graph network is further designed to achieve both instance-level interaction and frame-level matching, solving the misalignment caused by video-image heterogeneity. Third, we propose a multi-modal hard example mining, assisting the model in distinguishing highly similar products with fine-grained features across the video-image-text domain. Through extensive quantitative and qualitative experiments, we demonstrate the superior performance of our proposed SGMN model, surpassing the state-of-the-art methods by a substantial margin. The code is available at https://github.com/Huxiaowan/SGMN.<|reference_end|>
@article{hu2024spatiotemporal,
    title={Spatiotemporal Graph Guided Multi-modal Network for Livestreaming
  Product Retrieval},
    author={Xiaowan Hu, Yiyi Chen, Yan Li, Minquan Wang, Haoqian Wang, Quan Chen,
  Han Li, Peng Jiang},
    journal={arXiv preprint arXiv:2407.16248},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.16248},
    primaryClass={cs.CV cs.MM}
}