SketchTriplet: Self-Supervised Scenarized Sketch-Text-Image Triplet Generation
<|reference_start|>SketchTriplet: Self-Supervised Scenarized Sketch-Text-Image Triplet Generation: The scarcity of free-hand sketch presents a challenging problem. Despite the emergence of some large-scale sketch datasets, these datasets primarily consist of sketches at the single-object level. There continues to be a lack of large-scale paired datasets for scene sketches. In this paper, we propose a self-supervised method for scene sketch generation that does not rely on any existing scene sketch, enabling the transformation of single-object sketches into scene sketches. To accomplish this, we introduce a method for vector sketch captioning and sketch semantic expansion. Additionally, we design a sketch generation network that incorporates a fusion of multi-modal perceptual constraints, suitable for application in zero-shot image-to-sketch downstream task, demonstrating state-of-the-art performance through experimental validation. Finally, leveraging our proposed sketch-to-sketch generation method, we contribute a large-scale dataset centered around scene sketches, comprising highly semantically consistent "text-sketch-image" triplets. Our research confirms that this dataset can significantly enhance the capabilities of existing models in sketch-based image retrieval and sketch-controlled image synthesis tasks. We will make our dataset and code publicly available.<|reference_end|>
@article{wu2024sketchtriplet:,
    title={SketchTriplet: Self-Supervised Scenarized Sketch-Text-Image Triplet
  Generation},
    author={Zhenbei Wu, Qiang Wang, Jie Yang},
    journal={arXiv preprint arXiv:2405.18801},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.18801},
    primaryClass={cs.CV}
}