SGC-VQGAN: Towards Complex Scene Representation via Semantic Guided Clustering Codebook
<|reference_start|>SGC-VQGAN: Towards Complex Scene Representation via Semantic Guided Clustering Codebook: Vector quantization (VQ) is a method for deterministically learning features through discrete codebook representations. Recent works have utilized visual tokenizers to discretize visual regions for self-supervised representation learning. However, a notable limitation of these tokenizers is lack of semantics, as they are derived solely from the pretext task of reconstructing raw image pixels in an auto-encoder paradigm. Additionally, issues like imbalanced codebook distribution and codebook collapse can adversely impact performance due to inefficient codebook utilization. To address these challenges, We introduce SGC-VQGAN through Semantic Online Clustering method to enhance token semantics through Consistent Semantic Learning. Utilizing inference results from segmentation model , our approach constructs a temporospatially consistent semantic codebook, addressing issues of codebook collapse and imbalanced token semantics. Our proposed Pyramid Feature Learning pipeline integrates multi-level features to capture both image details and semantics simultaneously. As a result, SGC-VQGAN achieves SOTA performance in both reconstruction quality and various downstream tasks. Its simplicity, requiring no additional parameter learning, enables its direct application in downstream tasks, presenting significant potential.<|reference_end|>
@article{ding2024sgc-vqgan:,
    title={SGC-VQGAN: Towards Complex Scene Representation via Semantic Guided
  Clustering Codebook},
    author={Chenjing Ding, Chiyu Wang, Boshi Liu, Xi Guo, Weixuan Tang, Wei Wu},
    journal={arXiv preprint arXiv:2409.06105},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.06105},
    primaryClass={cs.CV}
}