Image and Video Tokenization with Binary Spherical Quantization
<|reference_start|>Image and Video Tokenization with Binary Spherical Quantization: We propose a new transformer-based image and video tokenizer with Binary Spherical Quantization (BSQ). BSQ projects the high-dimensional visual embedding to a lower-dimensional hypersphere and then applies binary quantization. BSQ is (1) parameter-efficient without an explicit codebook, (2) scalable to arbitrary token dimensions, and (3) compact: compressing visual data by up to 100$\times$ with minimal distortion. Our tokenizer uses a transformer encoder and decoder with simple block-wise causal masking to support variable-length videos as input. The resulting BSQ-ViT achieves state-of-the-art visual reconstruction quality on image and video reconstruction benchmarks with 2.4$\times$ throughput compared to the best prior methods. Furthermore, by learning an autoregressive prior for adaptive arithmetic coding, BSQ-ViT achieves comparable results on video compression with state-of-the-art video compression standards. BSQ-ViT also enables masked language models to achieve competitive image synthesis quality to GAN- and diffusion-based methods.<|reference_end|>
@article{zhao2024image,
    title={Image and Video Tokenization with Binary Spherical Quantization},
    author={Yue Zhao, Yuanjun Xiong, Philipp Kr"ahenb"uhl},
    journal={arXiv preprint arXiv:2406.07548},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.07548},
    primaryClass={cs.CV cs.IT cs.LG eess.IV math.IT}
}