Causal Image Modeling for Efficient Visual Understanding
<|reference_start|>Causal Image Modeling for Efficient Visual Understanding: In this work, we present a comprehensive analysis of causal image modeling and introduce the Adventurer series models where we treat images as sequences of patch tokens and employ uni-directional language models to learn visual representations. This modeling paradigm allows us to process images in a recurrent formulation with linear complexity relative to the sequence length, which can effectively address the memory and computation explosion issues posed by high-resolution and fine-grained images. In detail, we introduce two simple designs that seamlessly integrate image inputs into the causal inference framework: a global pooling token placed at the beginning of the sequence and a flipping operation between every two layers. Extensive empirical studies demonstrate the significant efficiency and effectiveness of this causal image modeling paradigm. For example, our base-sized Adventurer model attains a competitive test accuracy of 84.0% on the standard ImageNet-1k benchmark with 216 images/s training throughput, which is 5.3 times more efficient than vision transformers to achieve the same result.<|reference_end|>
@article{wang2024causal,
    title={Causal Image Modeling for Efficient Visual Understanding},
    author={Feng Wang, Timing Yang, Yaodong Yu, Sucheng Ren, Guoyizhe Wei, Angtian
  Wang, Wei Shao, Yuyin Zhou, Alan Yuille, Cihang Xie},
    journal={arXiv preprint arXiv:2410.07599},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.07599},
    primaryClass={cs.CV}
}