CViT: Continuous Vision Transformer for Operator Learning
<|reference_start|>CViT: Continuous Vision Transformer for Operator Learning: Operator learning, which aims to approximate maps between infinite-dimensional function spaces, is an important area in scientific machine learning with applications across various physical domains. Here we introduce the Continuous Vision Transformer (CViT), a novel neural operator architecture that leverages advances in computer vision to address challenges in learning complex physical systems. CViT combines a vision transformer encoder, a novel grid-based coordinate embedding, and a query-wise cross-attention mechanism to effectively capture multi-scale dependencies. This design allows for flexible output representations and consistent evaluation at arbitrary resolutions. We demonstrate CViT's effectiveness across a diverse range of partial differential equation (PDE) systems, including fluid dynamics, climate modeling, and reaction-diffusion processes. Our comprehensive experiments show that CViT achieves state-of-the-art performance on multiple benchmarks, often surpassing larger foundation models, even without extensive pretraining and roll-out fine-tuning. Taken together, CViT exhibits robust handling of discontinuous solutions, multi-scale features, and intricate spatio-temporal dynamics. Our contributions can be viewed as a significant step towards adapting advanced computer vision architectures for building more flexible and accurate machine learning models in the physical sciences.<|reference_end|>
@article{wang2024cvit:,
    title={CViT: Continuous Vision Transformer for Operator Learning},
    author={Sifan Wang, Jacob H Seidman, Shyam Sankaran, Hanwen Wang, George J.
  Pappas, Paris Perdikaris},
    journal={arXiv preprint arXiv:2405.13998},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.13998},
    primaryClass={cs.LG stat.ML}
}