WeatherFormer: A Pretrained Encoder Model for Learning Robust Weather Representations from Small Datasets
<|reference_start|>WeatherFormer: A Pretrained Encoder Model for Learning Robust Weather Representations from Small Datasets: This paper introduces WeatherFormer, a transformer encoder-based model designed to learn robust weather features from minimal observations. It addresses the challenge of modeling complex weather dynamics from small datasets, a bottleneck for many prediction tasks in agriculture, epidemiology, and climate science. WeatherFormer was pretrained on a large pretraining dataset comprised of 39 years of satellite measurements across the Americas. With a novel pretraining task and fine-tuning, WeatherFormer achieves state-of-the-art performance in county-level soybean yield prediction and influenza forecasting. Technical innovations include a unique spatiotemporal encoding that captures geographical, annual, and seasonal variations, adapting the transformer architecture to continuous weather data, and a pretraining strategy to learn representations that are robust to missing weather features. This paper for the first time demonstrates the effectiveness of pretraining large transformer encoder models for weather-dependent applications across multiple domains.<|reference_end|>
@article{hasan2024weatherformer:,
    title={WeatherFormer: A Pretrained Encoder Model for Learning Robust Weather
  Representations from Small Datasets},
    author={Adib Hasan and Mardavij Roozbehani and Munther Dahleh},
    journal={arXiv preprint arXiv:2405.17455},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.17455},
    primaryClass={cs.CV cs.AI cs.LG physics.ao-ph stat.ML}
}