Multi-Modal Vision Transformers for Crop Mapping from Satellite Image Time Series
<|reference_start|>Multi-Modal Vision Transformers for Crop Mapping from Satellite Image Time Series: Using images acquired by different satellite sensors has shown to improve classification performance in the framework of crop mapping from satellite image time series (SITS). Existing state-of-the-art architectures use self-attention mechanisms to process the temporal dimension and convolutions for the spatial dimension of SITS. Motivated by the success of purely attention-based architectures in crop mapping from single-modal SITS, we introduce several multi-modal multi-temporal transformer-based architectures. Specifically, we investigate the effectiveness of Early Fusion, Cross Attention Fusion and Synchronized Class Token Fusion within the Temporo-Spatial Vision Transformer (TSViT). Experimental results demonstrate significant improvements over state-of-the-art architectures with both convolutional and self-attention components.<|reference_end|>
@article{follath2024multi-modal,
    title={Multi-Modal Vision Transformers for Crop Mapping from Satellite Image
  Time Series},
    author={Theresa Follath, David Mickisch, Jan Hemmerling, Stefan Erasmi, Marcel
  Schwieder and Beg"um Demir},
    journal={arXiv preprint arXiv:2406.16513},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.16513},
    primaryClass={cs.CV}
}