FoME: A Foundation Model for EEG using Adaptive Temporal-Lateral Attention Scaling
<|reference_start|>FoME: A Foundation Model for EEG using Adaptive Temporal-Lateral Attention Scaling: Electroencephalography (EEG) is a vital tool to measure and record brain activity in neuroscience and clinical applications, yet its potential is constrained by signal heterogeneity, low signal-to-noise ratios, and limited labeled datasets. In this paper, we propose FoME (Foundation Model for EEG), a novel approach using adaptive temporal-lateral attention scaling to address above-mentioned challenges. FoME is pre-trained on a diverse 1.7TB dataset of scalp and intracranial EEG recordings, comprising 745M parameters trained for 1,096k steps. Our model introduces two key innovations: a time-frequency fusion embedding technique and an adaptive time-lateral attention scaling (ATLAS) mechanism. These components synergistically capture complex temporal and spectral EEG dynamics, enabling FoME to adapt to varying patterns across diverse data streams and facilitate robust multi-channel modeling. Evaluations across four downstream tasks demonstrate FoME's superior performance in classification and forecasting applications, consistently achieving state-of-the-art results. To conclude, FoME establishes a new paradigm for EEG analysis, offering a versatile foundation that advances brain-computer interfaces, clinical diagnostics, and cognitive research across neuroscience and related fields. Our code will be available at https://github.com/1061413241/FoME.<|reference_end|>
@article{shi2024fome:,
    title={FoME: A Foundation Model for EEG using Adaptive Temporal-Lateral
  Attention Scaling},
    author={Enze Shi, Kui Zhao, Qilong Yuan, Jiaqi Wang, Huawen Hu, Sigang Yu, Shu
  Zhang},
    journal={arXiv preprint arXiv:2409.12454},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.12454},
    primaryClass={cs.LG cs.AI eess.SP}
}