SEE: Semantically Aligned EEG-to-Text Translation
<|reference_start|>SEE: Semantically Aligned EEG-to-Text Translation: Decoding neurophysiological signals into language is of great research interest within brain-computer interface (BCI) applications. Electroencephalography (EEG), known for its non-invasiveness, ease of use, and cost-effectiveness, has been a popular method in this field. However, current EEG-to-Text decoding approaches face challenges due to the huge domain gap between EEG recordings and raw texts, inherent data bias, and small closed vocabularies. In this paper, we propose SEE: Semantically Aligned EEG-to-Text Translation, a novel method aimed at improving EEG-to-Text decoding by seamlessly integrating two modules into a pre-trained BART language model. These two modules include (1) a Cross-Modal Codebook that learns cross-modal representations to enhance feature consolidation and mitigate domain gap, and (2) a Semantic Matching Module that fully utilizes pre-trained text representations to align multi-modal features extracted from EEG-Text pairs while considering noise caused by false negatives, i.e., data from different EEG-Text pairs that have similar semantic meanings. Experimental results on the Zurich Cognitive Language Processing Corpus (ZuCo) demonstrate the effectiveness of SEE, which enhances the feasibility of accurate EEG-to-Text decoding.<|reference_end|>
@article{tao2024see:,
    title={SEE: Semantically Aligned EEG-to-Text Translation},
    author={Yitian Tao, Yan Liang, Luoyu Wang, Yongqing Li, Qing Yang, and Han
  Zhang},
    journal={arXiv preprint arXiv:2409.16312},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.16312},
    primaryClass={q-bio.QM cs.AI eess.SP}
}