BELT-2: Bootstrapping EEG-to-Language representation alignment for multi-task brain decoding
<|reference_start|>BELT-2: Bootstrapping EEG-to-Language representation alignment for multi-task brain decoding: The remarkable success of large language models (LLMs) across various multi-modality applications is well established. However, integrating large language models with humans, or brain dynamics, remains relatively unexplored. In this paper, we introduce BELT-2, a pioneering multi-task model designed to enhance both encoding and decoding performance from EEG signals. To bolster the quality of the EEG encoder, BELT-2 is the first work to innovatively 1) adopt byte-pair encoding (BPE)-level EEG-language alignment and 2) integrate multi-task training and decoding in the EEG domain. Inspired by the idea of \textbf{\textit{Bridging the Brain with GPT}}, we further connect the multi-task EEG encoder with LLMs by utilizing prefix-tuning on intermediary output from the EEG encoder. These innovative efforts make BELT-2 a pioneering breakthrough, making it the first work in the field capable of decoding coherent and readable sentences from non-invasive brain signals. Our experiments highlight significant advancements over prior techniques in both quantitative and qualitative measures, achieving a decoding performance with a BLEU-1 score of 52.2\% on the ZuCo dataset. Furthermore, BELT-2 shows a remarkable improvement ranging from 31\% to 162\% on other translation benchmarks. Codes can be accessed via the provided anonymous link~\footnote{https://anonymous.4open.science/r/BELT-2-0048}.<|reference_end|>
@article{zhou2024belt-2:,
    title={BELT-2: Bootstrapping EEG-to-Language representation alignment for
  multi-task brain decoding},
    author={Jinzhao Zhou, Yiqun Duan, Fred Chang, Thomas Do, Yu-Kai Wang,
  Chin-Teng Lin},
    journal={arXiv preprint arXiv:2409.00121},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.00121},
    primaryClass={eess.SP cs.AI cs.LG eess.AS}
}