DreamDiffusion: Generating High-Quality Images from Brain EEG Signals
This paper introduces DreamDiffusion, a novel method for generating
high-quality images directly from brain electroencephalogram (EEG) signals,
without the need to translate thoughts into text. DreamDiffusion leverages
pre-trained text-to-image models and employs temporal masked signal modeling to
pre-train the EEG encoder for effective and robust EEG representations.
Additionally, the method further leverages the CLIP image encoder to provide
extra supervision to better align EEG, text, and image embeddings with limited
EEG-image pairs. Overall, the proposed method overcomes the challenges of using
EEG signals for image generation, such as noise, limited information, and
individual differences, and achieves promising results. Quantitative and
qualitative results demonstrate the effectiveness of the proposed method as a
significant step towards portable and low-cost ``thoughts-to-image'', with
potential applications in neuroscience and computer vision.
649e4348d61f393407b23980
Yunpeng Bai
649e4348d61f393407b23981
Xintao Wang
admin_assigned
2023-06-30T17:53:08.979Z
60e272ca6c78a8c122b12127
https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
Xintao Wang
user
Xintao
649e4348d61f393407b23982
Yanpei Cao
admin_assigned
2023-06-30T17:53:34.881Z
638066faf022c8a5803f7eb8
/avatars/4cfd699c3f6c5461b12b7dc5e3fe183d.svg
Yanpei Cao
user
pookiefoof
649e4348d61f393407b23983
Yixiao Ge
admin_assigned
2023-06-30T17:53:54.915Z
640e9762b03f4cd29f58d982
/avatars/81da37d628163fe3e094b247c7c3a3b5.svg
Yixiao Ge
user
yxgeee
649e4348d61f393407b23984
Chun Yuan
649e4348d61f393407b23985
Ying Shan
DreamDiffusion: Generating High-Quality Images from Brain EEG Signals
This paper introduces DreamDiffusion, a novel method for generating
high-quality images directly from brain electroencephalogram (EEG) signals,
without the need to translate thoughts into text. DreamDiffusion leverages
pre-trained text-to-image models and employs temporal masked signal modeling to
pre-train the EEG encoder for effective and robust EEG representations.
Additionally, the method further leverages the CLIP image encoder to provide
extra supervision to better align EEG, text, and image embeddings with limited
EEG-image pairs. Overall, the proposed method overcomes the challenges of using
EEG signals for image generation, such as noise, limited information, and
individual differences, and achieves promising results. Quantitative and
qualitative results demonstrate the effectiveness of the proposed method as a
significant step towards portable and low-cost ``thoughts-to-image'', with
potential applications in neuroscience and computer vision.
DreamDiffusion generates high-quality images from EEG signals using pre-trained text-to-image models and CLIP image encoder for improved alignment and robust representations, overcoming noise and data limitations.
temporal masked signal modeling EEG encoder CLIP image encoder EEG signals image generation neuroscience computer vision