Neural-Driven Image Editing
Traditional image editing typically relies on manual prompting, making it
labor-intensive and inaccessible to individuals with limited motor control or
language abilities. Leveraging recent advances in brain-computer interfaces
(BCIs) and generative models, we propose LoongX, a hands-free image editing
approach driven by multimodal neurophysiological signals. LoongX utilizes
state-of-the-art diffusion models trained on a comprehensive dataset of 23,928
image editing pairs, each paired with synchronized electroencephalography
(EEG), functional near-infrared spectroscopy (fNIRS), photoplethysmography
(PPG), and head motion signals that capture user intent. To effectively address
the heterogeneity of these signals, LoongX integrates two key modules. The
cross-scale state space (CS3) module encodes informative modality-specific
features. The dynamic gated fusion (DGF) module further aggregates these
features into a unified latent space, which is then aligned with edit semantics
via fine-tuning on a diffusion transformer (DiT). Additionally, we pre-train
the encoders using contrastive learning to align cognitive states with semantic
intentions from embedded natural language. Extensive experiments demonstrate
that LoongX achieves performance comparable to text-driven methods (CLIP-I:
0.6605 vs. 0.6558; DINO: 0.4812 vs. 0.4636) and outperforms them when neural
signals are combined with speech (CLIP-T: 0.2588 vs. 0.2549). These results
highlight the promise of neural-driven generative models in enabling
accessible, intuitive image editing and open new directions for
cognitive-driven creative technologies. Datasets and code will be released to
support future work and foster progress in this emerging area.
6874a1ff257d4f0435370344
Pengfei Zhou
6874a1ff257d4f0435370345
Jie Xia
6874a1ff257d4f0435370346
Xiaopeng Peng
6874a1ff257d4f0435370347
Wangbo Zhao
6874a1ff257d4f0435370348
Zilong Ye
6874a1ff257d4f0435370349
Zekai Li
6874a1ff257d4f043537034a
Suorong Yang
6874a1ff257d4f043537034b
Jiadong Pan
6874a1ff257d4f043537034c
Yuanxiang Chen
6874a1ff257d4f043537034d
Ziqiao Wang
6874a1ff257d4f043537034e
Kai Wang
6874a1ff257d4f043537034f
Qian Zheng
6874a1ff257d4f0435370350
Xiaojun Chang
6874a1ff257d4f0435370351
Gang Pan
6874a1ff257d4f0435370352
Shurong Dong
6874a1ff257d4f0435370353
Kaipeng Zhang
6874a1ff257d4f0435370354
Yang You
Neural-Driven Image Editing
Traditional image editing typically relies on manual prompting, making it
labor-intensive and inaccessible to individuals with limited motor control or
language abilities. Leveraging recent advances in brain-computer interfaces
(BCIs) and generative models, we propose LoongX, a hands-free image editing
approach driven by multimodal neurophysiological signals. LoongX utilizes
state-of-the-art diffusion models trained on a comprehensive dataset of 23,928
image editing pairs, each paired with synchronized electroencephalography
(EEG), functional near-infrared spectroscopy (fNIRS), photoplethysmography
(PPG), and head motion signals that capture user intent. To effectively address
the heterogeneity of these signals, LoongX integrates two key modules. The
cross-scale state space (CS3) module encodes informative modality-specific
features. The dynamic gated fusion (DGF) module further aggregates these
features into a unified latent space, which is then aligned with edit semantics
via fine-tuning on a diffusion transformer (DiT). Additionally, we pre-train
the encoders using contrastive learning to align cognitive states with semantic
intentions from embedded natural language. Extensive experiments demonstrate
that LoongX achieves performance comparable to text-driven methods (CLIP-I:
0.6605 vs. 0.6558; DINO: 0.4812 vs. 0.4636) and outperforms them when neural
signals are combined with speech (CLIP-T: 0.2588 vs. 0.2549). These results
highlight the promise of neural-driven generative models in enabling
accessible, intuitive image editing and open new directions for
cognitive-driven creative technologies. Datasets and code will be released to
support future work and foster progress in this emerging area.
LoongX, a hands-free image editing system, uses multimodal neurophysiological signals and diffusion models to achieve performance comparable to text-driven methods and outperforms them when combined with speech.
diffusion models electroencephalography (EEG) functional near-infrared spectroscopy (fNIRS) photoplethysmography (PPG) head motion signals cross-scale state space (CS3) module dynamic gated fusion (DGF) module diffusion transformer (DiT) contrastive learning