![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00777v3-figure-90540515-7c31-4e8d-afa8-fee6858c52a8.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00777v3-figure-860aec7b-c1da-4c81-98b4-21243a3e5095.jpg)

## CATD: Unified Representation Learning for EEG-to-fMRI Cross-Modal Generation

Weiheng Yao, Zhihan Lyu, Mufti Mahmud, Ning Zhong, Baiying Lei, Shuqiang Wang

Abstract -Multi-modal neuroimaging analysis is crucial for a comprehensive understanding of brain function and pathology, as it allows for the integration of different imaging techniques, thus overcoming the limitations of individual modalities. However, the high costs and limited availability of certain modalities pose significant challenges. To address these issues, this paper proposes the Condition-Aligned Temporal Diffusion (CATD) framework for end-to-end cross-modal synthesis of neuroimaging, enabling the generation of functional magnetic resonance imaging (fMRI)-detected Blood Oxygen Level Dependent (BOLD) signals from more accessible Electroencephalography (EEG) signals. By constructing Conditionally Aligned Block (CAB), heterogeneous neuroimages are aligned into a latent space, achieving a unified representation that provides the foundation for cross-modal transformation in neuroimaging. The combination with the constructed Dynamic Time-Frequency Segmentation (DTFS) module also enables the use of EEG signals to improve the temporal resolution of BOLD signals, thus augmenting the capture of the dynamic details of the brain. Experimental validation demonstrates that the framework improves the accuracy of brain activity state prediction by 9.13% (reaching 69.8%), enhances the diagnostic accuracy of brain disorders by 4.10% (reaching 99.55%), effectively identifies abnormal brain regions, enhancing the temporal resolution of BOLD signals. The proposed framework establishes a new paradigm for cross-modal synthesis of neuroimaging by unifying heterogeneous neuroimaging data into a latent representation space, showing promise in medical applications such as improving Parkinson's disease prediction and identifying abnormal brain regions.

## Index Terms -Cross-Modal Generation, Representation

Weiheng Yao is with Southern University of Science and Technology, Shenzhen, 518055, China.

Weiheng Yao, and Shuqiang Wang are with Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China

Zhihan Lyu is with Department of Game Design, Faculty of Arts, Uppsala University, Sweden

Mufti Mahmud is with Department of Information and Computer Science , SDAIA-KFUPM Joint Research Center for AI, Interdisciplinary Research Center for Biosystems and Machines, King Fahd University of Petroleum and Minerals, Dhahran, Saudi Arabia

Ning Zhong is with the Faculty of Engineering, Maebashi Institute of Technology, Gunma, Japan and he is also with Chongqing University of Posts and Telecommunications,Chongqing, China.

Baiying Lei is with School of Biomedical Engineering, Health Science Center, Shenzhen University, Guangdong Key Laboratory for Biomedical Measurements and Ultrasound Imaging, Shenzhen, 518000, Guangdong, China

©2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

Fig. 1 : Comparison of Advantages and Disadvantages of BOLD fMRI and EEG

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00777v3-figure-1f17f5d0-7e1f-4f88-bada-0323dfff4907.jpg)

Learning, Diffusion Model, Functional Neuroimaging, Temporal Super-Resolution

## I. INTRODUCTION

T HE BOLD signal, measured by fMRI, provides a detailed and precise mapping of brain activity [1]. The sensitivity of this signal to changes in oxygenation and deoxygenation levels in the blood provides a dynamic image of brain function. This helps to understand and track various brain diseases and is considered the gold standard of modern functional neuroimaging [2]. Notably, BOLD fMRI has provided important insights into the brain dynamics of disorders such as ischemic stroke [3], schizophrenia [4], Alzheimer's disease [5], focal epilepsy [6], and depression [7]. This highlights its indispensable role in diagnosing and monitoring these diseases.

Although BOLD fMRI is highly regarded for its detailed imaging capabilities, acquiring such scans is not only costly [8] and time-consuming [9] but also subject to limitations in various clinical applications [10], [11]. Specifically, its poor temporal resolution limits the ability to accurately capture rapid neural dynamics that occur within milliseconds. The hemodynamic response delay inherent to fMRI makes it less suitable for real-time applications, such as brain-machine interfaces or neurofeedback systems [12]. At the same time, the high costs and limited portability of fMRI make it inaccessible for many patients, particularly in low-resource settings. In contrast, EEG has several advantages in reflecting brain activity. Because EEG directly measures brain electrical activity, it provides real-time insight with high temporal resolution, which

is critical for capturing rapid dynamic changes in neuronal circuits [13]. However, EEG's low spatial resolution significantly hinders its ability to localize brain activity accurately, limiting its use in tasks requiring precise spatial mapping [14]. The low signal-to-noise ratio of EEG data and its susceptibility to artifacts (e.g., muscle movement, eye blinks) also pose challenges for reliable interpretation, especially in complex cognitive or clinical scenarios. Fig.1 outlines the respective strengths and limitations of fMRI and EEG.

In recent years, with the advancements in information technology [15], [16] and artificial intelligence techniques [17]-[20], cross-modal neuroimage synthesis has gradually become an active area of research, especially with the wide application of generative models in different fields. Generative AI models, such as transformer [21], diffusion models [22][24] and generative adversarial networks (GANs) [25]-[27], demonstrate great potential in cross-modal data synthesis [28], [29], and their applications in brain imaging [30] and brain function [31], [32] are also gradually increasing. For example, Yan et al. used a deep convolutional generative adversarial network (DCGAN) to reconstruct missing BOLD signals for individual participants [33]. Calhas et al. [34] conducted early research on generative AI for EEG-to-fMRI synthesis, proposing an Autoencoder(AE)-based model for this purpose. In our experiments, we compared our proposed method to their AE-based model and demonstrated significant performance improvements in cross-modal synthesis. The success of generative AI is based on the existence of some of the same underlying information and potentially relevant features in different modalities. Although EEG and fMRI are acquired in different ways, they both reflect brain activity and EEG has the advantage of low cost and fewer limitations on its use. It has been found that there is a strong correlation between microstate transitions in EEG signals and BOLD signals [35] and that BOLD functional connectivity correlates with functional connectivity of EEG activity [36]. By simultaneously recording EEG and fMRI signals, researchers can observe the relationship between EEG activity and metabolic activity in the brain. When electrical brain activity increases, blood oxygen levels in the brain also increase, suggesting a oneto-one correspondence between changes in electrical brain activity and the state of metabolic activity in the brain [37], [38]. This correlation provides a theoretical basis for crossmodal generation and temporal resolution enhancement of fMRI using generative AI and EEG.

EEG and BOLD signals are highly heterogeneous timeseries data, requiring powerful generative AI models for crossmodal synthesis. Currently, diffusion transformer model [39] has demonstrated excellent performance in several areas such as image generation. The emergence of Sora [40] further demonstrates the potential application of diffusion models for temporal data synthesis. Drawing inspiration from these advancements, the CATD framework for the unified representation of EEG and BOLD signals is proposed. This framework is the first to achieve cross-modal synthesis of high-dimensional, heterogeneous brain functional data using a diffusion model. The novelties and contributions of this paper can be summarized in the following points:

- (i) A new paradigm based on generative AI for unified representation of neuroimaging is proposed. As far as we know, it is the first time a diffusion-driven end-toend framework is developed for EEG-to-fMRI synthesis. Utilizing low-cost, accessible EEG signals, the proposed CATD framework is capable of synthesizing high-cost, difficult-to-access BOLD signals. The proposed framework enables high-quality, stable cross-modal generation from EEG to BOLD signals, bridging the gap between different neuroimaging modalities.
- (ii) The CAB module is designed to align high-temporal, lowspatial resolution EEG signals with low-temporal, highspatial resolution BOLD signals within a latent space, facilitating a unified representation across modalities. The combination with DTFS module also leverages EEG's superior temporal resolution to improve the temporal super-resolution of BOLD signals, capturing detailed brain dynamics that outperform traditional methods.

The remainder of this paper is organized as follows. Section II introduces the proposed CATD framework in detail, including its structural design and methodology. Section III describes the experimental setup, datasets used, implementation details, and the experimental results. Section IV provides a comprehensive analysis and discussion of the experimental results as well as the reasons for their occurrence, the applicability and potential of the framework. Finally, Section V concludes the paper by summarizing the key contributions and highlighting future research directions.

## II. METHOD

## A. Overview

BOLD signals are valuable in reflecting brain activity and are essential for analysing, diagnosing and treating brain disorders. However, BOLD signal acquisition is not possible in some patients due to medical conditions or other limitations. In response, the paper proposes the CATD framework, an innovative EEG-to-BOLD signal conversion model based on Scalable Diffusion Models with Transformers (DiT) [39]. This approach addresses the limitations of cross-modal generation and temporal super-resolution of whole-brain BOLD signals, which are not possible with existing techniques. The CATD framework addresses the challenges posed by high dimensionality and asymmetry between EEG and BOLD data through a novel heterogeneous alignment method that facilitates dimensional matching and enhances signal compatibility. At the same time, it employs a cross-attention mechanism to efficiently generate cross-modal EEG-modulated BOLD signals. The proposed DTFS achieves the control of the EEG sampling rate by sliding the sampler, which in turn achieves the enhancement of the temporal resolution of the output BOLD signal. Fig.2 depicts the complete structure and functionality of the CATD framework.

## B. Data Alignment Method

EEG and BOLD signals are high-dimensional signals, one with high temporal resolution and the other with high spatial resolution, highly heterogeneous in scale, and both need to

Fig. 2 : The overall framework of the proposed CATD. The upper part of the figure shows how two different dimensions of data are processed to achieve the initial alignment. The lower half shows the generation pipeline of the BOLD signal under the control of the EEG condition based on the DiT structure.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00777v3-figure-049da2d9-93ed-437d-8d4f-e1b57ac4c03e.jpg)

be processed to the same scale using the following methods to perform integration operations. For preprocessing BOLD signals, we followed the method by Yan et al. [33], utilizing freesurfer [41] software to register brain BOLD activity to the cerebral cortex. Individual structural images were processed to generate cortical surface meshes, and structural and functional images were aligned using boundary-based registration with FsFast. The fMRI data were aligned to the fsaverage template and subsequently downsampled to the fsaverage4 template, creating a functional map with 2562 vertices per hemisphere using the mri surf2surf function in FreeSurfer. Additionally, a 6-mm full-width at half-maximum (FWHM) Gaussian smoothing kernel was applied to the fMRI data in surface space. These processed signals are fed into the encoder of a pretrained Variational Autoencoder (VAE) [42], specifically sdvae-ft-mse from the Diffusers library [43], for dimensionality reduction. The reduced data are then segmented into patches and transformed into input tokens via an embedding layer, producing the initial state x 0 for diffusion model training.

EEG signals, corresponding to the first 6 seconds of each BOLD functional map, are selected due to the 6-second delay between neuronal activity and blood oxygenation response [44], [45]. Feature extraction and dimensionality reduction are performed using a dynamic time-frequency analysis method based on the short-time Fourier transform. The reduced EEG data are then divided into segments and converted into tokens that match the BOLD frames in dimension and number through the EEG embedding layer. These tokens serve as the conditional signals c for the diffusion model, enabling alignment of the high-dimensional, heterogeneous time-series EEG data with BOLD signals.

For the BOLD signal super-resolution task, the sampling rate of the EEG signal is controlled using the DTFS module. During training, overlapping EEG signal samples are employed. Specifically, the EEG signal is sampled at one-third of its original interval while maintaining the same overall sampling duration. This fine-grained temporal segmentation approach allows the generated BOLD signal to achieve three times the temporal resolution of the original signal. As a result, this method generates more refined BOLD signal sequences, significantly enhancing temporal resolution and providing a more detailed representation of brain activity.

## C. EEG to BOLD Diffusion

1) Basic ideas: As one of the highest performing generative AI models available, diffusion models are known for their stable training process and superior quality of generated output. These models work by gradually transforming the data distribution into a Gaussian distribution through a forward process, and then learning the reverse transformation to generate new data samples. In the forward phase, the diffusion model starts with the original data x 0 and gradually increases the noise over multiple time steps, eventually reaching an almost completely random state x T , a process that can be described by the following Gaussian process:

$$q \left ( x _ { t } | x _ { 0 } \right ) = \mathcal { N } \left ( x _ { t } ; \sqrt { \bar { \alpha } _ { t } } x _ { 0 }, \left ( 1 - \bar { \alpha } _ { t } \right ) I \right ), \quad \left ( 1 \right )$$

## Algorithm 1: EEG to BOLD Signal Generation using Diffusion Model: Training Phase

## Define:

F ( x ) : Function to preprocess BOLD data

BOLD F EEG ( c, f s ) : DTFS process, with f s as sample rate Encoder ( x ) : Encoder for fMRI data Decoder ( x ) : Decoder for fMRI data CAB ( c ) : Condition-Aligned Block function α , β t t : Noise schedule parameters ϵ θ : Neural network for predicting noise L ( θ ) : Loss function

## Input:

x ;

// Raw fMRI data

c ;

// Raw EEG data

f s ;

// EEG sample rate

N ;

// Number of diffusion steps

## Get BOLD Data:

x ← Load raw fMRI data x ← F BOLD ( x )

x ← Encoder ( x )

## Initialize Diffusion Model:

x 0 ← x ; // Initialize preprocessed BOLD data

## Forward Diffusion Process:

for t ← 1 to N do

$$\left | \begin{array} { c } \text{Sample } \epsilon _ { t } \sim \mathcal { N } ( 0, I ) \ ; \\ x _ { t } \leftarrow \sqrt { \alpha _ { t } } x _ { t - 1 } + \sqrt { \beta _ { t } } \epsilon _ { t } \end{array}$$

end

## Reverse Denoising Process:

for t ← N to 1 do

$$\Big | \begin{matrix} \hat { \epsilon } _ { t } \leftarrow \epsilon _ { \theta } ( x _ { t }, C A B ( c ) ) \, ; \\ x _ { t - 1 } \leftarrow \frac { 1 } { \sqrt { \alpha _ { t } } } \left ( x _ { t } - \frac { \beta _ { t } } { \sqrt { 1 - \bar { \alpha } _ { t } } } \hat { \epsilon } _ { t } \right ) \end{matrix}$$

end

Loss Calculation and Backpropagation:

L ( θ ) = ∥ ϵ t -ϵ θ ( x , CAB c t ( )) ∥ 2 2

Update model parameters using backpropagation where ¯ α t is a hyperparameter that decreases over time. This process is characterised by a Markov chain and each step is typically governed by a variance-preserving transformation.

The inverse process seeks to reconstruct the original data from the noisy state. This is done by training a neural network to predict the noise added at each step of the forward process, and then iteratively removing this noise to recover the original data x 0 . This inverse process can be described by the following equation:

$$p _ { \theta } \left ( x _ { t - 1 } | x _ { t }, c \right ) = \mathcal { N } \left ( x _ { t - 1 } ; \mu _ { \theta } \left ( x _ { t }, c \right ), \Sigma _ { \theta } \left ( x _ { t }, c \right ) \right ), \ \left ( 2 \right ) \, \frac { \epsilon \alpha \alpha \pi } { c o m n }$$

where µ θ ( x , c t ) and Σ ( θ x , c t ) are predicted by a neural network with parameter θ . We adopt a transformer architecture for this network because of its flexibility and excellent ability to capture long-range dependencies, which makes it with

Algorithm 2: EEG to BOLD Signal Generation using

Diffusion Model: Inference Phase

## Initialize Diffusion Model for Inference:

x T ← Initializewithnoise

N ← Setnumberofdiffusionsteps

## Reverse Denoising Process:

$$\begin{matrix} \overset { \cdots } { \pm } \cdots \overset { \cdots } { \pm } \overset { \cdots } { \pm } \overset { \cdots } { \pm } \\ \text{for $t \gets N$ to $1$ do} \\ \Big | \overset { \hat { \epsilon } _ { t } \gets \epsilon _ { \theta } ( x _ { t }, C A B ( c ) ) \ ; } { x _ { t - 1 } \gets \frac { 1 } { \pm } \left ( x _ { t } - \frac { \beta _ { t } } { \underset { \pm } { \pm } } \hat { \epsilon } _ { t } \right ) } \end{matrix}$$

$$\Big | \begin{smallmatrix} \kappa & \kappa & \kappa \end{smallmatrix}, \begin{smallmatrix} \kappa & \kappa \end{smallmatrix}, \begin{smallmatrix} \kappa & \kappa \end{smallmatrix}, \begin{smallmatrix} \kappa & \kappa \end{smallmatrix}, \begin{smallmatrix} \kappa & \kappa \end{smallmatrix}, \begin{smallmatrix} \kappa & \kappa \end{smallmatrix}, \begin{smallmatrix} \kappa & \kappa \end{smallmatrix}, \begin{smallmatrix} \kappa & \kappa \end{smallmatrix}, \begin{smallmatrix} \kappa & \kappa \end{smallmatrix}, \begin{smallmatrix} \kappa & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}\begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \ kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{\smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{largematrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix } \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmath}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kag \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin {smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \\ \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallformatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix], \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k \\ \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \kappa \end{smallmatrix}, \begin{smallmatrix} \ k & \$$

end

## Decode Generated Signal:

x ← Decoder x ( 0 ) ; // Decode generated BOLD signal

## Output:

Reconstruct final BOLD signal from x ;

particularly suitable for processing time-series signals such as EEG. The inherent attention mechanism of the transformer allows for the integration of conditional information denoted by c through a cross-attention mechanism. This integration guides the generation process, thereby enhancing the model's applicability to cross-modal generation tasks involving highdimensional, heterogeneous data.

2) Architectures: As shown in Fig.2, our proposed EEG to BOLD diffusion model consists of a prediction network that consists of CAB, several layers of Transformer Blocks connected in series, an input embedding layer, and an output section. Each Transformer Block primarily contains one layer of a multi-head self-attention mechanism and one layer of multihead cross-attention mechanism. The cross-attention layer is responsible for incorporating the conditional information into the network. The CAB integrates EEG markers as condition c as well as the diffusion step t, t ∈ { 0 1 , , . . . , T } , and delivers this integrated conditional information to the cross-attention network of each Transformer Module as a way to achieve an effective input of conditional information.

3) Loss Function: According to Eq.1 and Eq.2, the inverse process is trained using a log-likelihood variational lower bound on x 0 , which can be simplified as

$$\underset { \text{data} } { \mathfrak { P } } \text{ is } \quad \mathcal { L } \left ( \theta \right ) = - \, p \left ( x _ { 0 } | x _ { 1 } \right ) + \\ \text{data} \quad \sum _ { t } \mathcal { D } _ { K L } \left ( q ^ { * } \left ( x _ { t - 1 } | x _ { t }, x _ { 0 } \right ) \, \| \, p _ { \theta } \left ( x _ { t - 1 } | x _ { t }, c \right ) \right ),$$

where q ∗ denotes is the true conditional distribution of x t -1 given x t and x 0 , and D KL denotes the KL dispersion of the two distributions.

In discussing the training process of the diffusion model, it is crucial to consider the consistency of the entire generation process, which requires the model to complete the computation of all time steps T before each parameter update. While this approach ensures the comprehensiveness of the learning process, it has a significant negative impact on the convergence speed, stability and optimisation efficiency of the model. At the same time, it also places a high demand

on computational resources. In view of this, we decided to simplify Eq.3. Instead of minimizing the KL divergence to ensure distributional similarity across all time steps, we instead turned to minimising the prediction error in each time step. This approach not only simplifies the computational process but also helps to improve the training efficiency and stability of the model. The expression for the simplified loss function is as follows:

$$\mathcal { L } \left ( \theta \right ) = \left \| \epsilon _ { t } - \epsilon _ { \theta } \left ( x _ { t }, C A B ( c ) \right ) \right \| _ { 2 } ^ { 2 }, \quad \ \ ( 4 ) \ \stackrel { \ u a t e \ \mathfrak { i } } { \text{Squar} }$$

where ϵ θ ( x , CAB c t ( )) denotes the noise predicted by the network and ϵ t denotes the ground truth sampled Gaussian noise.

## III. EXPERIMENT

## A. Experiment Settings

1) Dataset: The experiments were performed on the following datasets: the motor imagery dataset [46], the NODDI dataset [47], [48], and the EEG dataset of Parkinson's patients [49]. For evaluating the generated results, the XP1 part of the motor imagery dataset was used, which contained data from 10 subjects (2 females and 8 males, mean age: 28.4 years ± 10.6 years). Their paired 64-channel EEG and wholebrain fMRI scans were acquired simultaneously using a block design, with each block consisting of a 20-second rest and a 20-second motor imagery. The training set included data from 7 subjects (IDs xp101 to xp103, xp107 to xp110), while the test set consisted of data from 3 subjects (IDs xp104 to xp106). This division ensured the independence of the training and testing samples, providing a reliable assessment of the model's generalization performance across participants. The motor imagery task contained 48 blocks per subject during the test duration. The NODDI dataset included simultaneous resting-state 64-channel EEG and whole-brain fMRI scan data from 17 adult volunteers (11 males and 6 females, mean age: 32.84 ± 8.13 years). For evaluation, data from 2 participants (IDs 48 and 49) were selected as the test set, while the remaining data were used for training. This division allowed the model to accurately generate BOLD signals from EEG data. The EEG dataset of the Parkinson's patients contained 64-channel resting-state EEG data from 8 subjects (4 males and 4 females, mean age: 74.25 years ± 8.75 years). Due to the small sample size and significant clinical variability, data from 4 representative participants were selected based on factors such as age and gender for testing, highlighting the clinical potential of the method in real-world applications. EEG data in all datasets were obtained using the international 10-20 lead system.

2) Implementation Detail: The experiments were conducted on a server platform equipped with two NVIDIA Tesla A800 compute cards. For model parameters, the depth of the Transformer Blocks was set to 12, the hidden space dimension of the patch was 768, and the number of heads in the attention network was 12. Training was performed using the Adam optimizer with an initial learning rate of 0.0001, a batch size of 8, and 1000 epochs. To compute the experimental classification metrics, a five-fold cross-validation method was employed to ensure the reliability of the results. For calculating other quantitative metrics, five independent experiments were conducted, and the results were averaged to ensure data accuracy and stability.

3) Metrics: Variety of categorical metrics were employed, including Accuracy (ACC), Precision (PRE), Sensitivity (SEN), and F1-score, to demonstrate the performance of the synthesized results in downstream tasks. Additionally, to evaluate generation quality in the spatial dimension, Root Mean Square Error (RMSE) and Structural Similarity Index (SSIM) were used. For the temporal dimension, Cosine Similarity and Concordance Correlation Coefficient (CCC) were used to assess generation quality. For the temporal super-resolution experiments, Signal-to-Noise Ratio (SNR) was also used to evaluate the effectiveness of the synthesized signal, with SNR values being base-10 logarithms. To present the results more intuitively, graphical representations of classification and generation performance in different experiments were provided. For example, in the medical decision support experiment, the potential application of the method in medical tasks was illustrated through difference maps of BOLD function maps.

## B. Evaluation of the generated BOLD signal

To assess the effectiveness of the EEG-to-BOLD signal generation, both synthetic and real BOLD signals were utilized to differentiate between subjects' performance in motor imagery and resting states. The brain activity of the subjects was divided into 20-second chunks for both resting and motor imagery tasks. A clear distinction between these states by the model indicated effective learning of brain activity patterns. As shown in Fig.3(a), the cross-modal synthetic BOLD signals outperformed the real data in accuracy and precision. When combined with real data, the classification metrics improved significantly, although recall remained similar to the real data. Ablation experiments demonstrated that CAB enhances the model's ability to use EEG features to constrain BOLD signal synthesis, improving classification metrics compared to models without the CAB. At the same time, the results in Fig.3(b) provide a direct comparison with the existing AE-based model, and the results show that the proposed CATD framework achieves superior performance across all classification metrics.

Spatial (RMSE, SSIM) and temporal (cosine similarity, CCC) metrics of the synthesized BOLD signal were also calculated for the three motor imagery states. The results in Fig.4 show low RMSEs (all below 0.1) and high SSIM (all above 0.6735). The cosine similarity is approximately 0.85, and the CCC value is about 0.8, indicating a high spatio-temporal correlation with the real signal. The ablation experiments (e.g., Fig.3(a) and the No CAB section in Fig.4) confirm the effectiveness of the CAB in improving signal quality.

To further illustrate the capability of the proposed model in learning high-dimensional heterogeneous brain activity features and achieving cross-modal transitions, t-SNE plots were used (Fig.5). The distribution of BOLD signals synthesized with the full CATD framework more closely matches the real signal distribution, underscoring the critical role of the CAB in

Fig. 3 : (a) Radar plot comparing real BOLD signals, generated BOLD signals, their combination, and ablation results of conditioned blocks for motor imagery and resting state classification. (b) Radar plot of the prediction results of motor imagery and rest state for the real BOLD signal, the data generated by our method, and the data generated by the compared AEbased method. (c) Radar plot of BOLD signals synthesized from different EEG frequency bands in motor imagery and resting states using the proposed CATD framework. (d) Radar plot of BOLD signals synthesized from real EEG signals, and their combination, for predicting Parkinson's disease in a clinical decision support experiment.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00777v3-figure-ae54f914-9a44-41e7-afb7-b1ffa76666ce.jpg)

enhancing signal quality. As shown in Fig.5(c), the comparison with the AE-based model demonstrates that the CATD framework not only generates a more accurate distribution of BOLD signals but also effectively captures the underlying structure of the real data, whereas the baseline method exhibits significant deviations from the real signal distribution. This further validates the advantage of the proposed approach in preserving the complex spatio-temporal relationships inherent in brain activity signals.

resolution results were similarly evaluated in three different motor imagery states, as shown in Table I.

## C. Evaluation of temporal resolution enhanced BOLD signals

To verify that the CATD framework can leverage the high temporal resolution of EEG signals to achieve the temporal super-resolution of BOLD signals, temporal resolution enhancement experiments were conducted. The proposed DTFS was used for EEG to achieve triple temporal super-resolution, meaning that the temporal resolution of the generated BOLD signal was three times that of the actual BOLD signal. Since the application scenario for temporal super-resolution typically involves enhancing the existing BOLD signal rather than generating it in the absence of a BOLD signal, the original low temporal resolution BOLD signal was used as a known condition. The enhanced high temporal resolution BOLD signal was obtained by constraining the generated signal using the low-resolution original BOLD signal. The high temporal

An important advantage of high temporal resolution is the potential for signal-to-noise ratio (SNR) enhancement. A higher SNR implies a greater proportion of useful information relative to noise, enabling signals with higher SNR to more reliably reflect actual physiological activity. In the experiments, the SNR values of real and generated BOLD signals were shown in Table I. From the results, it could be seen that the signal-to-noise ratios in the two states were improved except for fmriNF, the motor imagery state. This was also consistent with the results demonstrated by t-SNE above, i.e., the results were worse in the fmriNF state and better in the other two states.

Data visualization was also performed. In the parietal region, where the correlation of motor imagery was strong, a node at the cortex corresponding to the C3 electrode portion of the international 10-20 lead system was selected. Curve graphs were used to display the signal intensities of the generated high temporal-resolution time-series signals and the real signals at this node at the corresponding time points. As shown in Fig.6, the generated high temporal resolution signal and the real signal exhibited a high degree of similarity in trend. This further suggests that the high temporal resolution results obtained using the CATD framework can effectively reflect the trend of blood oxygenation.

TABLE I : Comparison of Time Series Similarity and SNR Between Generated and Real BOLD Signals Across Three Motor Imagery States

|           | Time Series Similarity   | Time Series Similarity   | SNR     | SNR            |
|-----------|--------------------------|--------------------------|---------|----------------|
|           | Cosine Similarity        | CCC                      | Real    | Synthetic      |
| eegNF     | 0.98998±0.00008          | 0.98589±0.00003          | 12.9640 | 12.9995±0.0103 |
| fmriNF    | 0.99187±0.00007          | 0.98714±0.00006          | 13.4383 | 13.3112±0.0137 |
| eegfmriNF | 0.98740±0.00005          | 0.98209±0.00002          | 12.4715 | 13.0708±0.0533 |

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00777v3-figure-d29a0e45-c1c5-48b3-a776-01016784f2d2.jpg)

differ considerably in temporal similarity, resulting in a lower temporal index compared to the blood oxygenation activity reflected by the full-band brain electrical activity, i.e., the real BOLD signal. On the other hand, the gamma band is closely associated with higher cognitive and perceptual functions. In motor imagery tasks, the brain's requirement to understand instructions and make judgments involves higher cognitive and perceptual functions [51]. Therefore, the gamma band most closely matches the real results in both the classification index and timing index. The other three low-frequency bands: delta, theta, and alpha, performed poorly on both the classification metric (e.g., Fig.3(b)) and the temporal similarity metric (e.g., Fig.7), suggesting that these bands negatively affect the generated results by containing less information about the correlation between the EEG and the BOLD signal.

Fig. 4 : Results of quantitative spatial and temporal metrics for synthetic BOLD signals in three states of motor imagery from ablation experiments. The upper half shows spatial metrics, and the lower half shows temporal metrics, with light green indicating use of CAB and teal indicating no CAB.

## D. Evaluation of the effect of temporal features on the generated signals

Different frequency bands of EEG data reflect various types of brain activity, and these differences can significantly impact the results. The effect of EEG frequency bands on BOLD signal generated results was investigated. The EEG signals of different frequency bands were obtained by band-pass filtering, and BOLD signal generation was performed using these signals. As shown in Fig.3(b), the beta and gamma bands, which are related to motor imagery, performed better in the classification metrics. Additionally, the temporal metrics of the generated results were calculated. As shown in Fig.7, the gamma band performed well in temporal metrics, while the beta band exhibited lower temporal performance.

The reasons for these results are analysed below. The brain's state changes in preparation for or during imagined movements are generally slower and smoother, leading to eventrelated desynchronization (ERD) in the beta band [50]. This phenomenon may explain why the beta band excels in motor imagery and resting state categorization, even surpassing the full band's classification performance. However, this characteristic also causes the generated results of the beta band to

## E. Support for medical decision-making

To demonstrate the potential application of the proposed CATD framework in the medical domain, cross-modal generation experiments were conducted on Parkinson's patients to support medical decision-making. In this part of the experiment, the framework was first trained on the NODDI dataset containing paired EEG and BOLD fMRI data from healthy subjects at resting state. Subsequently, a dataset containing only resting-state EEG data from Parkinson's patients was used to reconstruct their cortical BOLD functional maps using the trained cross-modal generation model.

Predictions of Parkinson's disease were performed using the recorded EEG signal, the generated BOLD signal, and their combination. The classification metrics are shown in Fig.3(c). Results indicate that the simultaneous use of generated BOLD signals and recorded EEG signals significantly improves the accuracy of Parkinson's disease prediction and related metrics. This suggests that the CATD framework effectively captures the potential connection between BOLD signals and EEG signals and can be applied across different datasets, which is crucial for disease diagnosis. Consequently, the framework not only enhances the accuracy of existing diagnostic methods but also provides new tools and methodologies for the diagnosis and research of other neurological diseases.

In order to fully utilize the advantage of our CATD framework, i.e., to obtain BOLD signals with high spatial resolution without the condition of fMRI detection, we performed a difference analysis of generated BOLD functional maps in two healthy subjects and two Parkinson's patients. As shown

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00777v3-figure-325ac6a2-10d0-435c-9efb-bf0c2d878611.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00777v3-figure-f9fb1f2c-330c-469c-96b9-012b14f4563d.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00777v3-figure-2649257c-9bfc-4fdd-97de-6e7fc23e315b.jpg)

Fig. 5 : (a) t-SNE plots of synthesized versus real BOLD signal distributions in the CATD framework across two motor imagery states. (b) t-SNE plot of generated versus real BOLD signal distributions after ablation of the CAB. (c) t-SNE plots of BOLD signal distributions generated by the comparison method versus real signals.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00777v3-figure-29b76076-94d7-4e4c-b009-dcaf6473d036.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00777v3-figure-d533ca76-2825-415c-ade6-661cef83809c.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00777v3-figure-83ae9373-0d95-4df2-9f8b-9bb179f4a9b7.jpg)

Fig. 6 : Visualization of low-resolution real BOLD signal points and high-resolution synthetic BOLD signal curves, demonstrating consistent trends.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00777v3-figure-8b443f3e-333f-4e72-b14e-e3efdaa78d51.jpg)

in Fig.8, both showed significant abnormalities in the brain regions marked by red circles. This region is the lingual gyrus, the medial occipito-temporal gyrus and the lateral occipitotemporal gyrus [52], which, in patients with Parkinson's disease, shows significant structural changes [53], and in patients with PD accompanied by visual hallucinations, the atrophy of these three brain regions correlates with the severity of visual

Fig. 7 : Cosine similarity and CCC comparison of the timing of synthesized BOLD signals with real BOLD signals using EEG signals from different frequency bands.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00777v3-figure-bfce1dcf-1b9d-41e8-8df5-36a5f4f9f105.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00777v3-figure-be337dd7-407a-448e-9b5b-53b7e7fd5364.jpg)

hallucinations [54]. This result demonstrates the potential application of the proposed method in localizing regions of abnormal brain activity and further proves its practical value in medical diagnosis.

## IV. DISCUSSION

Cross-modal neuroimage synthesis is becoming crucial in neuroanalysis research. For the first time, we propose the CATD framework based on diffusion models to achieve crossmodal synthesis of temporal functional neuroimages, enabling the conversion of EEG to BOLD signals. This approach addresses limitations in BOLD acquisition, such as the inability of patients with metal implants to undergo fMRI scans. Enhancing BOLD temporal resolution facilitates the

Fig. 8 : Comparison of difference maps of BOLD function maps synthesized using EEG from two healthy subjects and Parkinson's patients, with the portion marked by the red circle being the area of the brain that embodies the abnormality in both difference maps.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00777v3-figure-03ac33a1-9d9a-445b-a509-0735f64a6287.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00777v3-figure-082d58c0-b55b-4a4b-a9c6-93fb365503f1.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00777v3-figure-7ad92972-7eca-4b06-94ba-51b66ceb2fda.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00777v3-figure-acb739cf-cd29-416d-a0d2-5a38e28b1e37.jpg)

study of sub-millimeter cortical structures and activities [55], improves the quality and reliability of functional connectivity and task-driven fMRI research through higher sampling rates [56], and allows for more accurate detection and localization of functional activation regions, capturing transient neural activities [57]. This work provides novel insights for functional neuroimage synthesis [58]. In the following, we will analyze the experimental results in detail to illustrate the performance of our framework.

EEG signals. Lower frequency bands such as beta show the event-related desynchronization (ERD) phenomenon in motor imagery tasks and therefore perform better in classification metrics. Whereas the higher cognitive functions involved in the motor imagery task are mainly determined by the gamma band, the generated results of the gamma band therefore show the highest similarity to the original signal.

In the experiments on the motor imagery dataset, our CATD framework demonstrates its superior performance in crossmodal feature learning and synthesis by detecting categorical metrics for both motor imagery states and resting states. Specifically, the framework is able to capture the features of EEG and BOLD signals well and perform effective crossmodal synthesis. This improved performance can be attributed to the ability to capture complex modes and integrate different modal features in the CATD framework, which enables it to achieve accurate transitions between various signal features. Ablation experiments further validate the effectiveness of the proposed CAB in aligning high-dimensional mismatched data pairs. CAB can learn and capture valid links between EEG and BOLD signals, ensuring the robustness of the model in crossmodal feature capture and synthesis. Compared to the AEbased model, the CATD framework achieves superior results due to the ability to extract advanced representations of brain activity and ensure better alignment of temporal and spatial features.

In the temporal super-resolution experiments, the results exhibit a high correlation in all three motor imagery states, while the signal-to-noise ratio is improved in two states compared to the original signal. This suggests that temporal resolution enhancement is indeed feasible and can provide a more detailed and accurate temporal representation of the BOLD signal.

In order to further verify which EEG frequency bands have a greater impact on the results and have the potential to improve the efficiency of cross-modal synthesis, a frequency band analysis was performed. It was found that the beta band had the greatest impact on the classification results, while the gamma band performed the best in terms of similarity to the real signal. This can be explained by the properties of

The potential of the proposed method to enhance the accuracy of disease diagnosis was validated through disease decision support experiments. The results demonstrate that the method is capable of identifying potentially abnormal regions in the brain by leveraging the high spatial resolution of synthetic BOLD signals, even when only EEG signals are available. This capability contributes to improved diagnostic accuracy.

The CATD framework's ability to synthesize BOLD signals from EEG has significant implications for clinical practice. By providing a non-invasive and cost-effective alternative to conventional fMRI, the framework can increase accessibility for patients who cannot undergo fMRI scans, such as those with metal implants. At the same time, the improved temporal resolution and functional maps generated by the framework can facilitate early diagnosis and intervention in neurological disorders like epilepsy and Parkinson's disease. For example, identifying abnormal brain activity regions with high temporal and spatial resolution could help refine therapeutic strategies. The ability to monitor disease progression using synthetic neuroimages also offers potential for personalized medicine approaches.

An exciting direction for future research is the potential to extend the CATD framework to predict neural activity. Leveraging EEG signals and integrating temporal forecasting techniques within the CATD framework could enable the anticipation of upcoming BOLD signal patterns. While the current implementation does not achieve real-time generation due to the computational complexity of diffusion-based models, advancements in GPU and hardware acceleration technologies could enable real-time prediction in the near future. Realtime capabilities would significantly enhance the framework's applicability in dynamic brain monitoring, real-time neurofeedback systems, and adaptive brain-machine interfaces.

While the results are promising, achieving accurate mapping

Fig. 9 : Typical failure case of our method, showing significant differences in the lower BOLD functional map (near the frontal lobe).

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00777v3-figure-662c63cd-5c2f-4b1a-93f4-5afb946cf3a2.jpg)

from EEG to BOLD signals still presents challenges. Fig.9 illustrates a typical failure case, where the lower regions of the generated BOLD map significantly differ from the real image. This discrepancy is likely due to the low spatial resolution and relatively low signal-to-noise ratio (SNR) of EEG signals. Even though our model successfully learns the relationship between EEG and BOLD signals, the process of upsampling to generate high-resolution images introduces inaccuracies. Addressing these issues of spatial resolution and SNR will be crucial for enhancing the precision of cross-modal synthesis in future work.

In addition to the failure case highlighted in Fig.9, other limitations exist. First, the limited number of paired data subjects constrained the training process. To address the limitations of dataset size, we have carefully selected, to the best of our knowledge, the relatively larger datasets among the publicly available ones with simultaneously acquired EEG and fMRI signals. However, the limited number of paired data subjects remains a constraint and poses challenges to achieving even greater robustness and generalizability. Introducing pre-trained models into the CATD framework could address the data scarcity issue by leveraging transfer learning to improve representation accuracy and enhance the model's feature extraction capability. Second, inter-subject variability in EEG and BOLD signal patterns poses a challenge for achieving consistent cross-modal synthesis across diverse populations. Finally, the computational complexity of the CATD framework, particularly with its reliance on the CAB module and diffusion model, may hinder its scalability for large-scale clinical applications. Addressing these challenges in future work will be essential to fully realize the framework's potential in both research and clinical settings.

## V. CONCLUSION

In this work, a novel CATD framework is proposed for the cross-modal conversion of functional neuroimages, specifically the synthesis of BOLD signals from EEG signals. To fully exploit the high temporal resolution of EEG signals, the DTFS module was designed to increase the sampling rate of the EEG signal as a condition signal, achieving temporal resolution enhancement of the synthesized BOLD signal. By constructing the CAB module, the alignment of high-dimensional heterogeneous functional neuroimages in the hidden space was realized. Qualitative and quantitative experimental results demonstrate that the proposed framework effectively achieves cross-modal synthesis from EEG to BOLD signals. The effectiveness of CAB was validated through ablation experiments, and the framework's value was illustrated in practical application scenarios through medical decision support experiments. Future studies will focus on further optimizing the model and improving the quality of the generated signals to achieve more comprehensive functional neuroimage synthesis.

## REFERENCES

- [1] S. I. Kronemer, M. Aksen, J. Z. Ding, J. H. Ryu, Q. Xin, Z. Ding, J. S. Prince, H. Kwon, A. Khalaf, S. Forman, et al. , 'Human visual consciousness involves large scale cortical and subcortical networks independent of task report and eye movement activity,' Nature Communications , vol. 13, no. 1, p. 7342, 2022.
- [2] A. R. Anwar, M. Muthalib, S. Perrey, A. Galka, O. Granert, S. Wolff, U. Heute, G. Deuschl, J. Raethjen, and M. Muthuraman, 'Effective connectivity of cortical sensorimotor networks during finger movement tasks: a simultaneous fnirs, fmri, eeg study,' Brain topography , vol. 29, pp. 645-660, 2016.
- [3] X. Hou, P. Guo, P. Wang, P. Liu, D. D. Lin, H. Fan, Y. Li, Z. Wei, Z. Lin, D. Jiang, et al. , 'Deep-learning-enabled brain hemodynamic mapping using resting-state fmri,' npj Digital Medicine , vol. 6, no. 1, p. 116, 2023.
- [4] U. Braun, A. Harneit, G. Pergola, T. Menara, A. Sch¨ afer, R. F. Betzel, Z. Zang, J. I. Schweiger, X. Zhang, K. Schwarz, et al. , 'Brain network dynamics during working memory are modulated by dopamine and diminished in schizophrenia,' Nature communications , vol. 12, no. 1, p. 3478, 2021.
- [5] J. Gonneaud, A. T. Baria, A. Pichet Binette, B. A. Gordon, J. P. Chhatwal, C. Cruchaga, M. Jucker, J. Levin, S. Salloway, M. Farlow, et al. , 'Accelerated functional brain aging in pre-clinical familial alzheimer's disease,' Nature communications , vol. 12, no. 1, p. 5346, 2021.
- [6] M. Zijlmans, W. Zweiphenning, and N. van Klink, 'Changing concepts in presurgical assessment for epilepsy surgery,' Nature Reviews Neurology , vol. 15, no. 10, pp. 594-606, 2019.
- [7] A. Talishinsky, J. Downar, P. E. V´ ertes, J. Seidlitz, K. Dunlop, C. J. Lynch, H. Whalley, A. McIntosh, F. Vila-Rodriguez, Z. J. Daskalakis, et al. , 'Regional gene expression signatures are associated with sexspecific functional connectivity changes in depression,' Nature communications , vol. 13, no. 1, p. 5692, 2022.
- [8] C. Demene, J. Baranger, M. Bernal, C. Delanoe, S. Auvin, V. Biran, M. Alison, J. Mairesse, E. Harribaud, M. Pernot, et al. , 'Functional ultrasound imaging of brain activity in human newborns,' Science translational medicine , vol. 9, no. 411, p. eaah6756, 2017.
- [9] G. St-Yves, E. J. Allen, Y. Wu, K. Kay, and T. Naselaris, 'Brainoptimized deep neural network models of human visual areas learn non-hierarchical representations,' Nature communications , vol. 14, no. 1, p. 3329, 2023.
- [10] D. Cruse, S. Chennu, C. Chatelle, T. A. Bekinschtein, D. Fern´ andezEspejo, J. D. Pickard, S. Laureys, and A. M. Owen, 'Bedside detection of awareness in the vegetative state: a cohort study,' The Lancet , vol. 378, no. 9809, pp. 2088-2094, 2011.
- [11] J. N. Keynan, A. Cohen, G. Jackont, N. Green, N. Goldway, A. Davidov, Y. Meir-Hasson, G. Raz, N. Intrator, E. Fruchter, et al. , 'Electrical fingerprint of the amygdala guides neurofeedback training for stress resilience,' Nature human behaviour , vol. 3, no. 1, pp. 63-73, 2019.
- [12] C. C. Casagrande, M. P. Rempe, S. D. Springer, and T. W. Wilson, 'Comprehensive review of task-based neuroimaging studies of cognitive deficits in alzheimer's disease using electrophysiological methods,' Ageing Research Reviews , p. 101950, 2023.
- [13] H. Luo, X. Huang, Z. Li, W. Tian, K. Fang, T. Liu, S. Wang, B. Tang, J. Hu, T.-F. Yuan, et al. , 'An electroencephalography profile of paroxysmal kinesigenic dyskinesia,' Advanced Science , vol. 11, no. 12, p. 2306321, 2024.
- [14] T. Fang, J. Wang, W. Mu, Z. Song, X. Zhang, G. Zhan, P. Wang, J. Bin, L. Niu, L. Zhang, et al. , 'Noninvasive neuroimaging and spatial filter transform enable ultra low delay motor imagery eeg decoding,' Journal of Neural Engineering , vol. 19, no. 6, p. 066034, 2022.
- [15] A. Y. Vaziri, B. Makkiabadi, and N. Samadzadehaghdam, 'Eegg: Generating synthetic eeg signals in matlab environment,' Frontiers in Biomedical Technologies , vol. 10, no. 3, pp. 370-381, 2023.

- [16] M. Bigonah, F. Jamshidi, and D. Marghitu, 'Immersive agricultural education: Gamifying learning with augmented reality and virtual reality,' in Cases on Collaborative Experiential Ecological Literacy for Education , pp. 26-76, IGI Global, 2024.
- [17] J. Lyu, X. Chen, S. A. AlQahtani, and M. S. Hossain, 'Multi-modality mri fusion with patch complementary pre-training for internet of medical things-based smart healthcare,' Information Fusion , p. 102342, 2024.
- [18] N. Fadavi and N. Fadavi, 'Early recognition of parkinson's disease through acoustic analysis and machine learning,' arXiv preprint arXiv:2407.16091 , 2024.
- [19] A. Ganj, M. Ebadpour, M. Darvish, and H. Bahador, 'Lr-net: a blockbased convolutional neural network for low-resolution image classification,' Iranian Journal of Science and Technology, Transactions of Electrical Engineering , vol. 47, no. 4, pp. 1561-1568, 2023.
- [20] M. Mehrnia, E. Kholmovski, A. Katsaggelos, D. Kim, R. Passman, and M. S. Elbaz, 'Novel self-calibrated threshold-free probabilistic fibrosis signature technique for 3d late gadolinium enhancement mri,' IEEE Transactions on Biomedical Engineering , 2024.
- [21] B. Lei, Y. Zhu, E. Liang, P. Yang, S. Chen, H. Hu, H. Xie, Z. Wei, F. Hao, X. Song, T. Wang, X. Xiao, S. Wang, and H. Han, 'Federated domain adaptation via transformer for multi-site alzheimer's disease diagnosis,' IEEE Transactions on Medical Imaging , vol. 42, no. 12, pp. 3651-3664, 2023.
- [22] J. Ho, A. Jain, and P. Abbeel, 'Denoising diffusion probabilistic models,' Advances in neural information processing systems , vol. 33, pp. 68406851, 2020.
- [23] M. Ozbey, O. Dalmaz, S. U. Dar, H. A. Bedel, S ¸. Ozturk, A. G¨ ung¨ or, ¨ ¨ and T. Cukur, 'Unsupervised medical image translation with adversarial ¸ diffusion models,' IEEE Transactions on Medical Imaging , vol. 42, no. 12, pp. 3524-3539, 2023.
- [24] W. Wu, Y. Wang, Q. Liu, G. Wang, and J. Zhang, 'Wavelet-improved score-based generative model for medical imaging,' IEEE Transactions on Medical Imaging , vol. 43, no. 3, pp. 966-979, 2024.
- [25] S. Hu, B. Lei, S. Wang, Y. Wang, Z. Feng, and Y. Shen, 'Bidirectional mapping generative adversarial networks for brain mr to pet synthesis,' IEEE Transactions on Medical Imaging , vol. 41, no. 1, pp. 145-157, 2021.
- [26] B. Hu, C. Zhan, B. Tang, B. Wang, B. Lei, and S.-Q. Wang, '3-d brain reconstruction by hierarchical shape-perception network from a single incomplete image,' IEEE Transactions on Neural Networks and Learning Systems , 2023.
- [27] Y. Xie, Q. Wan, H. Xie, Y. Xu, T. Wang, S. Wang, and B. Lei, 'Fundus image-label pairs synthesis and retinopathy screening via gans with class-imbalanced semi-supervised learning,' IEEE Transactions on Medical Imaging , vol. 42, no. 9, pp. 2714-2725, 2023.
- [28] K. Deng, T. Fei, X. Huang, and Y. Peng, 'Irc-gan: Introspective recurrent convolutional gan for text-to-video generation.,' in IJCAI , pp. 22162222, 2019.
- [29] Y. Takagi and S. Nishimoto, 'High-resolution image reconstruction with latent diffusion models from human brain activity,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 14453-14463, 2023.
- [30] S. You, B. Yuan, Z. Lyu, C. K. Chui, C. P. Chen, B. Lei, and S. Wang, 'Generative ai enables synthesizing cross-modality brain image via multi-level-latent representation learning,' IEEE Transactions on Computational Imaging , 2024.
- [31] Y. Zong, Q. Zuo, M. K.-P. Ng, B. Lei, and S. Wang, 'A new brain network construction paradigm for brain disorder via diffusion-based graph contrastive learning,' IEEE Transactions on Pattern Analysis and Machine Intelligence , 2024.
- [32] S. Xu, Y. Liu, H. Lee, and W. Li, 'Neural interfaces: Bridging the brain to the world beyond healthcare,' in Exploration , p. 20230146, Wiley Online Library, 2024.
- [33] Y. Yan, L. Dahmani, J. Ren, L. Shen, X. Peng, R. Wang, C. He, C. Jiang, C. Gong, Y. Tian, et al. , 'Reconstructing lost bold signal in individual participants using deep machine learning,' Nature communications , vol. 11, no. 1, p. 5046, 2020.
- [34] D. Calhas and R. Henriques, 'Eeg to fmri synthesis: Is deep learning a candidate?,' arXiv preprint arXiv:2009.14133 , 2020.
- [35] O. Al Zoubi, A. Mayeli, M. Misaki, A. Tsuchiyagaito, V. Zotev, H. Refai, M. Paulus, J. Bodurka, et al. , 'Canonical eeg microstates transitions reflect switching among bold resting state networks and predict fmri signal,' Journal of Neural Engineering , vol. 18, no. 6, p. 066051, 2022.
- [36] Y. Huang, P.-H. Wei, L. Xu, D. Chen, Y. Yang, W. Song, Y. Yi, X. Jia, G. Wu, Q. Fan, et al. , 'Intracranial electrophysiological and structural basis of bold functional connectivity in human brain white matter,' Nature Communications , vol. 14, no. 1, p. 3414, 2023.
- [37] D. Kwon, 'Brain imaging: fmri advances make scans sharper and faster.,' Nature , vol. 617, no. 7961, pp. 640-642, 2023.
- [38] D. J. Heeger and D. Ress, 'What does fmri tell us about neuronal activity?,' Nature reviews neuroscience , vol. 3, no. 2, pp. 142-151, 2002.
- [39] W. S. Peebles and S. Xie, 'Scalable diffusion models with transformers,' IEEE International Conference on Computer Vision , 2022.
- [40] Y. Liu, K. Zhang, Y. Li, Z. Yan, C. Gao, R. Chen, Z. Yuan, Y. Huang, H. Sun, J. Gao, L. He, and L. Sun, 'Sora: A review on background, technology, limitations, and opportunities of large vision models,' arXiv preprint arXiv: 2402.17177 , 2024.
- [41] M. Reuter, N. J. Schmansky, H. D. Rosas, and B. Fischl, 'Withinsubject template estimation for unbiased longitudinal image analysis,' Neuroimage , vol. 61, no. 4, pp. 1402-1418, 2012.
- [42] D. P. Kingma and M. Welling, 'Auto-encoding variational bayes,' International Conference on Learning Representations , 2013.
- [43] P. von Platen, S. Patil, A. Lozhkov, P. Cuenca, N. Lambert, K. Rasul, M. Davaadorj, D. Nair, S. Paul, W. Berman, Y. Xu, S. Liu, and T. Wolf, 'Diffusers: State-of-the-art diffusion models,' 2022.
- [44] C. H. Liao, K. J. Worsley, J.-B. Poline, J. A. Aston, G. H. Duncan, and A. C. Evans, 'Estimating the delay of the fmri response,' NeuroImage , vol. 16, no. 3, pp. 593-606, 2002.
- [45] M. R. Burke and G. R. Barnes, 'In pursuit of delay-related brain activity for anticipatory eye movements,' Plos one , vol. 8, no. 9, p. e73326, 2013.
- [46] G. Lioi, C. Cury, L. Perronnet, M. Mano, E. Bannier, A. L´ ecuyer, and C. Barillot, 'Simultaneous eeg-fmri during a neurofeedback task, a brain imaging dataset for multimodal data integration,' Scientific data , vol. 7, no. 1, p. 173, 2020.
- [47] F. Deligianni, D. W. Carmichael, G. H. Zhang, C. A. Clark, and J. D. Clayden, 'Noddi and tensor-based microstructural indices as predictors of functional connectivity,' Plos one , vol. 11, no. 4, p. e0153404, 2016.
- [48] F. Deligianni, M. Centeno, D. W. Carmichael, and J. D. Clayden, 'Relating resting-state fmri and eeg whole-brain connectomes across frequency bands,' Frontiers in neuroscience , vol. 8, p. 98767, 2014.
- [49] J. Cavanagh, 'Eeg: 3-stim auditory oddball and rest in parkinson's,' OpenNeuro, OpenNeuro , 2021.
- [50] G. Pfurtscheller and F. L. Da Silva, 'Event-related eeg/meg synchronization and desynchronization: basic principles,' Clinical neurophysiology , vol. 110, no. 11, pp. 1842-1857, 1999.
- [51] I. Velasco, A. Sipols, C. S. De Blas, L. Pastor, and S. Bayona, 'Motor imagery eeg signal classification with a multivariate time series approach,' BioMedical Engineering OnLine , vol. 22, no. 1, p. 29, 2023.
- [52] C. Destrieux, B. Fischl, A. Dale, and E. Halgren, 'Automatic parcellation of human cortical gyri and sulci using standard anatomical nomenclature,' Neuroimage , vol. 53, no. 1, pp. 1-15, 2010.
- [53] M. Vignando, D. Ffytche, S. J. Lewis, P. H. Lee, S. J. Chung, R. S. Weil, M. T. Hu, C. E. Mackay, L. Griffanti, D. Pins, et al. , 'Mapping brain structural differences and neuroreceptor correlates in parkinson's disease visual hallucinations,' Nature communications , vol. 13, no. 1, p. 519, 2022.
- [54] J. Pagonabarraga, H. Bejr-Kasem, S. Martinez-Horta, and J. Kulisevsky, 'Parkinson disease psychosis: from phenomenology to neurobiological mechanisms,' Nature Reviews Neurology , pp. 1-16, 2024.
- [55] L. Raimondo, T. Knapen, ´ . l A. Oliveira, X. Yu, S. O. Dumoulin, W. van der Zwaag, and J. C. Siero, 'A line through the brain: implementation of human line-scanning at 7t for ultra-high spatiotemporal resolution fmri,' Journal of Cerebral Blood Flow &amp; Metabolism , vol. 41, no. 11, pp. 2831-2843, 2021.
- [56] M. Narsude, D. Gallichan, W. Van Der Zwaag, R. Gruetter, and J. P. Marques, 'Three-dimensional echo planar imaging with controlled aliasing: a sequence for high temporal resolution functional mri,' Magnetic resonance in medicine , vol. 75, no. 6, pp. 2350-2361, 2016.
- [57] A. Y. Petrov, M. Herbst, and V. A. Stenger, 'Improving temporal resolution in fmri using a 3d spiral acquisition and low rank plus sparse (l+ s) reconstruction,' Neuroimage , vol. 157, pp. 660-674, 2017.
- [58] C. Yen, C.-L. Lin, and M.-C. Chiang, 'Exploring the frontiers of neuroimaging: a review of recent advances in understanding brain functioning and disorders,' Life , vol. 13, no. 7, p. 1472, 2023.
Weiheng Yao Zhihan Lyu Mufti Mahmud Ning Zhong Baiying Lei Shuqiang Wang
CATD: Unified Representation Learning for EEG-to-fMRI Cross-Modal Generation
Multi-modal neuroimaging analysis is crucial for a comprehensive
understanding of brain function and pathology, as it allows for the integration
of different imaging techniques, thus overcoming the limitations of individual
modalities. However, the high costs and limited availability of certain
modalities pose significant challenges. To address these issues, this paper
proposes the Condition-Aligned Temporal Diffusion (CATD) framework for
end-to-end cross-modal synthesis of neuroimaging, enabling the generation of
functional magnetic resonance imaging (fMRI)-detected Blood Oxygen Level
Dependent (BOLD) signals from more accessible Electroencephalography (EEG)
signals. By constructing Conditionally Aligned Block (CAB), heterogeneous
neuroimages are aligned into a latent space, achieving a unified representation
that provides the foundation for cross-modal transformation in neuroimaging.
The combination with the constructed Dynamic Time-Frequency Segmentation (DTFS)
module also enables the use of EEG signals to improve the temporal resolution
of BOLD signals, thus augmenting the capture of the dynamic details of the
brain. Experimental validation demonstrates that the framework improves the
accuracy of brain activity state prediction by 9.13% (reaching 69.8%), enhances
the diagnostic accuracy of brain disorders by 4.10% (reaching 99.55%),
effectively identifies abnormal brain regions, enhancing the temporal
resolution of BOLD signals. The proposed framework establishes a new paradigm
for cross-modal synthesis of neuroimaging by unifying heterogeneous
neuroimaging data into a latent representation space, showing promise in
medical applications such as improving Parkinson's disease prediction and
identifying abnormal brain regions.