extfieldsofstudy:
Computer Science
Medicine
Engineering

text:
Bidirectional Mapping Generative Adversarial Networks for Brain MR to PET Synthesis

Fusing multi-modality medical images, such as MR and PET, can provide various anatomical or functional information about human body. But PET data is always unavailable due to different reasons such as cost, radiation, or other limitations. In this paper, we propose a 3D end-to-end synthesis network, called Bidirectional Mapping Generative Adversarial Networks (BMGAN), where image contexts and latent vector are effectively used and jointly optimized for brain MR-to-PET synthesis. Concretely, a bidirectional mapping mechanism is designed to embed the semantic information of PET images into the high dimensional latent space. And the 3D DenseU-Net generator architecture and the extensive objective functions are further utilized to improve the visual quality of synthetic results. The most appealing part is that the proposed method can synthesize the perceptually realistic PET images while preserving the diverse brain structures of different subjects. Experimental results demonstrate that the performance of the proposed method outperforms other competitive cross-modality synthesis methods in terms of quantitative measures, qualitative displays, and classification evaluation.

I. INTRODUCTION
A S the cornerstone for precision medicine, medical images have become a requisite component of public health studies. It contains various imaging modalities that are used to provide an intuitive insight of the interior of the human body in order to assist the radiologists and clinicians to detect or treat diseases more efficiently [1]. It has been widely-recognized that each type of modalities presents various anatomical or functional information about human body. Consequently, complementary imaging modalities are always acquired simultaneously to indicate the disease areas, present the various tissue properties, and help to make an accurate and early diagnosis. For example, recent studies have reported that magnetic resonance (MR) imaging and positron emission tomography (PET) measurements are among the effective biomarkers for Alzheimer Disease (AD) progression and Mild Cognitive Impairment (MCI) conversion prediction [2], [3]. However, patients may not have all demanded imaging modalities due to a variety of reasons. And a typical example is the lack of PET data. Compared to the widely available and non-invasive MR imaging, PET scan is not offered in the majority of medical centers in the world. Moreover, it will increase lifetime cancer risk due to the usage of radioactive tracer. These factors jointly result in the scarcity of PET data.
In fact, the above observations reflect a common dilemma in the clinic analysis of multimodal medical imagessome imaging modalities are unavailable or lacking due to different reasons such as cost, radiation, or other limitations. In such cases, medical imaging synthesis is a novel and effective solution. Given a subjects image in source modality, the cross-modality synthesis of medical images is to accurately estimate the respective image of the same subject in target modality, such as MR-to-PET. Cross-modality synthesis has huge potentials in clinic applications, especially in the scenarios that offers an automatable, low-cost source of highly demanded imaging modality data for clinic diagnosis. In addition, it also can be used as an anonymization tool [4], which means the synthetic images can be shared in the public without caring the protection of personal health information.
Cross-domain synthesis of medical images has recently become one of the mainstream research directions in medical imaging. The existing methods are mainly divided into two categories, including registration-based methods and learningbased methods.
Most registration-based methods [5]- [7] need calculate an atlas according to the co-registered images. But these methods highly relied on the accuracy of registration, and their application is very limited, which suffers from across-subject differences in underlying morphology [8]. An alternative is to use learning-based methods that learning a non-linear mapping from source image to target image. For instance, Jog et al. [9] learned a nonlinear regression with random forest to carry out cross-modality synthesis of high-resolution images from low resolution scans. Huynh et al. [10] presented an approach to synthesize CT from MRI using random forest as well. The disadvantages of these traditional machine learning methods are they largely rely on the handcrafted features and require professional knowledge of domain experts, which results in a challenge for non-experts to make researches in this field.
In recent years, deep learning methods have made a major breakthrough in many fundamental computer vision applications, such as image classification [27] , object detection [28], and image caption [29]. Due to the strong feature extraction and automatic learning capabilities of convolutional neural networks (CNN), it has also achieved excellent performance in the cross-modality synthesis of medical images. To the best of our knowledge, the earliest published literature in this field may be [11]. This paper used a CNN to learn a nonlinear mapping from MR to Fludeoxyglucose PET (FDG-PET). And Xiang et al. [12] presented a deep auto-context CNN that generate full-dose PET images from the low-dose PET images. These models usually used the L 1 or L 2 distance as the loss function, which often results in the blurry estimations of generated images due to the averaged effect. The emergence of generative adversarial networks (GAN) [13] with outstanding generative capabilities has further encouraged the development of medical imaging synthesis. Ben et al. [14] presented a novel system combined by FCN and conditional GAN for the generation of virtual PET images using CT scans. Lei et al. [15] train the deep GAN in a quasi-3D way and simplify the generator, which leads to faster training yet better synthetic quality. Salman et al. [16] proposed a novel pGAN for paired image synthesis, and it yields visually and quantitatively enhanced accuracy in multi-contrast MR synthesis compared to the state-of-the-art methods.
Although these algorithms achieve remarkable results, they are suffer from the same problem: trivially appending a randomly sampled latent vector is difficult to generate plausible images with the diverse structures, because the generator learns to largely ignore the latent vector input without any prior knowledge in the training process of GANs [17]- [20]. Ideally, different latent vector input should correspond to the diverse synthetic images. But the generator is easy to map several different latent vectors to the same output in reality, especially for the generation of brain images that have diverse structural details (e.g. gyri and sulci) between different subjects.
The second problem is that most of previous methods are designed for synthesizing individual 2D slices along one of the axial, coronal and sagittal direction. This results in discontinuous estimation and the loss of spatial information, which are harmful for medical image synthesis. Although some recent works [21]- [23] generate 3D image patches and merge all patches by averaging the overlapped estimation values to obtain the whole 3D estimated image. But learning on imaging patches is insufficient to extract both local and global contextual information among volume voxels, which obviously affects the generative capacity of the deep models.
In this paper, inspired by the appealing success of GANs works [17], [24], [25], we propose a novel 3D end-toend network, called Bidirectional Mapping GAN (BMGAN), where image contexts and latent vector are effectively used and jointly optimized for the brain MR-to-PET synthesis. The novelties and contributions of this paper are as follows. (i) We proposed a novel end-to-end 3D medical imaging synthesis network with a bidirectional mapping mechanism. As an attempt to bridge the gap between network generative capability and real medical images, the proposed method not only focused on synthesizing perceptually realistic PET images from MR images, but also concentrated on reflecting the diverse brain attributes of different subjects. (ii) An advanced 3D DenseU-Net was adopted as the generator architecture to optimally preserved the spatial intrinsic features of medical images and eliminated the slices discontinuity problem caused by 2D network. Besides, the extensive functions containing the adversarial loss, the KL-divergence constraints, the reconstruction loss, and the perceptual loss were devised to further improve the visual quality of synthetic images.
The remaining parts of this paper are organized as follows. Section II provides a detailed description of the proposed BMGAN, including the basic ideas, the DenseU-Net generator architecture, and the extensive objective functions. This section supplies the information needed to the repetition of experiments. The extensive experiments and the ablation study of the proposed method are presented in section III, which demonstrate the performance of the BMGAN. Finally, we summarize the results of this paper and discuss possible future directions for medical image synthesis in section IV.
A preliminary version of this work has been presented at a conference [26]. Herein, we (i) extend our method by replacing the residual paths with the dense connection in the generator. (ii) evaluate and further analyze the contribution of the 3D network architecture and the adversarial training strategy. (iii) evaluate and further analyze the improvement after applying DenseU-Net generator architecture and the extensive objective function. (iv) include additional discussions that are not in the conference publication.

A. Overview
Assuming have a dataset containing the paired brain MR images x ∈ R h×w×d M R and PET images y ∈ R h×w×d P ET , our goal is learning a cross-modality mapping that can be formulated as f : x → y , where f denotes the complex non-linear mapping between the brain MR and PET images. Nevertheless, there are diverse geometric structures among brain anatomies of different subjects, which introduce a huge challenge to the generative capability of cross-modality networks. To address this challenge, we creatively design a 3D generative adversarial network with the bidirectional mechanism. Our proposed BM-GAN is illustrated in Fig. 1, which consists of three components:1) the generator network, 2) the discriminator network, and 3) the encoder network. We firstly propose an advanced DenseU-Net generator that combines the architectures of U-Net [41] and DenseNet [42] to synthesize the targeted PET images from the corresponding brain MR images. It should be stressed that the 3D convolutional operations are adopted to optimally model the spatial structural information of PET images and eliminated the slices discontinuity problem caused by 2D networks. Then we utilize the adversarial learning strategy for the designed network, where an additional patchlevel discriminator network is modeled. Last but not least, we introduce a bidirectional mapping mechanism by adding an encoder network to embed the semantic information of PET images into the latent space, which encourages the generator to preserve the diverse details of brain structures in synthetic PET images. Moreover, our generator is featured by incorporating the adversarial loss, the reconstruction loss, and the perceptual loss into the objective functions, with the goal of improving the visual quality of the synthetic target images. The details of the basic ideas, the architecture, and the extensive objective functions are described in the following Sections 2.1, 2.2 and 2.3.

III. BASIC IDEAS OF BMGAN
GANs [13] are the most popular generative models that have two sub-networks: a generator G and a discriminator D. In such cases, G learns to capture the real data distribution and maps the latent vector to the synthetic images. And D learns to estimate the classification probability of input samples that came from the real image rather than the synthetic images. During its adversarial training mechanism, both G and D are trained simultaneously, with the generator is trained to generate synthetic images that cannot be distinguished from the real images, while the discriminator aims to distinguish the synthetic and real images. Upon convergence, G is capable of producing realistic counterfeit images that D cannot recognize [13].
Mapping from a high-dimensional MR input to a highdimensional PET output is challenging. In order to guarantee the variability of the generated results, the latent vector sampled from a known distribution (such as a standard normal distribution) is usually set as an input of the generator. In this regime, the biggest advantage of the proposed model is that it provides an invertible connection between the brain PET images and the latent vectors, which encourages the generator to synthesize the perceptually realistic PET images while preserving the diverse details of brain structures in different subjects. Specifically, the proposed method not only learn a forward mapping from the latent vector to the PET images like traditional GANs, but also learn a backward mapping that returns the PET images back to the latent space by training an encoder at the same time. This mechanism explicitly encourages a bidirectional consistency between the latent space and the PET images, so as to the semantic information of PET images are embedded into the high-dimensional latent space.
In the training phase of our BMGAN, the forward mapping starts with encoding the PET images into the latent space. To ensure the encoded latent vector have a similar distribution with the sampled latent vector, the encoded vector is trained to conform to the standard normal distribution by optimizing the KL divergence. The generator then tries to map the input MR images and the encoded vector to the synthetic PET images. In the backward mapping, the generator is first used to synthesize PET images from the MR images along with the sampled latent vector. Subsequently, the synthetic PET image is fed to the encoder to attempt to reconstruct the input latent vector. Combining the bidirectional mapping, the proposed method can enforce the generator network to utilize the latent code with the semantic information of PET images. Here as well, the role of discriminator is to try to distinguish the synthetic PET images from the real PET images. In the inference phase of our BMGAN, a deterministic generator uses the MR input images along with the sampled latent vectors to synthesize the corresponding PET images.
A. Architectures 1) DenseU-Net Generator Network: The generator architecture is paramount to the quality of synthetic images. The CNN or FCN architectures are used as the generator in many computer vision researches. Since the original MR images and the targeted PET images belong to the same subject, there are numerous low-level features shared between them. But using the plain CNN or FCN architectures may lead to the information loss of low-level structural features with the network depth increasing, which is harmful for the medical image synthesis. The emergence of U-Net effectively addresses this difficulty. It is comprised of the contracting subnetwork and expanding subnetwork to facilitates the image processing at multi-scales and has been widely applicable for segmenting medical images due to it can be properly trained using little data comparing to the original FCN. Since the researches of medical image synthesis address the pixel-level prediction issue like the image segmentation, it is a practicable attempt to adopt U-Net architectures in medical image synthesis tasks. In particularly, the contracting paths between them ensure the fusion of high-level feature and low-level information, which enhance the spatial and structural information of the synthetic images. Therefore, the U-Net architecture is very suitable as a generator in medical image synthesis tasks, and its effectiveness has been proved in many prior works [15], [21], [24]. But as the model becomes increasingly deeper, an awkward problem emerges: when the input or gradient information passes through many network layers, it will vanish and dilute. To tackle this problem, the short paths from earlier layers to later layers should be enforced. And one of the most effective models in this part is the DenseNet [42]. The advantage of using DenseNet over other methods like ResNet [35] has been well demonstrated in both accuracy and efficiency of many computer vision tasks.
Inspired by the dense connections, we propose a 3D DenseU-Net generator in this work. The main idea of the proposed DenseU-Net is to combine the advantages of both DenseNet and U-Net together. Let x represent the output of th layer. The dense connection can be formally expressed as: where H are the convolutional operations, and [x −1 , · · · , x 0 ] represents the cascade concatenation.
The main difference the DenseU-Net from the naive U-Net is that the densely connected paths between layers are employed within each convolutional blocks of the encoder or decoder part. That means the dense blocks are formed of densely connecting the convolution layers with the same resolution in encoder and decoder part. Hence, the information flow and parameters efficiency of network can be improved for easily training the synthetic models, benefitting from the dense connection. At the same time, the long-range skip connection of U-Net improve the low-level structural feature preservation for better MR-to-PET synthesis.
The proposed DenseU-Net consists of 13 dense blocks, 7 transition layers and 7 upsampling layers, see Fig. 2 dense block includes two directly connected convolutional layers. The filter kernel size of each convolutional layer is 3 × 3 × 3. And the Leaky ReLU with the 0.2 negative slope is selected as the activation function. The number of the filters in each convolutional layer is marked in Fig. 2. To halve the size of output feature maps, the transition layers consisted of a 1 × 1 × 1 convolution layer followed by a max-pooling layer are used in the contracting subnetwork. And the compression rate of dense connection set as 1 in our experiments. In the expanding subnetwork, the upsampling layers with the factor of 2 are implemented by the transposeconvolutional layers. Using the contracting paths, the feature maps from the contracting subnetwork are concatenated with the feature maps of the expanding subnetwork, as denoted by the black straight arrows in Fig. 2. For the normalization layers, we use the instance normalization rather than batch normalization to largely ease the optimization and benefit the generalization of deep networks. It has been proved to have a better performance in image synthetic tasks [33], [34]. In the output layer, the normalization layer is removed and the Tanh function is adopted to obtain the synthetic PET images.

. Each
2) Patch-level Discriminator Network: For the discriminator, a typical CNN architecture including the convolution layer, rectified linear unit (ReLU) activation function and the maxpooling operation is utilized. As shown in Fig. 2, the patchlevel discriminator that classifies if each 32 × 32 × 32 patch of the input image is real or fake is employed rather than the plain pixel-level structure. This structure not only has a few parameters, but also can accurately reconstruct the local brain structure of PET images. The filter size is 3 × 3 × 3, and the numbers of the filters are 32, 64, 128 and 256 for the four convolutional layers, respectively. In the last convolutional layer, sigmoid activation function is used rather than ReLU function to represents the likelihood that the input is drawn from the real images.
3) ResNet Encoder Network: For the encoder, the ResNet architecture is deployed to instead of plain CNN to better encode the images. We follow the ResNet-34 network architecture with multiple Convolution-Normalization-Relu-Pooling components and several residual mapping to build the encoder network. The details of ResNet-34 can be found in [35].

B. Objective functions
Success in imaging synthesis tasks requires semantic reasoning. For MR-to-PET synthesis the synthetic PET output must be semantically similar to the corresponding MR input despite huge changes in appearance. In order to further improve the quality of generated PET images, an extensive objective function is used to optimize the network, including four types of terms: the adversarial loss, the KL-divergence constraint, the pixel-wise reconstruction loss, and the perceptual loss. The details will be introduced as follow.
The significance of the adversarial loss is to match the synthetic data distribution to the target data distribution. During the adversarial training process of GANs, the generator and discriminator are learned simultaneously. However, the adversarial objective of vanilla GAN cannot exactly reveal the inconsistency between the synthetic data distribution and the target data distribution. In such situation where the information between the original domain and target domain differs too much (such as MR-to-PET), the adversarial objective of vanilla GAN is hardly able to optimally converge, thus suffering from the mode collapse problem.
To deal with this challenge, the least-square adversarial objective function [30] is adopted to replace the objective of vanilla GAN in the proposed BMGAN, which turns out to be stable enough, thus remarkably addressing the convergence or mode collapse problem. Given an MR slice x ∼ P M R (x) and its corresponding PET image y ∼ P P ET (y), the loss objective can be expressed as (2), (3): To ensure the encoded vector have a similar distribution with the sampled latent vector, we enforce the KL-divergence constraint in encoder network, which means the difference between the encoded vector and the gaussian latent vector should be minimized, as shown below: where E denotes expected value, ∅ denotes KL divergence and z ∼ P z (z) represents sampling a latent vector from the Gaussian latent space.
Minimizing the adversarial objective function means that the generator G is making the discriminator D as confused as possible, in the sense that D cannot correctly discriminate the synthetic images and the real images. However, it is irresponsible for guaranteeing the structure of synthetic regions is consistent with that of real regions. For example, G always generate PET images that can confuse D but without being close to the real images. To address the problem, we introduce some extensive objective functions.
First, an L 1 pixel-wise reconstruction loss is incorporated to impose an additional structural constraint on the generator. That means the generator is not only needed to fool the discriminator, but also needed to minimize the absolute pixelwise intensity difference between synthetic PET images and real PET images. The L 1 pixel-wise reconstruction loss is formalized as follows: Unfortunately, despite the pixel-wise loss is able to capture the overall structure, it fails to capture perceptual differences between the synthetic and real images. For example, considering two identical MR images offset from each other by one pixel, they would be very different as measured by pixel-wise loss despite their perceptual similarity [31]. Accordingly, a perceptual loss that depends on the dissimilarities between highlevel feature representations is further introduced to generate PET images the higher quality, which is formalized as follows: where V is a set of feature maps right before the second maxpooling operation of pre-trained VGG-16 [32]. The aggregate objective functions of generator G can be written as: where λ 1 and λ 2 are the hyper-parameters for controlling the relative importance of individual loss terms.

A. Experiment settings
Alzheimers disease (AD) is a neurological and irreversible brain disease. The accurate diagnosis combining the different imaging modalities such as MR and PET plays a significant role in patient care, especially at the early stage. One of the impactive public neuroimaging datasets for the AD diagnosis is the Alzheimers Disease Neuroimaging Initiative (ADNI) [38], which includes MR, PET, and other imaging of different modalities.
The evaluated data of the proposed method was obtained from a subset of ADNI database, including 680 subjects. And each subject had images of two modalities (i.e. MR and PET).
In the experiments, the proposed method is to learn the nonlinear mapping from the MR images to the FDG-PET images. To avoid the influence of redundant information, the nonbrain tissues between the MR images and PET images were removed. Moreover, the two modalities were spatially aligned to the same standardized template space to make them rigidly aligned with each other. To reduce the computational cost, we resized both the MR images and PET images to 128 128 128 voxels. To make full use of the available data, the 10-fold cross-validation was performed, where 7 folds were used as the training set, 1 fold for the validation set and remaining 2 fold was used as the test set. The proposed BMGAN method was implemented in Python, using the PyTorch deep learning framework. And all experiments were conducted out on an NVIDIA GeForce GTX 2080 Ti GPU.
In this work, we employ the following four metrics to evaluate the performance in the synthetic medical images, including the mean absolute error (MAE), the peak-signalto-noise-ratio (PSNR), multi-scale structural similarity (MS-SSIM) [39], and Freshet Inception Distance (FID) [40].
To quantitatively evaluate the model performance in means of reconstruction we use MAE: where R P ET and S P ET represents the real PET images and the synthetic PET images, respectively. And i is the index of image pixels. In addition, PSNR has been also widely used to measure the quality of synthetic images: where M SE is the mean-squared error between the real PET images and the synthetic PET images. One of the most successful evaluation metrics about image similarity is MS-SSIM. It attempts to extract the multi-scale structural information from the image, where the higher MS-SSIM represents the better similarity between the real PET and synthetic PET. Let x and y represents two image patches extracted from the same spatial location from the synthetic PET and real PET, respectively), and let µ x , σ x and σ xy be the mean of x, the variance of x, and the covariance of x and y, respectively. For pixel i, SSIM is defined as x + σ 2 y + C 2 = l(i) · cs(i) (11) where C 1 and C 2 are the small constants avoiding zero denominator errors. However, the SSIM metric is a single-scale approach. MS-SSIM is proposed to measure the multiscale structural similarity.

MS-SSIM
cs βj j (i) (12) where l M and cs j are the terms we defined in SSIM at scale M and j, respectively. α and β j are the constants used to adjust the relative importance of different components, according to [39]. The Frchet Inception Distance (FID) is a special GAN metric to measures the quality and diversity of the synthetic images using the pre-trained Inception network [Rethinking the inception architecture for computer vision], where the lower FID the better performance.
where µ r and µ g are the empirical means of real gaussian variables and synthetic gaussian variables, which are modeled by the pre-trained Inception networks. Likewise, C r and C g are the empirical covariance of real gaussian variables and synthetic gaussian variables, respectively.

B. Comparison with 2D-based BMGAN architectures
To appraise the effectiveness of the 3D model, we compared it with the 2D variant of proposed model using the same network architectures in this section. The 2D BMGAN was trained with the slices of the axial, coronal, and sagittal planes. And it followed the same settings exploited in the proposed 3D model. It was noted that the 2D models takes the slices of one plane as input while 3D model uses the whole image as input. To qualitatively compare the results, the synthetic PET samples from the 2D BMGAN or the 3D BMGAN are shown in Fig. 3.
For illustration, we show the synthetic results of three subject samples between the proposed 3D method and the 2D variant (see Fig. 3). And the real PET images are also provided. As can be seen, compared to 2D variant, the proposed 3D BMGAN synthesized better visually pleasing results that have more similar appearance to the ground truth PET image in all three planes. To prove this more clearly, the difference maps between the synthetic images and the real PET are shown in Fig. 4. The pixel value in the difference maps is the absolute intensities difference between the synthetic images and real PET images. The pseudo-color processing is employed to emphasize the difference. Looking at the error maps in Fig. 4, the result of 2D variant has larger high-difference regions compared to the result of 3D proposed model. This may be because there is a problem of discontinuous estimation across slices in 2D synthetic model. By considering the 3D volumetric patches rather than slices by slices, the proposed 3D BMGAN could better synthesize PET images by taking volume structural information into account.
The averaged quantitative results achieved by the 2D and 3D models are presented in Table 1, in terms of four different evaluation metrics. We found that 3D BMGAN has higher PSNR and SSIM, lower MAE and FID than 2D variant, which   means it outperform the 2D variant in both image quality and diversity. Both qualitative and quantitative results demonstrate the benefits of employing the 3D convolutional operations over the commonly used 2D operations.

C. Contribution of the Adversarial training strategy
To study the efficacy of the adversarial training mechanism in the proposed model, we specially conduct comparison experiments between the proposed 3D BMGAN and the simplified model that remove the discriminator network (i.e., just the 3D DenseU-Net generator and the encoder network).
The axial slices of two synthetic 3D PET samples are visualized in Fig. 5, where the synthetic images, real PET images and the difference maps are provided at once. We can clearly see that the synthetic images without the adversarial training strategy are easily over-smoothed and fuzzy compared to with the ground truth PET images. We speculate that this is because the simplified model without the adversarial training strategy synthesizes the PET images in a voxel-wise manner, which ignores the global structure and the context information. In this voxel-wise estimation manner, each voxel of the synthetic images is determined by the averaged value of potential overlapping regions. With the adversarial training strategy, the results are clearer and with better visual quality.  Table 2 show the quantitative results. We can see that adversarial training strategy brings obvious improvement in all metrics. For example, compared with the simplified model removed the discriminator network, the averaged PSNR of proposed method increases approximately 2.34. The experimental results demonstrated that the adversarial training mechanism of GAN is useful for improving the quality of synthetic medical images, indicating the essentials of this mechanism in our 3D BMGAN.

D. Contribution of the DenseU-Net generator architecture
As mentioned before, we design the DenseU-Net architecture, a variant of U-Net that combing the dense connections as the generator. To justify the effectiveness of the DenseU-Net, the performance of three generator architectures were compared in this section, i.e. original U-Net, ResU-Net and DenseU-Net. The detailed quantitative comparison in terms of MAE, PSNR, MS-SSIM and FID for the three architectures is given in Fig. 6.
From Fig. 6, we can see that, just using the original U-Net as the generator architecture obtains the lowest performance. The main reason is that the learnable parameters update difficultly due to the vanishing gradient issues. Whats more, the model is easier to slide into the local optimum. Therefore, just using the U-Net architecture including skip connection leads to unsatisfactory results. Compared to using the original U-Net, employ the ResU-Net including residual paths achieves a better quantitative result. This is due to the fact that the synthetic model might be difficult to converge to the optimal value during the training phase because of the vanishing gradients or the cost of GPU memory. In such cases, the residual learning can boost the gradient flow across different layers and make the network much easier to converge to the optimal parameters, which help synthesize the PET images from the MR images. Nevertheless, it can be seen that using the DenseU-Net architecture as the generator achieves the best performance in all metrics. The main reason triggering this improvement is that the feature reuse of dense connection is more efficient in the parameter and ensures the maximum information flow. It can be seen as an extension of residual learning. The inconsistency of the feature distributions between the MR and PET images could be mitigated by the dense block and skip connection. The results suggesting that the DenseU-Net generator architecture can further improve the image quality of the synthetic PET images.

E. Contribution of the extensive objective functions
As mentioned in the Method Section, the extensive objective function is employed to optimize the network, including the adversarial loss, the KL-divergence constraint, the pixel-wise reconstruction loss (L 1 loss) and the perceptual loss. To study the contribution of the extensive objective functions for MR-to-PET synthesis, we use 1) adversarial loss and KL-divergence constraint, 2) adversarial loss, KL-divergence constraint, and L 1 loss, 3) adversarial loss, KL-divergence constraint, L 1 loss, and perceptual loss (ours) for training the proposed model, respectively. Results are shown in Table 3. Note that we observed that there is a mode collapsed problem when using the adversarial loss of vanilla GAN to train the model. Therefore, the adversarial loss of LSGAN was used in the proposed model to slightly address the problem. For more information about the comparison of adversarial loss between LSGAN and vanilla GAN, please refer to the [30].
In the trained model without the reconstruction loss and perceptual loss, the results seem tolerable but the quality of synthetic images is not very good. The main problems are that the adversarial loss and KL-divergence constraint focus on the global information and fails to pull the distribution between the real PET and synthetic PET closely enough in the pixel level.
The first kind of extension losses is the pixel-wise reconstruction loss (L 1 loss) added to the adversarial loss and KLdivergence constraint. From the Table 3, adding the pixel-wise loss significantly improve the quality of synthetic images in all four metrics. It is helpful to capture the overall structure in relation to the PET context. Then, the second kind of extension losses is the perceptual loss. Compared to the previous result, adding the perceptual loss brings a modest improvement. And the improvement on MS-SSIM is relatively significant. This is because the perceptual losses help the model to keep more detailed feature representations of the synthetic PET images.
Above all, with the extensive objective functions including the adversarial loss, the KL-divergence constraint, the pixelwise reconstruction loss and the perceptual loss, our model is able to performs stable and robust MR-to-PET synthesis on the current applications.

F. Comparison with the competitive synthetic methods
In this section, we compare our method with the existing medical imaging cross-modality synthesis models to prove its effectiveness. The original FCN and GAN was used as the baseline models. A typical 3D FCN is employed to perform this task, while the pooling operation is not used due to it potentially leads to the loss of information and resolution. Moreover, the compared GAN consists of the FCN generator and the classic discriminator. To further enhance the reliability, some comparisons with the models which have done specifically for MR-to-PET synthesis were made, such U-Net [36] and Cycle-consistent GAN (CycleGAN) [37]. Finally, we compared the synthetic results of proposed model with that of pGAN [16], which was the state-of-the-art model in cross-modality synthesis models. It was noted that we reimplemented these models according to their original papers, due to their codes were unavailable. The topology of models was retained, while some necessary changes were made to adapt to current task. Fig. 7 is the illustrative figure showing the synthetic images from each model in axial planes. In Fig. 7, we show both the synthetic results and real PET images from four different samples, which have diverse brain structure. To clearly show the improvement of synthetic PET images, the difference maps of an axial image from each model were calculated (see Fig. 8).
As can be seen in Fig. 7 and Fig. 8, there is a significant appearance difference between the output of the FCN and the ground truth images. This may be because the network structure restricts its generative capability under the current limited training dataset. Meanwhile, the synthetic results of U-Net roughly restore the skeleton of real PET images while with blurry estimations in structural details. We speculate that the advantage of U-Net compared to FCN is that it can be properly trained when the training samples is relatively limited, which is quite common in medical image analysis tasks. And the U-Net network architecture with the skip connection may be more suitable for the medical image analysis tasks because of the better preservation of context information. To proceed, compared with the FCN without adversarial training, the synthetic results of GAN using the FCN as the generator are closer to the real PET images, which further proves the advantages of adversarial training strategy. However, it should be noted that the quality of its synthetic images is still very poor, with many artifacts.
As the state-of-the-art synthetic models, the synthetic results of CycleGAN and pGAN obviously achieved better visual quality than the above three models. But comparing to the real PET images, we can observe that their synthetic results are coarser within the region close to the boundary, while in the results of our proposed BMGAN, these regions are sharper and clearer. Whats more, the synthetic images by our method appreciably improve the detail and diversity, which are closest to the real images between the multiple subjects with different brain structure. In contrast, the synthetic images of CycleGAN or pGAN have worse appearances in the case that the ground-truth images have large structural changes. And some wrong results may be produced. The potential reason is that the structural and pixel distribution between the brain PET from different subjects may have a huge difference. In the current limited training dataset, CycleGAN or pGAN tend to synthesize the plausible but similar results, resulting in poor diversity of generated images.
The superiority of the proposed BMGAN in this scenario are clearly shown. In the proposed BMGAN, the latent space are embedded into some priori semantic information about the PET images. In this way, even if the structures of groundtruth images are diverse, the latent vectors with semantic prior knowledge can provide more auxiliary information for the GAN model to help it synthesize realistic and diverse PET images. From the figure, we can conclude that the proposed BMGAN has satisfying performance on synthesizing PET images from MR images.
The quantitative comparison in terms of MAE, PSNR, MS-SSIM and FID metrics also provided in Fig. 9. As observed, our proposed method achieves the best performance, which considerably outperforms other competitive methods. The lowest MAE and the highest PSNR represent the synthetic results by our method have the best quality, while the highest MS-SSIM and the lowest FID represent the synthetic results by our method have the best diversity. Both qualitative and quantitative results suggest that, compared with the competitive synthetic models, the proposed method can generate plausible and diverse PET images with better synthetic performance.

G. Classification experiment on the synthetic images
Nevertheless, the used evaluation metrics for synthetic images were not directly relevant to the diagnostic quality. To study the effectiveness of synthetic PET images in disease diagnosis quantitatively, we further calculated the classification accuracy of the synthetic PET images on a binary classification task (i.e. AD vs. Normal). VGG-16 network [32], a classic imaging classification networks in computer vision, is employed as the gold-standard classifier in the current study. For a fair comparison, we first pre-trained the VGG-16 using the real PET images. Then, the synthetic PET images from different methods were fed to the pre-trained VGG-16 to make a classification, respectively. The area under the ROC curve (AUC) was used as the evaluation metric to solve the classimbalanced problem between different classes. And the results of classification experiment are shown in Fig. 10. It can be observed that the AUC of the proposed BMGAN outperformed the other compared methods. According to the above experimental results, it can be concluded that the proposed BMGAN is capable of generating a substantial amount of realistic and diverse PET images with the detailed attributes of brains. And it also proved that the proposed BMGAN can be used as an effective data augmentation method.

V. CONCLUSION
In this work, we proposed a novel 3D BMGAN for synthesizing the brain PET images from the brain MR images. As an attempt to bridge the gap between the network generative capability and real medical images, the bidirectional mapping mechanism was introduced to encourage the generator to synthesize the perceptually realistic PET images while preserving the diverse details of brain structures in different subjects. Also, the 3D DenseU-Net generator architecture and the extensive objective functions were used to further improve the visual quality of synthetic PET images. The performance of our proposed method is evaluated in a subset of ADNI database. The experiment demonstrates that the diverse details of brain structures in different subjects can be faithfully preserved by our proposed method. Moreover, both qualitatively and quantitatively experimental results suggest that our proposed method can effectively synthesize PET images from MR images, outperforming the baseline deep learning methods and the existing state-of-the-art cross-modality methods. Please note that, the proposed method can be used in wider medical imaging synthesis applications where a cross-modality mapping from source domain to target domain is needed. In the future, we will further investigate the potential of incorporating the multi-modality information for better synthesis.


extfieldsofstudy:
Physics
Computer Science

text:
Asynchronous Communication for Improved Data Transport in The Network

Currently there are two types of interaction in web systems mediated by synchronous and asynchronous communication, each of them with its programming and interaction paradigms. Although intuitively we can say that asynchronous communication is more efficient in data transport, there is not much research on this topic. We present a comparison of these two types of communication by requesting the same data from a server and receiving a client. The requests are through the traditional web forms and an experimental semantic interface, which only needs the data in plain text as a response from the server. The results found are of interest for the problem of excessive traffic on the network. This research can open new lines of investigation or generate new paradigms and programming models to improve communication between client and server.

Introduction
The creation of the Internet has been a great revolution in the context of communication in humanity, the evolution of communication on the internet has generated the creation of advanced web systems, which have grown in complexity over time [1]. Nowadays, the creation of a web system needs the interaction and communication between different types of languages and software to obtain the required functionality by final users [2]. The programming of a web system requires tag-based languages such as: HTML, XHTML, XML, HTML5, styles sheet such as CSS, client Script languages such as: JavaScript, Visual Basic Script, Action Script, etc. [3]. Server-side script languages such as: PHP, JSP, ASP, etc., Database Management System (DBMS) such as: PostgreSQL, MySQL, etc. [4]. And web server software such as: Apache, Tomcat, Internet Information System (ISS), etc. [5]. In this way, the complexity of the programming increases dramatically with the complexity of the system because there are many actors that need to be synchronized to work together [6].
A web system has two types of communication to interact in the client-server model: synchronous and asynchronous communication. According to the website Internet Live Stats, currently there are 2 billion websites in the world, where the vast majority of them use synchronous communication as a means for interaction in web systems. This is mainly because web forms are the basic and common way to access databases [7]. Development platforms and programmers continue to use web forms as the main part in the interaction of web systems based on synchronous communication and not taking advantage of the potential of asynchronous communication. This research focuses on showing the differences in network traffic when using these two types of communication.  interaction between client and server side, we can see that the server does much of the work. When the user interacts through web forms, it must wait for the answer to continue interacting with the system.

Interaction process with web forms
The user interacts with the web form.
Client side waits for client writing on the web form. The most important characteristics to highlight in this type of communication are: • The interaction requires two web pages, the first containing the form to send the data and the second which is obtained from the server with the response. • The server needs to build a complete page every time it responds to a request.
• The server sends complete pages, which constantly transports repeated data of the page structure in each response. On the other hand, to test the asynchronous communication, an interface based on semantic queries was created; this interface uses a visual interaction where the elements contained in it contain listening events that trigger queries when the event is pressed on an element of the interface. The client side manages this whole process and sends data to the server to build a query and the server returns the data in plain text. When data is returned the client side controls the process of incorporating it into the interface dynamically by creating tags on the fly for each piece of data, adding and modifying the internal structure of the document. Figure 2 shows this process by means of an interaction behavior: Figure 2. The figure shows how the asynchronous interaction modifies the workload towards the client side. The server does not build a complete page, it only retrieves data in plain text and sends it to the client side, which is responsible for formatting and creating the structure to incorporate it into the structure of the html document dynamically.
The most important characteristics to highlight in this type of communication are: • The initial page is only sent the first time, and then it is constantly updated given the new data incorporated in each request. • The largest workload is concentrated on the client side.
• Event capture, request submission, information retrieval, dynamic tag generation, HTML structure update, the client side controls everything. • The client side only receives plain text minimizing data transport. Now, with these two types of processes we are going to generate the same query and compare the results obtained. In this research, only the response data sent by the server to the client are of interest, since we are interested in knowing the amount of data transport traffic in each response.

Results
Now in this section, we show the results obtained when requests are made with traditional web forms, versus an interface with an experimental semantic interface based on asynchronous communication. The request is the same each time, seven different requests are made with which it is possible to obtain the continuous pattern for a number n of times, and the first results are shown in table 1.

Interaction process with semantic queries
The user interacts with the interface Client side captures events from interface Client side identifies the event and tag that generated it Client side builds the data request to the server Client side sends an asynchronous request to the server Server side verifies data and builds query Server side gets the data Server side sends it in plain text to the client side. Client side receives the data and separates the data Client side creates elements according to the structure The client side formats the data and add it into corresponding tags Client side assigns identifiers and listening to events to tags Client side assigns style and incorporate tags into the interface The Interface is waiting for another event  Table 1 shows a first request that contemplates the construction of the initial structure of the page without data, these data do not include images, CSS or Scripts because these vary depending on the design and the main interest in this research is the transport of data in each request. The first request the server sends the complete page so it sends the same amount of bytes, but in the following requests, we can see the difference between the two types of communication. Once we have obtained this data, we can observe in a graph the difference in behavior as shown in figure 3. Figure 3 shows three graphs that show in red the data transport in synchronous communication and in cyan the data transported in asynchronous communication. Figure 3 (a) shows how when communication is done synchronously the server needs to transport the whole structure again and again of the same page, but with different data. On the other hand, in asynchronous communication, a minimum of data transport can be observed, which is very close to the minimum possible. Figure 3 (b) shows in another visual representation how the synchronous graph tends to be exponential while the asynchronous one tends to be linear. Finally, figure 3 (c) shows the totality of transported data where the first one observes 41292 bytes (41K) in total, while the asynchronous total is 1757 bytes (1 K), this shows a difference of 39535 bytes between one and another to show in an interface the same information. This means that if a page is consulted thousands, or millions of times it continuously transports unnecessary repeated and redundant data of the structure of each page. These graphs show the result for a simple basic consultation, but if we look further imagine for hundreds of millions of pages it is easy to see that due to this situation there is unnecessary and useless traffic on the network. Although the asynchronous recovery of plain text data allows a lot of efficiency, this can be done in different ways. Data recovery can be done through XML, JSON format or between HTML, XHTML, HTML5 tags. Each format must comply with certain specifications, commonly the data must be enclosed in tags according to the specified recovery format.    Table 2 shows the types of formats used to send from the server to the client, in the first column we see the data to be sent in plain text, in this case only four characters need to be sent. But the server does not send the data this way, because the client side cannot distinguish when several data are sent. Therefore, each data needs a separator or tags that distinguish one from another. The second column shows that if the data is sent in XML format it needs 25 characters. Although this number varies depending on the name assigned in the semantic logic that the programmer assigns to the tags. The third column shows HTML5 format, and this number also varies according to the specific tag used, depending on the type of data sent. The fourth column shows the JSON format commonly used by many programmers and it also varies according to the name assigned to the label. The fifth column PlainTS (Plain Text-Symbol) which is the format used for this study, it is composed only of the data and a separation character, thus the number of characters carried by data = data + 1 character. Figure 4 shows how depending on the format used, the number of characters accompanying the data may vary. However, the format used by us PlainTS also offers visible advantages in terms of data transport. This format is very efficient since it is only the data and a separation character. Occupying this format involves generating client script code to perform data separation and can be embedded in the HTML document. The arrival format Plain TS is as follows:: data1¬data2¬data3¬...datan.  For any format such as XML, JSON etc, there are complete libraries which are often unnecessarily large and the programmer depends on their understanding and use of some features. Also, these libraries are composed of thousands and thousands of lines of code completely unnecessary in the vast majority of cases, which brings us back to the problem of transporting completely useless code and causing unnece ssary traffic. Due to this situation we found that having a plain text with only one symbol to differentiate between them, helps greatly to avoid transporting libraries with thousands of lines of unusable code. The PlaintTS format creates a tagless atomic data, and maximizes the efficiency of data transport, because it only moves the necessary data by greatly reducing unnecessary data traffic.

PlainTS
This means that there must be more programming on the client side, in order to maintain control of the requests and control of the data as it is received. As the data arrives without labels these need to be generated in a dynamic and automatic way, which the use of DOM helps us to achieve this objective, generating the tags on the client side also affects data transport, as can be seen in figure 5 (a) .With all these results we can get an overview of how programming in a synchronous way generates some inefficiency in the communication between client and server. The figure 5(a) shows the amount of bytes generated dynamically and automatically from the client side, which implies that they were not transported. So, from 6385 bytes needed, only 1757 were transported, which multiplied by millions of queries if everyone worked in this way would imply a decrease in network traffic very large. Figure 5 (b-c) shows very interesting data that allows to generate discussion about the need to evolve the current programming and generate new paradigms based on asynchronous communication. In the figure 5 (b) we can visualize in the first column the bytes transported the first time to transport the initial page with 4144 bytes beginning. The second column shows the number of bytes of the data transported from the queries in total 1757 bytes. The third column shows the total number of bytes generated by the labels generated automatically and dynamically on the client side 4618 bytes. The fourth column shows the total number of bytes contained in the final page at the end of the interaction 10519 bytes with the data included. Here we can see an interesting fact, of the total 100% in bytes of the final page, only 56% was transported from the server to the client and 44% was generated dynamically and automatically on the client side, this means a shared work between client and server. Because, the client side worked that 44% and reduced the server load by processing less data and sending less amount of data, which impacts the overall performance and in the network. The fifth column shows the total data transported in an asynchronous way and represents the 56% mentioned of the total data in the final page. Figure 5 (c) shows the comparison of the above graph with the total data carried by the synchronous method. Now we can observe an 86% reduction of data transport of the asynchronous vs.  Figure 5. The figure shows the relationship between all parties involved as: data transported in the initial page, asynchronous data transported, dynamically generated tags, and the comparison between data transported between the interaction models. The inverse relationship shows that 700% more bytes in the use of the traditional method of web form. All these results allow to open the discussion about the future need of new programming techniques, which allow efficiency in data transfer between client and server. We occupied an experimental semantic interface with asynchronous communication for exploration purposes in scientific databases, and created traditional web forms for the same queries to compare data transport showing the same results, but using different programming and interaction techniques. we think that these results can open new lines of research to standardize semantic queries, and create a standard with new and innovative techniques to improve the processing of the client side, and minimize the load on the server and this substantially increase its performance and improving the traffic of the network.

Conclusions
We present a study on how the type of communication and programming has an impact on data traffic from the server to the client. New web development paradigms and new asynchronous communication models are needed. In addition, new programming languages to take advantage of this ability to provide maximum performance of both web servers and the network. This topic can open the possibility to the generation of new open, free, generic, and multilingual development platforms can be a route for much more powerful and faster applications than the current ones. Research on new models that allow independence of programming language, software or operating system to connect multiple devices locally without saturating the network is a good possibility.