extfieldsofstudy:
Computer Science
Engineering

text:
Fast Computational Ghost Imaging using Unpaired Deep Learning and a Constrained Generative Adversarial Network

The unpaired training can be the only option available for fast deep learning-based ghost imaging, where obtaining a high signal-to-noise ratio (SNR) image copy of each low SNR ghost image could be practically time-consuming and challenging. This paper explores the capabilities of deep learning to leverage computational ghost imaging when there is a lack of paired training images. The deep learning approach proposed here enables fast ghost imaging through reconstruction of high SNR images from faint and hastily shot ghost images using a constrained Wasserstein generative adversarial network. In the proposed approach, the objective function is regularized to enforce the generation of faithful and relevant high SNR images to the ghost copies. This regularization measures the distance between reconstructed images and the faint ghost images in a low-noise manifold generated by a shadow network. The performance of the constrained network is shown to be particularly important for ghost images with low SNR. The proposed pipeline is able to reconstruct high-quality images from the ghost images with SNR values not necessarily equal to the SNR of the training set.

1.Introduction
Computational imaging has been highly impacted by the advances in deep learning-based algorithms throughout recent years [1][2][3][4][5][6][7][8]. Particularly, ghost imaging (GI) [9][10] which is an exotic imaging technique can be benefited widely from the potentials offered by deep learning [11][12][13][14][15]. GI can replace the conventional imaging techniques in situations where the object is difficult to reach, or the environment is noisy or harsh for setting-up complex optical systems [16,17]. GI was first introduced within the realm of quantum optics through photon entanglement [18,19] and it has been later verified for classical pseudo-thermal light sources [9][10]20]. Interestingly, in GI an object is imaged by collecting the light using only a single pixel, "bucket" detector which does not have any resolution. A level of image reconstruction is possible by taking the correlation pattern of the beam scattered from an object and captured by the bucket detector with a reference beam, never seen by the object but spatially correlated with the beam illuminating it. The reference beam pattern at the correlator plate does not need to be detected by a highresolution detector and there is no need for availability of such beam, since it can be precomputed numerically as in the case for computational ghost imaging (CGI) [11,14,16,17,21,22].
In practice, to obtain high-resolution images via GI or CGI, it is important to illuminate the object by the pseudo-thermal light for a long period of time to capture enough correlation patterns [12][13][14][15]23]. Long exposure time leads to generation of images with sufficient signal-to-noise (SNR) values [23][24][25]. The number of measurements for high SNR ghost imaging should be much larger than the number of pixels required to resolve an object [11][12][13]. Aside from the fact that imaging time becomes quite long, this is particularly not ideal for situations involving moving objects or when real time imaging is required.
On the other hand, a not sufficiently long exposure time results into faint ghost images of the object. It has become possible to reconstruct images with better quality from the faint ghost images by employing algorithms such as compressed sensing and Gerchberg-Saxton [26][27][28][29][30] or using deep learning-based ghost imaging (GIDL) as an inverse problem solver [11][12][13][14][15]. The latter method has shown to be superior to the former in terms of higher reconstruction quality for shorter exposure time [12][13]. In other words, GIDL has shown to be able to recover images of an object with a number of measurements lower than the number of pixels needed to resolve the image.
While GIDL can offer an end-to-end and generalized technique through which a variety of images can be reconstructed, still it heavily counts on the availability of a large, paired set of the data which is composed of faint ghost images and their corresponding ground truth copies [11][12][13][14][15]. As discussed earlier, the exploiting of such data is time consuming and cumbersome if not impractical. Furthermore, the requirement for a set of paired images can be particularly difficult in most practical situations where the high SNR images of the object cannot be acquired simultaneously along with the faint ghost images. One practical example of such situations is as follows. The pseudo-thermal statistics of the illuminating light beam can be emulated by modulating of the light wavefront using a spatial light modulator (SLM) [13]. To obtain high SNR ghost images given a limited acquisition time, the SLM pattern should change fast to provide enough number of ensemble random noise patterns. This requires a high-speed SLM which simply may not be available. On the other hand, the high SNR images cannot be obtained through numerical simulations since it may cause spurious artifacts.
Unpaired approaches have been used in the context of unsupervised learning by employing generative adversarial networks (GANs) for translation between different images [30][31][32] or low-light image enhancement [33]. Inspired by this capability, we propose an end-to-end learning pipeline based on Wasserstein GAN (WGAN) [34,35] which generates high SNR images from low SNR, hastily shot ghost copies without paired supervision. We used WGAN since it has shown to provide smoother objective function through working with a critic network in place of the discriminator network [34,35].
In our method, a maximum posteriori estimation approach [36], which favors images with higher probability conditioned on the faint ghost images is adopted. This results in a regularization term in the generator cost function which quantifies the similarity between the generated images and the faint ghost images. Therefore, the generator has to play a game with the constrain of generating images as similar as possible to its input. We discuss that the effect of this regularized term can be diminished when the ghost images are overly noisy as a result of very short exposure time. Hence, this term cannot dictate an effective similarity constrain between a faint ghost image and its corresponding generated image. To overcome this difficulty, we propose to use a shadow generator network which follows the changes of the original generator network. The shadow network transfers the ghost image, and its reconstructed copy produced by the generator to a low-noise manifold. The original generator cost function is then further penalized for generated images which have large distance with the ghost images in the new manifold. Using this proposed architect, ghost images with SNR coefficients between 0.25-1 are retrieved without the access to the exact paired high-quality ground-truth images. Moreover, the pipeline developed here, is shown to be able to reconstruct high-quality images from the ghost images with SNR values different from the SNR of the training set.

Computational Ghost Imaging
The concept of classical GI is based on the correlation of a pseudo-thermal light beam scattered from an object and captured by a bucket detector with a reference beam [9][10]. Based on this idea a source of a pseudo-thermal light beam is split into two twin waves with a high degree of spatial correlation as shown in Fig.1 (a). One wave is aimed at an object and then captured by a bucket detector while the other part (reference beam) is detected by a pinhole detector [9][10]. Then, the correlation of the two signals detected by the pinhole and bucket detectors is taken to reveal the information about the object. Later it has been proved that the detection of the reference light by the pinhole detector is not necessary as it can be numerically computed, leading to the emergence of CGI [21,22]. The concept of CGI is plotted in Fig. 1(b). In practice, the pseudo-thermal light is emulated by modulating the beam of a laser using an SLM. The SLM pattern is varied over time and at each time increment Dt, one random pattern is printed on the SLM. A total number of M patterns can be printed on the SLM for a given amount of time. More accurate images of the object can be obtained after allowing a long enough interaction time or equivalently a large M value. As mentioned previously, in CGI, the time statistics of the pseudo-thermal light source is pre-computed and can be modeled as the illuminations with a number of random and independent speckle intensity patterns [11,14,15]. The bucket detector integrates the modulated images of the object with the random speckle patterns. The bucket detector outputs the following variable: in which x and y are the transverse coordinate of the object plane, ( , ) is a 2D object and ! ( , ) is the random speckle pattern at time mDt. The ghost images are reconstructed by taking the correlation of Bm and the speckle intensity patterns, that is: where <.> denotes the ensemble average. The SNR of the images is directly related to the exposure time or the number of speckle patterns [25,26]. The SNR is proportional to the SNR coefficient b, which is equal to M/N in which M is the number of random speckle patterns used for the illumination of object and N is the number of speckles of each random intensity pattern [13,25].
Usually, the value of N is fixed in GCI as speckle patterns are precomputed based on the setup specifications. A larger b value means a larger SNR and a higher-quality ghost image. On the other hand, a larger b value implies a larger acquisition time which may be practically difficult to achieve. In practice, it is required to reprogram the SLMs with a large number of speckle patterns during a limited acquisition time. This may further complicate the optical setup.
In order to decrease the acquisition time or meet the restriction imposed by the optical system, the capabilities of deep learning architectures have been exploited to recover high SNR images of the object from the ghost images acquired over a low exposure time. In this case, deep learning enables the ghost imaging system to enjoy a much shorter acquisition time.

Wasserstein GAN for unpaired computational ghost imaging
GANs are employed for unpaired image-to-image translation [30,31]. However, for our application the GAN's ability to generalize the knowledge learnt from a given dataset can come at a price of generated images being irrelevant to its original high SNR version. This problem may overcome by using the concept of cycle-consistency which uses two reciprocal GANs learning the forward and inverse functions [32]. Cycle-consistent GANs are time-consuming to be trained as a result of adding a second trainable network. To avoid the cycle-consistent approach, a regularization term is added to the objective function. This term measures the distance between the generated images and the faint ghost images. In addition, to exploit the functionality of the GAN networks with a more stable and smoother objective function, a refined version of GAN, namely Wasserstein GAN is implemented here. The Mathematical foundation of the developed pipeline is as follows.
Let x ~ PX(x) and y ~ PY(y) denote samples drawn from the original and low SNR ghost images respectively. The problem of reconstructing high SNR images from the low SNR images can be transformed to maximizing a posteriori probability "|$ ( | ). This can be further converted to finding a high SNR image x that maximizes the following: in which l1 is a hyperparameter. Assuming a function G that maps the low SNR images to high SNR images, the above optimization problem becomes The above equation can be transformed to minimizing a regularized Kullback-Leibler (KL) distance [37] between the probability distributions of generator function, PG, and that of high SNR original images, PX, as: One can show that the solution to the above optimization problem is identical to the solution of a zero-sum game with the following goal function [34]: in which JS(D,G) is the Jensen-Shannon distance [37] between the discriminator, D, and generator functions. It should be noted that, aside from the last regularization term, the above equation represents the well-known vanilla GAN objective function in which the two networks of generator and discriminator act in opposite directions to reach to a Nash equilibrium [38]. The problem with vanilla GAN is that for a sub-optimal generator there is a possibility that the gradient of the objective function may shrink, and the generator stops effective learning. Also, the JS divergence in eq. (6) may not be continuous which adds to the difficulties in training step. To remedy such difficulties in training and end up with a GAN architecture that offers a smoother objective function, the WGAN [34,35] was introduced which leverages the Wasserstein distance instead of JS distance [39]. In this new architecture, the discriminator network D, now called as the critic, has to belong to a k-Lipchitz family of functions [34,35] and provides a continuous score instead of a bounded function. In this regard, the objective function in eq. (6) becomes in which ℒ denotes family of k-Lipchitz functions. The last term of the above equation is a regularization term that penalizes the objective function for generated images that have a discrepancy with the input ghost images. This term makes the one-path training possible despite the lack of one-to-one relation between the low SNR and high SNR original images. In such, in contrast to cycle consistent GANs, there is no need to train another network that models the inverse problem of generating low SNR images from high SNR images and input the generated images of that network to the original network.

Ground truth images; N pixels
Low SNR ghost images b=M/N

Network Architecture
The concept of fast high SNR ghost imaging using an unpaired set of data and a one-path GAN is plotted in Fig. 2. In the first step, the low SNR ghost images y ~ PY are generated numerically from a subset A of original images x ~ PX as follows: (i) a series of M random intensity speckle patterns representing the SLM masks are multiplied with the original N-pixeled image x ~ PX. (ii) each generated intensity pattern is integrated over all its pixels to model the bucket detector output. (iii) the correlation of the random speckle patterns and the output of the bucket detector will be obtained which results in generation of ghost images with In the second step, the ghost images y with SNR b, are fed to the generator network G which plays a zero-sum game with the critic network D as explained in the previous section. The images at the output of the generator G(y), along with the subset B of original images (mutually exclusive with subset A) are loaded into the critic network to produce a critic score. Through the game, the critic network learns to estimate a scoring function that maximizes the Wasserstein distance between the high SNR generated images and the original set of images. Moreover, the generator's goal on one hand is learning to generate high SNR images resembling the original set of images and on the other hand is to generate images as close as possible to its input low SNR ghost copies due to the regularization term. The mean-square-error (MSE) which measures the similarity between the ghost images at the input of the generator network and the generated images at the output of the generator is subtracted from the Wasserstein distance according to eq. (7). This is to penalize the objective function for the images irrelevant to the input ghost images. This latter term can be particularly important for our application. The reason is a good generator network tends to generate plausible images which could have been randomly drawn from the real image dataset. As a result, the high SNR generated images, although good enough to fool the critic, may not have any relevance to the ghost images fed to the generator. Although the relevancy of the generated images can be preserved to some extent by the regularization MSE term ! (‖ ( ) − ‖ " ); through our tests, it has been observed that for lower M values corresponding to lower SNR, the generator can produce high SNR images not related to its noisy input image. To rectify this, we propose to consider another regularization term which measures the similarity between the generated images and their corresponding ghost images after transferring them into a domain with higher SNR. To do so, a shadow network Gsh, which has a similar structure to the generator network is considered. We consider the following update rule for the shadow network: in which G i sh is the shadow generator at the i th step and is the update rate. Throughout our experiments has been kept to 0.1. The generated images and the ghost images are gated to the shadow network to generate Gsh(G(y)) and Gsh(y) respectively. The distance between Gsh(y) and Gsh(G(y)) is expected to show the relevance of the images at the input and the output of the generator. Figure 3 illustrates how this distance can more appropriately indicate the similarity between the generated and ghost images than the distance between G(y) and y. In fact, when the noise level is strong, the low SNR ghost images are not distinguishable and therefore, they will have an almost similar and small distance with each other. However, since the shadow network will follow the generator network, it can map the low SNR ghost images to a new manifold where images are more distinguishable or are adequately far apart. This is due to the fact that the generator network has learnt to generate high SNR images through the selfsupervised approach of GAN. As shown in Fig. 3, the input domain of shadow network is mapped to a new manifold in which ghost images are separated with larger distances. Therefore, while in the input domain of the shadow network, the ghost images can have similar distances to a particular high SNR image, in the output manifold there is a more unique relation between the distances of the mapped ghost images and their original high SNR copies.

Shadow network output manifold
Shadow Network Since the distance at the output of the shadow network can better reflects the faithfulness of the generated images, a new MSE term is included in the objective function which penalizes the generator for generated images which are not close enough to original images at the output manifold of the shadow network. The complete WGAN objective function is as follows: (9) in which, LWGAN is the WGAN objective function with gradient penalty term [35] and l1 and l2 are hyperparameters. The MSE1 is the regular MSE of the ghost and the generated images or ! (‖ ( ) − ‖ " ) and MSE2 is the l2 distance between the transferred ghost and the transferred generated images at the output of the shadow network or ! (‖ #$ ( ( )) − #$ ( )‖ " ).

Training and results
A set of 70,000 handwritten digits of 28 by 28 pixels from MNIST database is chosen and the relevant ghost images of these digits are numerically produced for different M values of 196, 392 and 784 corresponding to SNR coefficients β, equal to 0.25, 0.5 and 1 respectively. A set of 10,000 images of ghost images is set aside for evaluating the network in the test step. To obtain an unpaired training set of ghost and ground truth images, two mutually exclusive subsets of 30,000 ghost and ground truth images are selected from larger 60,000 sets. In this way, we make sure there is no ghost image of a particular original image present in our training set.     Regularized metric

(a) (b) (c) (d)
The set of 30,000 ghost images are fed to a generator network which has a conventional encoder-decoder [40] architecture as depicted in Fig. 2. The generator is composed of 5 convolutional blocks consisting of convolutional, batch normalization and LeakyReLU layers for the encoder and 8 de-convolutional blocks consisting of de-convolutional, batch normalization and LeakyReLUs for the decoder part. The critic network is made of 4 convolutional blocks including convolutional and LeakyReLU layers and a flatten layer composed of 2048 neurons.
Although the WGAN distance provides a smooth objective function and is related to the evolution of the critic and the generator networks through the game, it does not speak well to the quality and diversity of the generated images. To address this, a classifier network is pre-trained on MNIST handwritten digits to extract the inception score. The inception score has been proved to better reflect the convergence of GANs working on image data as it is attributed to the ability of the generator to produce varieties of images [41]. Our pre-trained classifier yields an inception score of 9.87 for the original MNIST dataset. In the light of this modification, a customized metric, called regularized inception score is introduced which is defined as: in which, IS is the inception score of the generated images. It should be mentioned that MSE1 and MSE2 controls the relevance of the generated images to the input ghost images as discussed earlier. The Adam optimizer with learning rate of 1e-4 and a batch size of 256 is selected. The training process takes around 5 hours on an NVIDIA Tesla P-100 GPU accelerator. Figure 4 shows different metrics of our GAN during the training of the network on an unpaired set of ground truth and ghost images with b = 0.25. Obviously, generator and critic losses progress oppositely which embodies the zero-sum gain between the two networks. The inception score and the regularized metric rise with advancement through the epochs which reflects the generation of digits of different classes while maintaining the maximum resemblance to the input ghost images. Figure 5 shows sample generated images and the summary of different performances for training on multiple sets of ghost images with b = 0.25, 0.5 and 1. The performances are evaluated in terms of the inception score and macro F1-score of the    generated images from ghost copies with b = 0.25, 0.5 and 1. Note that the higher inception score shows the ability of our generator network to produce variety of the classes. On the other hand, the F1-score average quantifies the degree to which these different classes have been correctly generated. As inferred from table in Fig 5(a), the highest inception score and macro F1-score are obtained when the network is evaluated on test images with the same noise level as its training dataset. However, the network is still able to perform well for test images generated from ghost images with different SNR values. Moreover, it is important to investigate the effect of different regularization terms, MSE1 and MSE2 on the generated images. This is depicted in Fig. 6. A set of ghost images with b = 0.5 is chosen for the training set. The network is trained with different values for the regularization coefficients of l1 and l2. The performances of different trainings are assessed by multiple sets of test data having different SNR coefficients. Among different combinations of l1 and l2, the best performance is obtained for l1 = l2 = 10. It can be seen that, when l2 = 0, the inception score decreases moderately while the macro F1-score shows a more drastic decrease compared to the best result. This is because when l2 = 0, the similarity between generated and ghost images is solely measures in high-noise domain. Therefore, for low SNR ghost images with b = 0.5, the generated images do not have high correlation with their ground truth counterparts resulting the macro F1-score to become smaller. However, the inception score can still be relatively high due to the good performance of the GAN network for generation of images belonging to different classes. The ability of the trained GAN to produce variety of images is clear from images generated for l1 = l2 = 0. As expected, such a trained network shows a very poor F1-score of 11% and a good inception score of ~ 9. The effect of shadow network is evident from the performance achieved with regularization coefficients of l1 = 0 and l2 = 20. In this case, both inception and macro F1-scores decrease from the values obtained for l1 = l2 = 10. However, the F1-score is decreased by a noticeably smaller amount in this case compared to the case of l1 = 20 and l2 = 0. This is due the fact that MSE2 which measures the fidelity of the generated images uses a distance metric in a lower-noise manifold.

Original images
In conclusion, we have developed an end-to-end deep learning pipeline based on a constrained WGAN to reconstruct high-SNR images from low SNR ghost images. The proposed constrained WGAN is trained on two mutually exclusive sets of high SNR and low SNR ghost images and therefore the training is done in an unsupervised fashion on unpaired sets of data. To keep the generated images relevant, two regularization terms are included in the objective function which (1) quantifies the similarity between the reconstructed images and the faint ghost images and (2) measures the distance between the reconstructed images, and the faint ghost images in a low-noise image manifold generated by a non-trainable shadow network. High SNR version of ghost images with SNR coefficients of 0.25-1 are reconstructed using the developed pipeline. The individual and mutual effects of the two regularized terms are investigated in terms of the inception score and macro F1-score average of the reconstructed images. It has been found that at SNR coefficients of < 0.5 the effect of the second regularization term is more crucial.


text:
A Better and Faster End-to-End Model for Streaming ASR

End-to-end (E2E) models have shown to outperform state-of-the-art conventional models for streaming speech recognition [1] across many dimensions, including quality (as measured by word error rate (WER)) and endpointer latency [2]. However, the model still tends to delay the predictions towards the end and thus has much higher partial latency compared to a conventional ASR model. To address this issue, we look at encouraging the E2E model to emit words early, through an algorithm called FastEmit [3]. Naturally, improving on latency results in a quality degradation. To address this, we explore replacing the LSTM layers in the encoder of our E2E model with Conformer layers [4], which has shown good improvements for ASR. Secondly, we also explore running a 2nd-pass beam search to improve quality. In order to ensure the 2nd-pass completes quickly, we explore non-causal Conformer layers that feed into the same 1st-pass RNN-T decoder, an algorithm we called Cascaded Encoders. Overall, we find that the Conformer RNN-T with Cascaded Encoders offers a better quality and latency tradeoff for streaming ASR.

INTRODUCTION
Having an end-to-end (E2E) speech recognition model [5][6][7][8][9][10], that matches/surpasses the quality of a conventional model (which has separate acoustic, pronunciation and language model components) across all dimensions of quality and latency, has become a challenging research direction over the past few years.
On the latency side, one such requirement of E2E models is that they have to be streaming. They must emit words to the screen with minimal delay as the user speaks. Models such as Recurrent Neural Network Transducer (RNN-T) [8], Neural Transducer [11,12], Monotonic Attention [13,14], Triggered attention [15] and Scout Network [16] have been explored to address the streaming nature. Another requirement is that the model must stop decoding quickly once the user stops speaking, a problem known as endpointing. Folding the endpointer into the E2E model helped address this issue [2].
On the quality side, we require a model which achieves better word error rate (WER) compared to a conventional model. Using a 1st-pass streaming RNN-T model coupled with a 2nd-pass non-streaming attention-based model helped address quality concerns [17]. In addition, improving the performance of the model to recognize proper nouns was achieved with contextual biasing [18].
Putting many of these above components together, we recently released an on-device model that surpassed a conventional model in terms of both WER and endpointer latency [1]. However, one of the * Equal contribution issues with that model is that partial hypotheses were still delayed to the end compared to a conventional model trained with an alignment.
In this work, we explore ideas to improve the model's partial latency. Speech-to-text alignment can be used to improve the token prediction timing [19,20], which normally causes large quality degradation. We adopted the recently proposed FastEmit [3]. It regularizes the RNN-T lattice to encourage non-blank tokens and surpass the blank token across the entire sequence based on the forwardbackward probabilities. It is found to be effective in reducing models' partial latency with much less quality degradation.
One can expect that any research attempt to improve partial latency results in a slight quality degradation. To address this, we explore improving our core E2E model by replacing the recurrent LSTM layers in the encoder with Conformer layers, which have shown promise in Librispeech [4]. In addition, we also look at improving our 2nd-pass strategy so that we can run a beam search rather than a rescorer. We explored the improved latency and quality ideas on a Voice Search task, which yielded 10% relative WER reduction with 250ms and 170ms reduction in median partial and prefetch latency respectively than [1].

Conformer
We use Conformer [4] to improve the model quality. The model used in [4] is designed for tasks with non-streaming, single-domain and short utterances. In this work, we adapt it for our streaming multi-domain task [1]. We first limit the self-attention, the convolution and the normalization layers to take in only the previous context for streaming applications. We also replaced the full context self-attention with local self-attention which helps generalize to long utterances [21]. The same as [1], we train our systems on a combination of data from different domains. For efficiency, we aggregate utterances with similar lengths before creating batches (i.e. bucketing). Due to the different length distributions across domains, for a particular batch the data may bias towards certain domain. This could lead to biased statistics for the batch normalization used in the original Conformer layer, and then limit the model's generalization capability across domains. We addressed this issue by replacing the batch normalization layer with the group normalization layer [22]. This is found critical for the quality of our multi-domain models.
Additionally, we speedup the training and inference by removing the original relative positional encoding [23]. Instead, we reuse the existing convolution module that aggregates information from neighboring context to implicitly provide relative positional information [23,24]. This is done by simply swapping the order of the convolution module and the multi-head self-attention module, leading to the improved Conformer block depicted in Figure 1.

Two-pass with Cascaded Encoders
A big goal of two-pass models is to make up for the quality degradation in the 1st-pass. To improve our current two-pass approach, which is to run rescoring, we look to run a beam search instead. One of the challenges with this beam search is that we need the model to be robust to decoding short search utterances as well as long caption utterances. Typically attention-based models do very poorly on long-form [21].
To address this, we look at running a 2nd-pass beam search with RNN-T itself, which has shown better robustness to long-form [25]. The model explored, which we call Cascaded Encoders, is shown in Figure 2. To add the non-causal aspect of LAS (which improves quality) [17] into a streaming, causal RNN-T system, we add additional non-causal encoder layers on top of the causal encoder layers. The 1st-pass uses only the causal encoder and the RNN-T decoder. In the second pass, the additional non-causal layers take in both left and right context of the 1st-pass encoder outputs, and again feeds to the same decoder. A single RNN-T decoder is shared between the 1st and 2nd-passes for smaller model size and on-device benefits.

LATENCY IMPROVEMENTS
In this section, we present latency metrics for streaming ASR and techniques to reduce them. Different from the average token latency [20], we adopt metrics that are directly related to streaming speech applications such as Voice Search and Assistant.

Endpointer Latency
In streaming ASR based applications, when the user finishes speaking is a critical signal that triggers the next step fulfillment. For Voice Search/Assistant, it signals the ASR system to finalize the recognition result and initiate either a server request for more information or an action on the client side. Endpointer latency measures the time difference between when the user finishes speaking and when the system predicts the end of the query (EOQ). This is the latency metric we normally referred to in previous work [1,2]. A lower Endpointer latency leads to a lower application latency and faster system responses. We adopt the same RNN-T Endpointer model together with early and late penalties on EOQ during training to reduce Endpointer latency [2]. We report both the median (i.e. 50 th percentile, EP50) and the 90 th percentile (EP90) latency for comparisons.

Prefetch Latency
Prefetching is a common technique used in web-based applications to reduce network latency involved in the application stack [26,27]. Instead of waiting for the EOQ signal, streaming applications normally send early server requests (either retrieving relevant information or necessary resources for next step actions) based on partial/incomplete recognition hypotheses. If a prefetch is based on a partial hypothesis that does not match the finalized hypothesis at the time EOQ is generated, it is discarded as a premature prefetching, increasing the overall computation cost. However, if the partial hypothesis used in a prefetching matches with the final result, the system can immediately execute the following action without waiting for additional server responses. This saves the time used to compute and fetch responses from the server and reduces the network processing latency. Prefetch latency is defined as the time difference between when the first correct prefetch is triggered and when the user finishes speaking [28]. A prefetch is correct only if the partial recognition result it used matches the final recognition result. If no correct prefetch is found, the latency falls back to Endpointer latency. Besides latency, the prefetch rate, which is the number of prefetches for each query, is also an important metric. For comparison, we report both the 50 th and 90 th percentile latency (PF50 and PF90) together with the prefetching rate (PFR).

E2E Prefetching
There are multiple ways to trigger prefetching [28]. It normally relies on observing a fixed interval of silence either from the decoding or an external voice activity detector (VAD). An E2E Prefetching has recently been proposed and yielded a much better prefetch latency and prefetch rate tradeoff [28]. It leverages on the EOQ prediction of the RNN-T Endpointer model for prefetching decisions. Different from endpointing, we want to generate EOQ for every partial result but avoid changing decoding. To achieve that, for each frame before EOQ shows on the top beam, we artificially add it to the end of the current hypothesis to compute its probability given the observations and the best path so far. This allows us to predict how likely the query is completed at each time frame and the corresponding partial result while decoding following frames. If the probability is greater than a predefined threshold, the system declares a prefetch decision.

Partial Latency
Prefetching relies on partial results generated by the ASR model. Systems that output partial hypothesis with less latency are preferred. We hence define partial latency as the time difference between when the first correct partial hypothesis is generated by the model and when the user finishes speaking. Similarly, the correctness of a partial hypothesis refers to whether it matches the finalized recognition hypothesis. Unlike prefetching latency, partial latency is inherent to the ASR model itself and does not rely on additional logic implemented in the application pipeline such as the decision to trigger prefecthing. Partial latency is the lower bound for prefetch latency. Developing techniques that reduce models' partial latency would give more room for reducing the overall application latency and is independent of the actual application pipeline.

Constrained Alignment
One way to improve partial latency is to constrain the model's prediction based on alignments generated from other models [19]. Similar to the RNN-T EP model that constrains the EOU token, constrained alignment penalizes the prediction of each word's beginning and ending tokens if they fall out of a buffer centered around the corresponding ground truth obtained from the alignment. It has been shown to yield more accurate prediction timing information and can be used to improve partial latency.

FastEmit
In conventional RNN-T optimization, emitting a non-blank token and the blank token are treated equally, as long as the log-probability of the target sequence is maximized. However, in streaming ASR systems, the blank token (i.e. output nothing) should be discouraged as it leads to higher latency. A simple and effective sequencelevel emission regularization technique, FastEmit [3], is hence developed. It encourages predicting non-blank tokens and suppresses blank tokens across the entire sequence based on transducer forwardbackward probabilities. It is implemented by adding a regularization hyper parameter λFastEmit to the original Equation 20 in [29] (please refer to [29] for detailed notion definitions): Intuitively, FastEmit applies a "higher learning rate" (λFastEmit 0) to the prediction of non-blank token when back-propagating into the streaming ASR network, while the prediction of blank token remains the same. It has been shown to give significant lower latency while retaining recognition accuracy on different RNN-T models. More importantly, FastEmit does not require any prior alignment information [19,20] and has no additional training or serving cost.

EXPERIMENTAL SETUP
Model Architecture: All models are trained using a 128D log-Mel feature frontend [25]. The features are computed using 32ms windows with a 10ms hop. Features from 4 contiguous frames are stacked to form a 512D input representation, which is further subsampled by a factor of 3. A 16D one-hot domain-id vector is then appended to form a 528D input vector as the inputs to the model. Following [4], we use 512D Conformer layers in the encoder. Causal convolution and left-context attention layers are used for the Conformer layer to strictly restrict the model to use no future inputs. 8-head attention is used in the self-attention layer and the Convolution kernel size used is 15. The encoder consists of 17 Conformer layers. The RNN-T decoder consists of a prediction network with 2 LSTM layers with 2048 units projected down to 640 output units, and a joint network with a single feed-forward layer with 640 units. Two non-causal Conformer layers that take in additional 5.04s right context are used in the Cascaded Encoders on top of the streaming encoder before the decoder. All the E2E models are trained to predict 4,096 word pieces [30]. The decoding is rescored with a 2nd-pass MaxEnt language model [31].
The Conformer RNN-T model has 137M parameters. The two non-causal Conformer layers in the Cascaded Encoders model bring in additional 12M parameters. All models are trained in Tensorflow [32] using the Lingvo [33] toolkit on 8 × 8 Tensor Processing Units (TPU) slices with a global batch size of 4,096 utterances. Models are optimized using synchronized stochastic gradient descent with the Adam optimizer [34]. Datasets: All the E2E models are trained on audio-text pairs the same as [1], which is a small fraction of data compared to the trillion-word text-only data a conventional LM is trained with. To increase vocabulary and diversity of training data, we explore using more data by incorporating multi-domain utterances as described in [25]. These multi-domain utterances span domains of search, farfield, telephony and YouTube. All datasets are anonymized and hand-transcribed; the transcription for YouTube utterances is done in a semi-supervised fashion [35]. In addition to the diverse training sets, multi-condition training (MTR) [36] and random data down-sampling to 8kHz [37] are also used to further increase data diversity. Noisy data is generated at signal-noise-ratio (SNR) from 0 to 30 dB, with an average SNR of 12 dB, and with T60 times ranging from 0 to 900ms, averaging 500ms. Noise segments are sampled from YouTube and daily life noisy environmental recordings. Both 8 kHz and 16 kHz versions of the data are generated, each with equal probability, to make the model robust to varying sample rates. The test set includes around 12K Voice Search utterances with duration less than 5.5s long. They are anonymized and handtranscribed, and are representative of Google's Voice Search traffic.

Quality Improvement with Conformer
First we experimented with Conformer encoders for quality improvement. Table 1 presents the comparison on the Voice Search testset for RNN-T models with different encoders. The RNN-T model with an LSTM encoder serves as our baseline (B1), which has been shown to give a better quality and latency performance than conventional hybrid ASR systems [1]. Limiting the Conformer encoder presented in [4] to use only left-context for streaming applications, system C0 shows significant WER degradation due to the biased batch normalization statistics in our multi-domain task. This can be resolved by randomly sampling utterances (i.e. no bucketing) instead of aggregating utterances with similar lengths into each batch (C0 No bucketing in Table 1) with the cost of large training speed reduction due to the lower computation utilization per batch. By replacing the batch normalization with the group normalization, system C1 addressed the issue with similar WER gains but less training speed regression. Further swapping the order of the convolution module and the self-attention module, system C2 achieves both WER gains and training speedups. Comparing to B1, C2 yields a total of 7% relative WER reduction and 35% speedup. Note for B1 we did not see quality gains with more LSTM encoder layers.

Latency Improvement
In previous work [1,2], we focused on improving RNN-T models' endpointer latency and have shown that E2E models have a better WER and endpointer latency tradeoff than conventional hybrid ASR systems. However, even with that we still observe large total latency in E2E based streaming applications. This is then tracked down to  Table 2).

Reducing Partial Latency with Constrained Alignment
In this experiment, we explored constraining the RNN-T prediction with alignment information following [19]. When applied to B1, the use of constrained alignment achieves 210ms and 230ms latency reduction for PR50 and PR90 respectively with the cost of a quality degradation from 6.0% to 6.9% (B2 vs. B1 in Table 2). Negative latency suggests the model is capable of hypothesizing the whole query even before seeing the end of the speech.

Reducing Partial Latency with FastEmit
We then applied FastEmit to the same baseline system B1. From Table 2, B1 with FastEmit (B3) gives 180ms and 130ms reduction for PR50 and PR90 with much less WER regression (6.0% to 6.2%). Additionally, no alignment is needed during B3 training. To address the quality degradation, we used FastEmit on a better E2E model -C2 from Table 1. Although FastEmit still brings in 0.2% absolute WER regression, the system 'C2+FastEmit' (i.e. C3) has a WER of 5.8% with a -110ms PR50 and 90ms PR90, which is better both in quality and latency than B1.

Reducing Prefetch Latency with E2E Prefetching
Next, we explored the use of E2E prefetch to minimize the gap between partial and prefetch as prefetch latency is the one that is reflected in the total latency. E2E prefetch can only be used in E2E systems and is not applicable to conventional models, where silencebased prefetching is used. From Table 2, adding E2E prefetch to C3 (i.e. C4) brings down the prefetch latency to -50ms for PF50 and 110ms for PF90 with a PFR of 1.86. The WER increased to 6.0% which is still comparable to B1.

Two-Pass with Cascaded Encoders
To further address the quality degradation due to latency related optimization in the 1st-pass, a LAS-based two-pass rescoring model [1,17] was firstly applied, which is referred to as T1 in Table 2. It reduces the WER to 5.3% with comparable endpointer and partial latency metrics; however, the E2E Prefetch is less effective as it only applies to RNN-T EP model. We then train a Cascaded Encoders model (T2 in Table 2). In the causal only mode, 'T2 Single-pass Casual' performs similar to C4; while with the additional non-causal 2nd-pass at EOQ, 'T2 Two-pass' achieves a similar WER as T1 but with much better prefetch latency benefiting from the E2E prefetching. In non-streaming application, we can use 'T2 Single-pass Noncausal' to get an even better WER of 4.8%. The computation cost of T2 is much higher than T1 due to the use of beam-search instead of rescoring. We will further investigate techniques in reducing it in future work. With the two-pass Cascaded Encoders model T2, we can achieve a much better WER and latency (including endpointer, partial and prefetch) comparing to the previous published best E2E system B1.

Comparison to Conventional Model
We also included the performance of a conventional hybrid ASR model (B0 in Table 2) that consists of a low-frame-rate acoustic model emitting context-dependent phonemes [38] (0.1GB), a 764kword pronunciation model (2.2GB), a 1st-pass 5-gram languagemodel (4.9GB). RNN-T models (except for B2) have a better quality and endpointer latency comparing to the conventional system B0, reconfirming the finding in [1]. With FastEmit, we can achieve a better quality with very close partial latency (C3). The -150ms median partial latency (PR50) for B0 mainly comes from the stronger LM trained on large amount of text only data that can potentially hypothesis the full query even before the speech finishes, which E2E models currently lack of. However, even with slightly worse partial latency, E2E systems can still achieve much better prefetch latency with the help of E2E prefetching (C4), which directly affects the total latency of streaming ASRs. With an additional non-casual 2nd-pass (T2 Two-pass), we can further bring down the WER to 5.4% while still having a faster prefetch than B0.

CONCLUSIONS
In this work, we further improve the quality and latency of the RNN-T based E2E models for streaming applications. We replaced the encoder LSTM layers with the recently developed Conformer layers [4] for quality and adopted a simple yet effective latency regularization technique, FastEmit [3], which is generic to any transducer based models. Furthermore, we developed a Cascaded Encoders RNN-T model that shares the same RNN-T decoder for the two passes. It gives us a system that is better and faster than the previous best E2E system [1] and surpassing the conventional model in quality and all latency metrics.


Skin Cancer Detection Using Kernel Fuzzy C-Means and Improved Neural Network Optimization Algorithm

Early diagnosis of malignant skin cancer from images is a significant part of the cancer treatment process. One of the principal purposes of this research is to propose a pipeline methodology for an optimum computer-aided diagnosis of skin cancers. The method contains four main stages. The first stage is to perform a preprocessing based on noise reduction and contrast enhancement. The second stage is to segment the region of interest (ROI). This study uses kernel fuzzy C-means for ROI segmentation. Then, some features from the ROI are extracted, and then, a feature selection is used for selecting the best ones. The selected features are then injected into a support vector machine (SVM) for final identification. One important part of the contribution in this study is to propose a developed version of a new metaheuristic, named neural network optimization algorithm, to optimize both parts of feature selection and SVM classifier. Comparison results of the method with 5 state-of-the-art methods showed the approach's higher superiority toward the others.

Introduction
Cancer is fundamentally an uncontrolled cell division disease. Its growth is usually associated with a group of variations in the action of the cell cycle regularized. e inhibitors of the cell cycle stop cell division while situations are not proper; therefore, so little action of these inhibitors can cause cancer. Cancer cells disregard the signals that must cause cells to prevent dividing. For example, when common cells growing in a culture medium are surrounded on all sides by adjacent cells, they no longer divide (contact inhibition). On the contrary, cancer cells divide into layers in a mass and stack on top of each other, and contact inhibition does not prevent them from growing. One of the most dangerous cancers around the world is skin cancer. e skin, as the first cell layer of the body that is in connection with the outside environment, can suffer from many injuries and diseases. e most common cancer in the United States is skin cancer, which happens in the tissues of the largest section of the skin body. Skin cancer generally occurs on the outer layer of the skin, which may first appear as a swelling, bulge, or different section of the skin [1]. e skin blocks heat, sunlight, sores, and infections; it also warms the body and stores fat water in the body and produces vitamin D. e skin owns 2 principal layers: the superficial layer of the skin (epidermis) and the inner skin (dermis) [2]. e skin's surface layer (epidermis) is mainly composed of flat, scaly cells named Squamous cells. Round cells, called basal cells, are located beneath the superficial cells of the squamous melanocyte in the surface layer [3]. Prolonged exposure to sunlight enhances melanoma skin cancer's risk over a lifetime. One of the most dangerous kinds of cancer is Melanoma, which is caused by damage caused by overexposure to the sun and some other factors. Melanoma can be diagnosed with skin biopsies. Figure 1 shows the statistical information of the lead cancers in quantity and death value in 2019 [4].
If melanoma is diagnosed and treated early, it can be treated, and late diagnosis can lead to the patient's death. Only experienced physicians can diagnose melanoma on time using appropriate tools and histological reports. One of the devices used to diagnose melanoma is dermatoscopy. With this tool, changes in the pigmentation of skin lesions in diseases can be evaluated. With the development of science in recent years, digital dermatoscopy has been replaced by conventional dermatoscopes with the ability to capture and store skin images. With the development of digital dermatoscopes, the development of an algorithm for the diagnosis of melanoma was considered by researchers, and then, other methods were proposed to enhance the accuracy of skin lesion diagnosis. Some of the selected researches are explained in the following.
Astorino et al. [5] suggested an approach for melanoma diagnosis. e method uses multiple instance learning (MIL) approach on some dermoscopy images to categorize between melanomas and common nevi. e method was then analyzed by comparing it with some state-of-the-art classification methods, like support vector machine. Simulation outcomes demonstrated that the suggested approach accomplishes better precision. A leave-one-out authentication is carried out on a dataset and showed the method's higher efficiency.
Mohamed et al. [6] suggested an image processing approach for melanoma diagnosis. e method used a simple pipeline image preprocessing technique including methods like image conversion, noise reduction, and hair removal. Also, the melanoma area is threshold to segment the area, and then, feature extraction was used for taking out the main features of the image using the ABCD rule.
Barros et al. [7] proposed a real-time cancer diagnosis system based on hardware designing by the field-programmable gate arrays (FPGA). A multilayer perceptron artificial neural networks (ANN) is utilized for this purpose. e proposed image processing is used to extract the characteristics of the skin from the images and classify them based on the features and using the ANN classifier. Simulation results of the proposed approach are confirmed by an open-access database. Ultimate results of the suggested approach were then compared with the results of another hardware-based technique using ARM A9 microprocessor to demonstrate the method's proper performance.
Santos and Espitia [8] presented an approach for the diagnosis of uveal melanoma (UM), which is a sort of intraocular cancer. e suggested technique combined iris segmentation methods and designed a method for UM diagnosis based on neural networks and fuzzy logic. Simulation results of the study indicated 96.04% accuracy for the artificial neural networks and 76% correct classification for the fuzzy logic system. Wang [9] proposed an all-inclusive method to provide a proper segmentation technique. e method is applied based on deep convolution networks on the hyperspectral pathology images. e study employed a 3D fully convolutional network, called Hyper-net, to obtain the best performances of segmentation from the hyperspectral pathology images. e loss function was then modified for improving the model sensitivity. Ultimate outcomes demonstrated that the suggested approach has higher performance than the 2D models in the aim of segmentation. e present research suggests a novel optimized pipeline methodology for automatic detection of skin cancer detection. e method includes four main phases. e first phase is preprocessing of the input image to prepare it for the main processing. e first phase here includes two parts: noise reduction and contrast enhancement. e next phase is to segment the region of interest. Afterwards, the features are extracted, and the best ones selected for injecting to a support vector machine (SVM) as the final phase. is investigation employs an improved version of a new metaheuristic, called neural network optimization algorithm, to optimize both parts of feature selection and SVM classifier.
is can improve the accuracy of the system as can be seen in the next sections. e main contribution of the paper has been highlighted as follows:  2 Computational Intelligence and Neuroscience (i) An early diagnosis system for skin cancer from dermoscopy images is proposed (ii) Kernel fuzzy C-means is used to segment the region of interest (ROI) (iii) Some features from the ROI are extracted for the final diagnosis (iv) An improved neural network algorithm is used for optimal feature selection (v) Support vector machine (SVM) is used for final classification (vi) Comparison results are compared with 5 state-ofthe-art methods Figure 2 illustrates the flowchart of the suggested approach.

Noise Reduction.
Ordinary nonpolarized light is used in medical images. Since, according to "Fresnel relations," the reflection of light from the surface of matter and its underlying layers and scattering from uneven and rough surfaces is affected, as well as a function of the state of polarization of light, it is possible to eliminate some annoying and destructive reflected or diffused lights by using a suitable filter to achieve a high signal to noise ratio. erefore, the first step for starting the image analysis of the cancer images is to eliminate, or, at least, reduce, this noise to have a high quality of images for processing. Different methods have been introduced for this purpose. One simple and popular method for noise reduction is Median filtering. However, this filter is nonlinear; it has some significant advantages such as keeping the main edges of the image after filtering. In the median filter, the pixels have been substituted by the median value of their neighbors, i.e., where β signifies the neighborhood of the considered pixel in position (m, n). e present study uses a 3 × 3 mask for the median filter. is selection is based on trial and error and may be different for other databases. us, it should be the point that using a higher mask can improve the probability of edges losing. Figure 3 shows an example of using the median filtering for noise reduction of the input images with considering 0.1 noise density salt and pepper noises.

Contrast Enhancement.
In some situations, due to different reasons, like the sufficient user experience's lack in imaging and the bad quality of measurement devices and sensors utilized in them, the contrast of the images has lessened, such that its intensity gets darkened, or overexposed. To correct this deficiency from the images, we need to implement the image contrast enhancement. erefore, in this study, we can use an image contrast enhancement step on the low-quality images to improve the image contrast to simplify the segmentation step. e present study considers global contrast enhancement based on Lookup Table to reach this purpose [10]. e method of global contrast enhancement is formulated as follows: where Out H signifies the improved output image, and Min H and Max H demonstrate the minimum and the maximum levels of gray values of the original image histogram, orderly. We used an 8-bit lookup table for the method. A simple contrast enhancement on a medical image is shown in Figure 4.

Segmentation Based on Kernel-Based Fuzzy C-Means.
One of the significant issues in medical image processing is the image segmentation of the medical image into its components. Image segmentation describes the success or the final failure of image analysis methods. However, there is no general method for successful segmentation of all medical images, and it still has good research areas due to its wide application. e accuracy of this study is crucial in areas such as medicine, which helps preserve and protect human life. Due to the wide range of applications, it provides image segmentation and the use of methods in various fields. e importance of this part especially in mammographic images is too high, because medical images have a naturally low quality, which makes them a complicated problem for mass segmentation. is research uses an enhanced multistep fuzzy C-means method for performing on the skin cancer images for their mass segmentation. Based on this method, the first step is to generate superpixels using a simple linear iterative clustering (SLIC) algorithm [11]. e SLIC algorithm generates superpixels based on CIE LAB color space in 5D space. Afterwards, texture and color properties are employed to separate the superpixels and finally to find the image elements. en, the color feature from the superpixels has been extracted and has been arranged, such that, for all images in HSV space, three separate histograms for S, H, and V channels have been considered. To reduce time complexity, the histograms have been quantized into 8, 4, and 2 subdistances. Afterward, to find the texture features, the NSCT algorithm has been used. en, the generated data is clustered by a fuzzy kernel, and all superpixels with cluster tags have been placed in one cluster. is study uses kernel fuzzy C-means (KFCM) algorithm employing segmentation. is technique offers a kernel-based version of fuzzy C-means to evaluate the data point's distance from the center of clusters.
e kernel function in this study is achieved as follows: First, the membership function has been evaluated. en, based on a similarity measure of kernel fuzzy, the belonging of each data sample to the clusters has been Computational Intelligence and Neuroscience 3 achieved.
e pseudocode of the algorithm is given as follows: (1) Apply KFCM to cluster the set of objects and generating a U membership matrix [12] (2) In all elements, x i , x j , the number of t nearest neighbor is found.
where ⊕ defines the exclusive OR that totally indicates the overlap between two fuzzy sets. (6) Here, diagonal matrix, D, is formulated as follows: And, it is normalized as follows: e number of K bigger eigenvector than L (the first vector) is found (the first vector is selected), and the matrix P � [p 1 , p 2 , . . . , p k ] has been formed, and then, the algorithm normalizes the rows in the P matrix to form matrix Y. (7) Each line of Y has considered a point in space R k and at then, the final clustering has been established by the K-means algorithm.

Improved Neural Network Algorithm
Optimization. Generally speaking, in most applications of engineering, optimization is a vital subject. Optimization is to make the best decision to get the optimal (minimum or maximum) result for the considered problem. Several methods have been introduced for optimization. However, classic methods as exact methods can solve these problems, and they fail in some cases that the problem is nonlinear or complicated. To overcome this problem, another technique, called metaheuristic, has been introduced [13]. Metaheuristics include a set of optimization algorithms that are inspired by nature, human behaviors, animals' competitions, etc. e main advantage of using metaheuristic algorithms is  Computational Intelligence and Neuroscience that they use random structure instead of using gradient methods. is simplifies the optimization process [14]. Furthermore, the most complicated problems are that their number is increasing day by day and they cannot be solved by the classic methods, but these methods, based on their stochastic nature, can find a near-optimal result in a logical time. ere are different types of these algorithms like ant lion optimizer (ALO) algorithm [15], chimp optimization (CO) algorithm [16], Harris Hawks optimization [17], and world cup optimization (WCO) algorithm [18]. Recently, a new metaheuristic algorithm, called neural network algorithm (NNA), is introduced, which is inspired by the concepts of biological nervous systems and artificial neural networks (ANNs). Based on [19], each ANN includes some artificial neurons, which are inspired by the biological nervous systems. e relationships among units principally specify the network function. e NNA approach is illustrated in the following.

Neural Network Algorithm.
Like any other metaheuristic algorithm, the NNA starts with an initial population that is called the pattern. In NNA, just like ANN, the main idea is to update the pattern population to minimize error among the forecasted data and the desired output data. Here, the best solution is the desired output that can be updated in each iteration for achieving the minimum error amount by moving the pattern population in the direction of the desired solution. In the following, the algorithm methodology is briefly explained.

Initialization.
e first operation in this algorithm, like any other metaheuristic, is to generate some random population (decision variables) for initial evaluation. By considering a D dimensional problem, the considered pattern solution vector will be an array of 1 × D, representing input data in the NNA. Moreover, by considering the D dimension and N pop several random candidates, the initial pattern population can be considered as follows: where the matrix X is made randomly between the minimum and maximum limitations of a problem. And, the cost value is achieved as follows: where f describes the objective value.
After evaluating the objective value of each pattern solution, the best one is selected as the best pattern solution.
e NNA has N pop input data and D dimension with only one target data. Figure 6 shows this structure.
After defining the target solution (X T ) among the other pattern solutions, its weight (W T ) has to be chosen from the population of weight (weight matrix).
In an ANN, the neurons connect with dendrite using a simple summation. e output is connected to the input layers based on weighted (w) interconnection. Initial weights are random values in this algorithm, and then, they have been updated based on some equations to provide the minimum network error. e initial weights in the algorithm are as follows: where W includes uniformly distributer random values in the range [0, 1]. e first and the second subscripts of weight relate to the pattern solution and participated with the other solutions of the pattern. All pattern solutions have their weight value to generate a new candidate solution. ese weights have also a constraint that should not exceed 1 and mathematically formulated as follows: where i, j � 1, 2, . . . , N pop . Weight is randomly distributed values in the range between 0 and 1, where their aggregate in a solution of the pattern must not be more than one, which is because of the bias control of the movement and to generate the new Computational Intelligence and Neuroscience pattern solutions. If the constraint is not considered, the probability of sticking in the local optimum by the values of the weight will be increased. After weight matrix formation, new pattern solutions (X N ) are evaluated as follows: After achieving the new pattern solutions using the best weight amount that is named target weight, this updating is established by the following equation: Another operator for updating in NNA is the bias operator.
is operator is employed for modifying the exploration part of the algorithm by using a specific percentage of the candidates in the new population (X New i (t + 1)) and updated weight matrix (W updated i (t + 1)). e pseudocode of this operator is as follows:

End for End if End for
Here, LB and UB represent the problem minimum and maximum bounds, orderly. c describes the modification factor that defines the candidates' percentage. c � 1 at first, and then, it decreases gradually during the process based on any decreasing formulation as follows: In the NNA, there is also a transfer function operator to transfer the new candidates to the new updated positions to provide better value against the target solution. e solution reimproved by giving the novel pattern solutions to the best solution direction (target solution). e transfer function operator (TF) can be formulated as follows: e pseudocode of the collaboration between bias and TF operators is tabulated below: Apply the transfer function operator End if End for 4.3. Improved Neural Network Algorithm. However, the neural network algorithm provides good results to solve different applications of optimization problems [20][21][22], and it is sometimes stuck in the premature convergence that gives a high impact on the solution accuracy. In this investigation, to modify this drawback, 2 mechanisms are considered. e first mechanism is to employ the Chaos theory. is mechanism explores novel situations, which are dynamic discrete-time, i.e., In this research, the logistic map function has been used as the modification mechanism. is mechanism could be mathematically modeled as follows: where i signifies the populations' number, β j,i,q defines the value for the i th chaotic iteration, ρ describes a constant, which is set 4, q demonstrates the iteration number, j illustrates the generators' number in the system, and β 0 describes the initial value of β i with a random value between 0 and 1 [23,24]. With assuming the above explanation, the set of initial variables w ij is rewritten as follows:  Computational Intelligence and Neuroscience is study also used Lévy flight (LF) as a second modification. e LF is a popular technique for improving optimization algorithms [25]. e LF uses random motion to control the local searching. is is obtained by the following equations: where τ represents the Lévy flight index in the range [0, 2] (here, τ � 1.5 [26]), w describes the step size, A, B ∼ N(0, σ 2 ), Γ(.) describes Gamma function, and the samples generate a Gaussian distribution with σ 2 variance and zero mean value. By assuming the LF mechanism, the updated equation for the new pattern solutions based on the best weight value is achieved as follows:

Algorithm Validation.
To assess the effectiveness of the proposed improved neural network algorithm, six standard benchmark functions have been utilized, and the results are compared with some new state-of-the-art algorithms, containing multiverse optimizer (MVO) [27], moth-flame optimization (MFO) algorithm [28], world cup optimizer (WCO) [19], and the original neural network algorithm (NNA). e parameter settings of the algorithms are as follows: for MVO: wormhole existence probability (WEP); WEP min � 0.2; WEP max � 1; Coefficient(P) � 6. For MFO: logarithmic spiral shaped(b) � 1; for WCO: α � 0.5, playoff � 4%. e population size for whole algorithms is set to 100, and their maximum iteration is set to 100. Table 1 tabulates the studied benchmark functions for the analysis. e benchmark functions are considered with 30 dimensions.
To validate the algorithms based on the studied functions, four statistical indicators including minimum amount (Min), maximum amount (Max), mean amount (Mean), and standard deviation amount (Std) are employed, and the results are indicated in Table 2.
According to Table 2, the outcomes of the suggested INNA have the proper outcomes in terms of lower, higher, and amount. Because it provides the minimum value for these values that is the main purpose of these benchmark functions, therefore, the proposed INNA has the highest accuracy. Furthermore, the proposed method with a minimum standard deviation value provides the highest reliable solution for the studied benchmark functions.

Feature Extraction and Selection Based on INNA
Feature selection is one of the most important methods and techniques for data preprocessing and data mining. Due to the introduction of new programs for large data mining, media information retrieval, and medical data processing that requires the processing of large volumes of data, it is important to limit the number of features. e purpose of feature extraction here is to take out the important characteristics from the region of interest in skin cancer images to simplify the diagnosis process. In this paper, 19 different features have been used for extraction from the image. Table 3 indicates the used features for the extraction. erefore, here, we utilized a feature selection technique to reduce and select some more important features and to eliminate the low-cost features. Different methods have been introduced for this purpose. Metaheuristics are a kind of new technique for feature selection.
Feature selection using metaheuristics is one of the subsets of feature extraction and is discussed in various fields of machine learning and data mining. In general, this problem does not have a definite solution, and so far, no precise method has been proposed to solve it. Various classical approaches have been proposed to these problems, but usually, the quality of their solutions is generally not very good. In contrast, intelligent optimization methods can provide far better solutions to these problems. erefore, one of the effective and constructive methods in solving feature selection problems and related issues is the use of metaheuristic optimization methods and evolutionary algorithms.
Here, we employed a cost function for optimum se- e idea is to minimize the abovementioned function with selecting the high cost and important features. Based on the designed INNA, the above function has been minimized, and less-cost features are eliminated from the processing. e next step is to employ an efficient method for classifying these features.
After performing the binary optimization technique on the features, the following features with higher effectiveness are selected for training: elongation, area, mean, correlation, entropy, and elongation. is selection is performed based on a metaheuristic technique based on a stochastic nature.

Skin Cancer Image Classification
Classification is the process of separating an input image into predetermined classes. Image classification is considered the last part of the diagnosis in medical image analysis. is step Computational Intelligence and Neuroscience can decrease the time for the cancer diagnosis process by decreasing the search space in it. A popular and useful method in classification is to use a support vector machine (SVM). e SVM seeks the best hyperplane that operates as a multipart (here two-part) data separator in the input space.
For reaching the constraints of the optimization problem, the support vector machine has been employed to evaluate the normal vector, w, in the hyperplane, the bias b, and the slack variable, η for incorrectly assigned training patterns that support the generalization, which is defined as follows:

Function
Formula Minimum Limitation Rosenbrock Table 2: e simulation results of the comparative algorithms on the studied benchmark functions, moth-flame optimization (MFO) algorithm [28], world cup optimizer (WCO) [19], and the original neural network algorithm (NNA). Algorithm 4π × area/perimeter 2 Standard deviation Variance 1/2 Form factor Area/a 2 MN describes the image size, B p defines the external side length for the boundary pixel, p(i, j) defines the pixels intensity amount at position (i, j), μ and σ describe the mean value and the standard deviation, orderly, and a and b present the major axis and the minor axis, respectively. However, some of the above features have a high impact, and some others have a low impact on feature extraction. 8 Computational Intelligence and Neuroscience y � min With separation function: Feature space is highly affected by the separating function in equation (21), so that a function selection needs to be performed correctly to generate optimal output. Here, for minimizing equation (20), the suggested INNA has been utilized. Accordingly, the support vector machine uses a kernel that is employed to alter data to the feature space. In this research, three different kernels including linear, polynomial, radial basis function (RBF), and sigmoid have been used. Figure 7 shows a typical SVM.
As can be explained, the SVM technique is a proper tool for classifying the images. Medical images are also a part of this event. erefore, after performing feature extraction on the images for decreasing the operation complexity, the skin cancer features for each image can be injected into the SVM, and the SVM will present the final classification, which will be healthy or cancer.

Implementation Details.
As mentioned before, the proposed skin cancer detection system contains different modules that are generated in MATLAB R2019b environment. e modules include image segmentation, image acquisition, feature extraction and choice, and the last classification. In the image acquisition stage for dermoscopic images, American Cancer Society (ACS) [30] and PH 2 [31] databases are used: where TP, TN, FP, and FN denote the true positive, true negative, false positive, and false negative. So, by analyzing the efficiency of the INNA-SVM method by considering different kernels based on the abovementioned indicators, we have the following.
It can be observed from Table 4 that the linear kernel of the proposed INNA-SVM, in addition to simplicity, results in the best values for all indicators for both databases: sensitivity, accuracy, and precision. erefore, this kernel is utilized here for the classification of the features. For a comprehensive investigation about the proposed method's efficiency, five evaluation measurements including specificity (SP), positive predictive value (PPV), negative predictive value (NPV), F1 score, and Matthews correlation coefficient (MCC) are considered, which are mathematically as follows: e general proposed method has been compared with five new methods, and the comparison results have been indicated in Table 5. e compared methods include Astorino's method [5] based on multiple instance learning (MIL), Hassan's method [6] based on a simple pipeline image preprocessing technique, Barros's method [7] based on hardware designing by the multilayer perceptron, Santos's method [8] based on neural networks and fuzzy logic, and Wang's method [9] based on deep convolution networks. e results are achieved from the mean value of both databases.
According to Table 5, the F1-score of the suggested method with 84.39% that defines its accuracy based on the precision and recall of the test data is the highest among all of Computational Intelligence and Neuroscience the compared methods. Also, the value of MCC in the proposed method with 94.21% is the highest among the others, and because it uses all of the analysis terms, it shows the method's higher efficiency than the others. Also, the higher value of the NPV and PPV with 95.57% and 85.09% against the other methods shows higher condition occurrence of the method to control the likelihood of a test identifying cancer toward the others. Finally, the higher SP value based on the proposed method reports higher occurrence-independent results of the algorithm.

Conclusions
Skin cancer is one of the most common cancers, and malignant melanoma is the most invasive and deadliest kind of this cancer. Early detection of this cancer by physicians has a high effect on the treatment of this cancer. Recently, several machine vision-based techniques are used for helping physicians with accurate early detection. is paper proposed a new approach for optimal skin cancer detection. e first section in the proposed method was to establish a preprocessing technique including noise reduction and contrast enhancement. en, the region of interest (ROI) was segmented by a kernel fuzzy C-means segmentation method. e features of the ROI were then extracted, and the most important features were selected optimally. Afterward, the selected features were injected into an optimal classifier using support vector machine (SVM) for classification. e main contribution of this study was to present a developed version of a new metaheuristic, called neural network optimization algorithm, to optimize both parts of feature selection and SVM classifier in the system. e superiority of   the proposed method was proved by performing a comparison among the suggested approach and five state-of-theart methods.
Data Availability e data are available at https://www.cancer.org/.

Conflicts of Interest
e authors declare no conflicts of interest.