Hybrid Time-Series Framework for Daily-Based PM2.5 Forecasting

The impact of fine particulate matter on health has captured attention worldwide. Many studies have proven that fine particulate matter harms the respiratory system and the cardiovascular system. To prevent people from being harmed, many scientific research studies on PM2.5 prediction have been conducted in recent years. Accurate PM2.5 forecasting can not only alert people to stay away from concentrated areas but also provide the government with environmental policies in the future. In this paper, we propose a hybrid time-series prediction framework for daily-based PM2.5 forecasting. The proposed framework consists of three components: the autoencoder, the dilated convolutional neural network, and the gated recurrent unit. The experimental dataset with 76 monitoring stations from the Taiwan Environmental Protection Administration is applied for comparison of the baseline and the proposed models. The proposed model is not only for the specified city-/county-wide region but also for the particular monitoring station/site to predict PM2.5 concentration. By considering air quality data, meteorological data, and geographical data simultaneously, the proposed model can increase the accuracy of PM2.5 prediction. In addition, the proposed PM2.5 forecasting model can learn the location-centric spatial features and the daily-based temporal features simultaneously. The experimental results show that the prediction accuracy of the proposed model is superior to those of the baseline models.

I. INTRODUCTION
From many research studies of air quality in recent years, the results revealed that rapid climate change and serious environmental pollution have impacted human health worldwide. Additionally, many studies have shown that air quality prediction becomes more important for the living environment, particularly fine particulate matter (PM 2.5 ). From the viewpoint of the government, the warning of PM 2.5 concentration can be helpful to make environmental policies and to remind citizens to stay away from polluted areas. Hence, PM 2.5 forecasting and monitoring are not only national but also international topics for humans.
To live in a healthy environment, many studies have addressed air pollution intensity and air quality forecasting [1], [2]. Additionally, most of the research studies applied either theoretical methods or simulation models to The associate editor coordinating the review of this manuscript and approving it for publication was Turgay Celik . highlight the situation under air pollution [3]. Machine learning has been applied to predict air quality. Dong et al. [4] proposed a method for PM 2.5 forecasting by using the hidden semi-Markov model (HSMM). Donnelly et al. [5] proposed real-time air quality forecasting, which is based on integrated parametric and nonparametric regression. Furthermore, weather and climate trends are considerably relevant to air quality; hence, applying traditional machine learning models is insufficient for air quality prediction.
In recent years, some studies have proposed a novel method that is used for prediction models. The equipment degradation processes possessed long-range dependence and multimode characteristics. The causes of the multimode characteristics include the external environment and operating conditions, as well as the equipment loads throughout its lifetime. Duan et al. [6] developed a multimodal fractional Lévy stable motion degradation model, which is used to predict the product technical life or the remaining useful life of equipment. Liu et al. [7] proposed a prediction model of the remaining useful life based on the generalized Cauchy stochastic process. Reliable gearbox prediction is a complex problem. To overcome the gearbox reliability problem, a hybrid model based on the fractional Lévy stable motion, the gray model and the metabolism method was proposed [8]. In this model, the feature extraction method is used to reveal gearbox degradation and to solve gearbox insensitivity to weak faults. In 2021, a new long-range-dependent degradation model was proposed to predict the remaining useful life of rolling bearings [9]. The degradation model is based on the generalized Cauchy process and can describe the local irregularities and the global correlation characteristics of the time-series data. Linear mathematical models cannot adequately describe wind-speed characteristics. To overcome the limitations of the linearity assumption, a novel model based on the generalized Cauchy process was introduced [10]. In this model, the fractal dimension and Hurst parameter are combined for simulation and forecasting of the wind speed. Furthermore, the prediction model is applicable to describe the local irregularity and global correlation of wind speed.
Many studies have addressed the interrelation between air pollution factors such as PM 2.5 and meteorological data. Traditional machine learning lacks the adoption of datadriven approaches to process time-series air quality data [11], [12]. Nevertheless, deep learning models can apply to data-driven methods [13], in which the features of air quality data are competently extracted. In the domains of image classification, speech recognition, and natural language processing, deep learning models have made remarkable achievements [14]- [16]. Shallow learning models are utilized to predict air quality, and deep learning approaches are appropriately applied to predict time-series air quality data [17]- [20]. In [21], a multivariate time-series method was conducted for air quality prediction. In addition, deep learning models are most suitable for dealing with data-driven approaches and applicable to handle time-series data.
A hybrid time-series framework for daily-based PM 2.5 forecasting is proposed in this paper. The nationwide and city-/county-wide regions for air quality prediction are all covered. Both the temporal and spatial correlation dependencies of features are learned from air quality time-series data, such as wind speed, PM 2.5 , and coordinates. We summarize the major contributions of this paper in the following paragraphs.
First, we develop a hybrid time-series deep learning model, which is composed of three components. Autoencoder (AE) and dilated convolutional neural network (CNN) learn spatial features through air quality and geographical data. Gated recurrent unit (GRU) extracts temporal features through air quality and meteorological data. Compared with the existing models, such as the ST-DNN model proposed by Soh et al. [22], the proposed model can decrease the average MAE and RMSE values by 16% and 18%, respectively. In addition, our model also shortens the average training time by 12%.
Second, according to the experimental results, the proposed model is not only accurate on a nationwide scale but also adequate for region-wide prediction of time-series data such as air quality. Furthermore, our model can be applied to predict the air quality of the target site. The proposed model is superior to the existing prediction models.
The rest of this paper is organized as follows. Section II describes the related research works. The methodology of the proposed PM 2.5 forecasting framework is presented in Section III. Section IV depicts the experimental setup and the prediction comparison, of which the prediction error and the training time are considered in our work. We show the conclusion and future work in Section V.

II. RELATED WORKS
For the literature on PM 2.5 forecasting, almost all existing works dealt with the prediction accuracy of air pollutants by using machine learning models and statistical methods [3], such as HMM [4], regression [5], artificial neural networks [23], and ARIMA [24]. Zhang et al. [1], [2] proposed a real-time air quality prediction approach, which focuses on the analysis of the major research trends and current status, as well as future directions. In Zhou et al. [25], the hidden temporal dependencies regarding PM 2.5 were addressed with Lasso-Granger by developing a probabilistic dynamic causal (PDC) model. The hybrid model is based on the regression neural network and empirical mode decomposition for the previous 24-hour PM 2.5 prediction proposed by Zhou et al. [23]. In Deleawe et al. [26], a machine learning model that conducts air quality measurements in the urban environment was used to predict the CO 2 levels.
In addition, many studies show that deep learning models have been used for air quality prediction. Air quality data possess time series and nonlinear characteristics, and thus, data-driven models are directed to address the topic of urban computing [27]. Moreover, many PM 2.5 forecasting studies are based on big data, which can obtain the predicted results by adopting many historical and multivariate data [28]. Zheng et al. [11] proposed a semisupervised learning model that combines CRF with ANN classifiers. Hsieh et al. [12] developed a semisupervised method to conduct fine-grained and real-time air quality data. An air quality prediction framework in real time that applies data-driven models was presented in Zheng et al. [29].
With regard to the capability of data-driven methods and nonlinear problems, deep learning models have generally been adopted to solve time-series and sequence data problems [6], [30], [31]. That is, air quality data possess characteristics such as time-series data. In [32], Li et al. [32] presented a novel air quality forecasting model by utilizing spatial-temporal deep learning (STDL), which considers the temporal and spatial correlations. To predict air pollution, Ong et al. [33] developed a deep recurrent neural network (DRNN) by adopting the autoencoder approach. In [18], Qi et al. [18] proposed a deep air learning (DAL) model to deal with feature analysis, interpolation and forecasting.
In [34], Zhang et al. [34] developed a deep residual neural network to extract the features of time-series data and analyzed the congestion of citywide crowds.
In [35], a hybrid deep learning framework was developed, which was combined with multiple deep neural network models. The hybrid framework has also been applied to the topics of video classification and face detection [36]. Additionally, hybrid deep learning frameworks have not been well fitted to handle air quality forecasting issue predictions [37]. Many researchers have shown that hybrid deep learning models have exceptional performances compared to classic deep learning models [36].
Many works have shown that convolutional neural networks (CNNs) have excellent performance in video processing and image recognition [16]. Indeed, it is also applied to time-series data prediction [31]. Among deep learning models, CNNs are superior to time-series data and multivariate data. Recurrent neural networks (RNNs) are applicable for learning the time dependencies and extracting the temporal features of time-series data. Furthermore, to mitigate the vanishing gradient problems of RNNs, Hochreiter et al. [38] developed a variant of RNN, LSTM, which refers to the internal states of the memory cells for mitigating the vanishing gradient problems. RNN can predict time-series data; moreover, LSTM is excellent in time-dependent feature extraction.
Du et al. [37] proposed a hybrid deep learning model to predict PM 2.5 in Beijing, and the hybrid model was composed of 1D-CNN and Bi-LSTM. In his work, two datasets were applied to PM 2.5 forecasting: the Beijing PM 2.5 Dataset from the US Embassy in Beijing, and the Urban Air Quality Dataset from the Urban Air Project of Microsoft Research. The experimental results show that 1D-CNN can effectively extract the local trends and spatial features from the air quality time-series data. Soh et al. [22] developed an adaptive deep learning model to forecast PM 2.5 in Taiwan and Beijing. In his work, the Taiwan dataset was collected from the Taiwan Environmental Protection Administration, and the Beijing dataset was provided by the Urban Air Quality Dataset from the Urban Air Project of Microsoft Research. The experimental results also showed that CNN can extract the surrounding targets and spatial features from the air quality data.
Du et al. [37] proposed a deep air quality forecasting framework (DAQFF), which is composed of various deep learning models. The main idea of the DAQFF is not only to deal with time-series forecasting issues but also to address spatiotemporal data features. In addition, the DAQFF correlates the multivariate air quality data. The DAQFF can extract the temporal features as well as the spatial features from air quality data. One-dimensional CNN and bidirectional LSTM are two components of DAQFF; the former deals with the spatial data features, and the latter deals with the temporal data features [39], [40].
In [41], Yu and Koltun proposed a convolutional network approach to conduct dense forecasting, and the proposed model applies dilation factors to aggregate multiscale contexts without degrading the resolution. The proposed model introduces dilation factors for the sake of expanding the receptive fields. In addition, dilated convolution can increase the accuracy and efficiency of dense forecasting.
Soh et al. [22] developed an adaptive air quality prediction model that includes multiple deep learning models. The spatial-temporal deep neural network (ST-DNN) considers terrain and meteorological data concurrently, which means that ST-DNN can extract spatial and temporal features from air quality data. ST-DNN combines three deep learning models: the first two models are artificial neural network (ANN) and long short-term memory (LSTM), which extracts the temporal features from the air quality data; the last one is the convolutional neural network (CNN), which extracts the spatial features from the air quality data. In summary, ST-DNN can extract spatial correlations and the temporal dependencies of neighboring locations. Zhang et al. [42] combined CNN and LSTM to achieve higher forecasting accuracy of air pollution.

III. METHODOLOGY
In this section, we first present the framework of this paper and then state three main components in the framework. That is, the autoencoder, one-dimensional CNN and GRU are described in order.
In this work, our motivation is to develop a deep learning framework for time-series PM 2.5 forecasting. To consider the comparability and fairness of the model performances, we compare the classic and existing deep learning models and choose a traditional machine learning model for comparison. In this paper, PM 2.5 forecasting considers both the location correlation of multiple monitoring stations and the time dependency of a single monitoring station. CNNs can effectively extract the local trends and spatial features of different districts. The GRU possesses a memory mechanism, so it can effectively extract the short-and long-term temporal features of a particular district. Fig. 1 depicts the components and functionalities of our framework. On the left-hand side, the air quality and geographical data are first input into the autoencoder layers and then into the dilated convolution layers. On the right-hand side, the air quality and meteorological data are input into the GRU layers. Afterward, the air quality input data are output from the abovementioned layers and merged with the concatenate layer for data fusion. One flattened layer is then introduced to feed the following dense layer with the input data. Finally, the predicted PM 2.5 values are generated.
By considering the interrelated factors of variant data sources, we consider the correlation of geographical areas, meteorological conditions, and air quality time-series data simultaneously. AE and dilated CNN can effectively extract PM 2.5 concentrations from the specified district based on historical air quality, and GRU can effectively extract PM 2.5 concentrations from the seasonal climate based on historical air quality. To predict the PM 2.5 concentration of particular monitoring stations under various climate circumstances, we combine the learning results of AE and dilated CNN with GRU for all the time periods.
The proposed model is named the hybrid time-series framework (HTSFW), which combines unsupervised and supervised models for daily-based PM 2.5 forecasting. The HTSFW model applies AE and dilated CNN to extract the local trends and spatial features. Additionally, HTSFW utilizes GRU to extract the long dependencies and temporal features.
For conducting the spatiotemporal features of the air quality data, we extract the spatial features from the PM 2.5 values correlated with the monitoring station locations, such as the coordinates; in addition, we concurrently retrieve the temporal features from the PM 2.5 records interrelated to the weather and climate factors, such as the wind speed.
The air quality-related dataset is composed of geographical data and meteorological data, such as PM 2.5 , longitude, latitude, SO 2 , CO, O 3 , NO 2 , PM 10 , wind speed, temperature and humidity. To increase prediction accuracy, the HTSFW model consolidates the training results of the geographical data and the meteorological data. In the HTSFW model, the missing values of the air quality dataset were padded with zeros. In other words, the same experimental dataset was applied to all the baseline and HTSFW models. The data contents are recorded by day; hence, PM 2.5 forecasting generates daily time frames.
As is known, the innovation of the HTSFW model can not only consider a single monitoring site for time-dependent meteorological factors but also examine wide regions for location-interrelated geographical factors. In addition, the HTSFW model combines the unsupervised with the supervised models, which has excellent performance in PM 2.5 forecasting. The details of the HTSFW model are discussed in the following subsections.

A. AUTOENCODER
The first model used in the proposed framework for the air quality and the geographical data and autoencoder is an unsupervised deep learning model [10]. The simplest autoencoder network has one hidden layer. The input layer first encodes the high-dimensional data; then, the hidden layer stores the low-dimensional codes as the intended data features. Additionally, the functionality of the output layer is to use the low-dimensional codes to reconstruct the high-dimensional input vectors.  The dimensionality of the input layer is five, and the input data are encoded and then stored in the hidden layer. Therefore, fewer neurons of the hidden layer lead to the outcome of data compression or dimensionality reduction. With the same dimensionality as the input layer, the purpose of the output layer is to decode the hidden representation from the previous layer and reconstruct it to the original input data.
Furthermore, multiple hidden layers can be constructed to form the stacked autoencoder (SAE) network. In the encoding phase, the dimensionality of the next layer has fewer neurons than the previous layer, which means that each neuron ignores useless data and keeps meaningful data. During the training process, backpropagation can be utilized for fine-tuning the connection weights. In the decoding phase, in contrast to the encoding phase, the dimensionality of the next layer possesses more neurons than the previous layer; each neuron learns the data features and reconstructs the original input.
In addition to dealing with the geographical relationship set of the target stations for PM 2.5 forecasting, we develop an approach to select the target region or site, which is simultaneously applicable for extracting the local trends and long dependencies. Algorithm 1 depicts the proposed method.
Here, d i and s j indicate the particular district and station, respectively, and i and j denote the district and station if d i = d target then 6: sort s j contained in d i by ascending order 7: sort t k recorded in s j by ascending order 8: end if 9: end for 10:end if identifiers, respectively. Moreover, t k denotes the timestamp of the data record, and k represents the timestamp index.
Algorithm 1 first sorts the district identifiers in ascending order; the HTSFW model defines the target district identifier afterward. Then, the proposed algorithm searches the matching district identifier and sets the matching district to be the target district. After that, the monitoring stations of the target district are sorted by the station identifiers in ascending order. Finally, each target monitoring station is sorted by the timestamps in ascending order.
As shown in Algorithm 1, the target district identifier can refer to a single district or multiple districts. In other words, the regional coverage of PM 2.5 forecasting can be dynamically conducted in accordance with the proposed algorithm.
In this paper, an experimental dataset with 76 monitoring stations (Jan. 2014 to Jun. 2019) in Taiwan is downloaded, which was provided by the Taiwan Environmental Protection Administration (TWEPA). In the downloaded dataset, the missing values of the data items were recorded to be empty. To deal with the missing values in the dataset, we padded the mentioned data items with zeros.
B. ONE-DIMENSIONAL DILATED CONVOLUTIONAL NEURAL NETWORK SVM was developed in 1992 and is a supervised machine learning model. It has been widely used for data classification and regression. SVM is also a forecasting method based on a statistical learning framework. One of the SVM models, LSSVM, is a least-squares version of SVM, which is to minimize the sum of the squared errors of the objective function. SVR, another version of SVM, was proposed in 1996 and is mainly used for data regression. Du et al. [37] proposed the DAQFF model, which can accurately predict PM 2.5 concentrations. To compare the performances of the proposed and baseline models, SVR, ARIMA, LSTM, GRU, CNN and RNN were tested in his work. For the next one-hour prediction of the Beijing PM 2. 5  The next model used in the proposed framework for the air quality and geographical data is a one-dimensional dilated CNN. The main consideration is stated as follows. CNNs are well suited for spatial feature extraction, while one-dimensional CNNs apply to time-series data. The dilated convolution network can extract air quality features from the input time-series data. In addition, using dilation factors can expand the receptive fields and further enhance the training efficiency. Both are discussed in the following subsections.

1) 1D-CNN FOR TIME-SERIES DATA FEATURE EXTRACTION
In the image processing field, convolutional neural networks are commonly adopted [16]. Nevertheless, CNN is also applied to time-series data prediction. The classical CNN commonly consists of convolutional, activation and pooling layers. Furthermore, the two-dimensional CNN is popularly utilized for image classification [35]. In this work, the HTSFW model utilizes the one-dimensional CNN to predict the PM 2.5 concentration. In general, the activation function of the one-dimensional convolutional layer is depicted as follows: As shown in Equation (1) is the activation function of the convolutional layer, where * indicates a convolution operator, w t i and b t i represent the weights and biases, respectively, and x t represents the t-th time step of the input data.
The HTSFW model uses two connected one-dimensional convolutional layers to extract the spatial features from the geographical data, and the two connected layers constitute the hierarchy to represent the local trend features. In other words, the two connected one-dimensional CNNs can learn the local trend features of a single monitoring station and can also extract the hidden spatial correlation features from multiple stations.
In Fig. 3, the time-series features of the input air quality data are first filtered through the one-dimensional convolution kernel, and the activation function processes the input features, weights and biases. After that, the extraction process of the activation function generates the output targets.
Since the air quality input dataset contains time-series data items, the one-dimensional CNN is adopted to compress the length of multivariate input data and learn the air quality data features. Due to the local perception and weight sharing of the one-dimensional convolution network, the number of parameters decreases, and the learning efficiency improves.

2) DILATION FOR LEARNING PHASE EFFICIENCY
In our model, we applied 1D-CNN to extract the local trend and spatial features from the time-series data. Since air quality data are related to the time sequence, we apply the dilation factors in the hidden layers to expand the receptive fields.
When the air quality data are inputted to 1D-CNN, by using the dilation convolution, our model can speed up the learning process both in the single time step and the multiple time steps.
Yu and Koltun [41] developed a convolutional network module that applies dilation factors to aggregate multiscale contextual information. The dilated convolution module can exponentially expand the receptive fields without losing coverage. The dilated convolution is defined as follows: Equation (2) states the one-dimensional dilated convolution, in which dilation rate l convolves input F with kernel k, where * l denotes a dilated convolution operator and p = α + l;α = {l−1, 3l−1, 5l−1, 7l−1, . . . ,Ml−1−l} ;M is the input bucket size.   4 represents the dilated convolution operation. First, the one-dimensional dataset is input into the dilated convolution layers. Next, the dilation factors are implemented in the hidden layers; to put it differently, dilation factors are set to one, two, four and eight. Afterward, as previously mentioned, the receptive fields of the next layers can expand exponentially. As a result, the one-dimensional dilated CNN can decrease the number of parameters and reduce the training time. For example, Fig. 4 shows three hidden layers with 16 input data. Based on the depiction of the elapsing time sequence in Fig. 3, the indices start from zero and pass through the left to the right.
In [43], Zhen et al. presented a dilated CNN approach for sequence prediction. The approach utilizes dilation factors to extend the receptive fields and introduces residual connections to form a deeper network. The experimental video analysis results revealed performance enhancements with fewer parameters and shorter running times. Hence, inspired by the convolutional neural network with dilation factors, this work proposes a one-dimensional dilated CNN to expand the receptive fields and increase the training efficiency for air quality forecasting.
In this paper, two one-dimensional convolution layers are concatenated, and the dilation rate is applied to two for both layers. The number of output filters is 64, and the length of the 1D convolution window is one in the first dilated convolution layer. In the second dilated convolution layer, we set the number of output filters to 128, and the length of the 1D convolution window is one. The padding of the two dilated convolution layers is parameterized to the same dimension for both the input and output.

C. GATED RECURRENT UNIT
The exploding and vanishing gradient problems of traditional RNNs are inevitable. To mitigate gradient problems, a long short-term memory (LSTM) network was developed in 1997 [38]. LSTM refers to the internal states within the memory cells for mitigating the mentioned gradient problems. In 2014, another RNN variant, the gated recurrent unit (GRU), was proposed by Cho et al. [14]. LSTM is composed of three main building blocks: the input gate, forget gate, and output gate. In other words, GRU has a simpler network than LSTM. The GRU consists of two main components: the reset gate and update gate. For this reason, the GRU training process results in better efficiency.
The reset gate is responsible for the short-term memory, while the update gate is in charge of the long-term memory. In this work, GRU is used for certain reasons: one is the functionality of hidden state handling, which implies that GRU is well suited to time-series data prediction; the other is a simpler network, which results in a faster learning process. Therefore, GRU is feasible for extracting the long-term temporal dependency features. Fig. 5 shows the main components of the GRU building block. The main components are combined to handle the hidden states and retain the temporal features over a period of time. The main components of the single GRU block are represented as follows: (6) VOLUME 9, 2021 As denoted in the above formulas, r t represents the reset gate that determines the amount of previous information to be ignored, z t represents the update gate that determines the amount of previous information to be passed,ĥ t denotes the candidate hidden state, h t−1 represents the previous hidden state, and h t represents the current hidden state. In addition, x t indicates the input data, W and U are the weights, and b is the bias.
In this work, GRU is applied to retrieve the long-term and time-series features of the air quality data. The HTSFW model extracts the local trend features using AE and a one-dimensional dilated CNN; additionally, the long-term spatial-temporal correlation features hidden in multivariate time-series data are extracted by using GRU.
The local trend features and the long-term correlation features are concatenated afterward, followed by the flattened layer, which transforms the features into a vector. In addition, the vector dimensionality is reduced by the fully connected layer. Eventually, the output of the training process results in PM 2.5 forecasting.

IV. EXPERIMENTS
In this section, we first describe the experimental dataset of this work and interpret the experimental parameters and the settings afterward. Moreover, we conduct performance comparisons of the baseline and proposed models, the ST-DNN [22] and the proposed models.

A. DATASET
In this paper, the experimental dataset is sourced from the Taiwan Environmental Protection Administration (TWEPA). The dataset contains air quality-related data that cover all of the cities and counties in Taiwan, and each data record is a daily average. The data features include air quality, geographical and meteorological data such as PM 2.5 , longitude, latitude, SO 2 , CO, O 3 , NO 2 , PM 10 , wind speed, temperature and humidity.
The time of the dataset is from Jan. 2014 to Jun. 2019, and the data interval is 24 hours. There are 76 monitoring stations built around all cities and counties in Taiwan. To deal with the missing values in the dataset, those recorded as empty were padded with zeros.
Many works show that air quality is highly related to meteorological circumstances. For instance, moderate air pollution is due to high wind speed, good air quality can be due to high atmospheric pressure, and high humidity deteriorates the PM 2.5 concentration [11], [29].
To prepare the experimental dataset, we sort the data items by geographical location after downloading the raw data from the TWEPA. That is, the neighboring cities/counties are sorted in order. For data preprocessing, we extract the spatial correlation features between the air quality and the geographical regions.
In the experimental dataset, each monitoring station possesses air quality, geographical and meteorological data features. In addition, each data item contains a recorded timestamp. The 76 monitoring stations are distributed in Taiwan; therefore, this work can predict the PM 2.5 concentration in Taiwan and the city-/county-wide regions, as well as the particular sites. The HTSFW model is compared with five deep learning models and one machine learning model. The baseline models embrace convolutional neural networks (CNNs), recurrent neural networks (RNNs), and two variants of RNNs, which include long short-term memory (LSTM) and gated recurrent units (GRUs), and support vector regression (SVR). To further compare with the ST-DNN model [22], it is also considered a baseline model.
In our work, the default parameters in Keras are applied to weight initialization. During the training phase, for overfitting prevention, a dropout rate of 0.3 is configured. In addition, the lookup size, batch and epoch are 1, 32 and 100, respectively. The activation functions of CNN and RNN (including LSTM and GRU) are ReLU and tanh, respectively. Adam is used as the optimizer. The learning rate is set to 0.001. The values of beta one and beta two are 0.9 and 0.999, respectively. The epsilon is 1e-7.
For the baseline models, the number of hidden layers is set to one by default, and each hidden layer possesses 128 neurons. In the HTSFW model, we first utilize two AE layers that concatenate with two one-dimensional convolution layers with a dilation factor of two for air quality and geographical feature extraction. In addition, the number of output filters and the length of the convolution window of each layer are applied to (64, 1) and (128, 1), respectively.

TABLE 1. Prediction error comparison between the baseline and proposed models.
We also adopt each of the two GRU layers with 128 hidden neurons for air quality and meteorological feature extraction. The mean square error (MSE) is used as the loss function in the training process, and the activation function used in the output layer for target prediction is tanh. Furthermore, the input time-series data are normalized to [0, 1] by using the min-max function, while the missing values in the dataset of the data items padded the data items with zeros. The dataset is divided into two parts: the first four years of data, and the last 18 months of data. The former part is used for training, and the latter part is used for testing.
For training process evaluation, two metrics, MAE and RMSE, are applied to measure the learning performances. The two error indices are denoted as follows: where n stands for the number of testing data, y i indicates the actual PM 2.5 value, andŷ i represents the predicted PM 2.5 value.

C. PERFORMANCE COMPARISON OF BASELINE AND PROPOSED MODELS
In this section, the HTSFW model is compared with the baseline models. It is noted that the coverage area of PM 2.5 prediction is all the cities and counties in Taiwan, and the air quality data are from the 76 monitoring stations. In addition, both the single and multiple time steps of PM 2.5 forecasting are shown in Tables 1 and 2.
Since all the data items are 24-hour averages, one step indicates one day later, and multistep represents the specified days later. The prediction size in Tables 1 and 2  of baseline models and the proposed HTSFW model are compared by indices MAE and RMSE, respectively.
As shown in Table 1, the performance of PM 2.5 forecasting 24 hours later shows that the HTSFW model can decrease the ratio of MAE by 26.52%, 25.92% and 40.12% by using CNN, RNN and LSTM, respectively. The HTSFW model can reduce the percentage of RMSE by 22.57%, 7.32% and 42.11% by applying the respective CNN, RNN and LSTM models.
In Table 2, for the PM 2.5 prediction one day later, the performance indicates that the HTSFW model can decrease the proportion of MAE by 36.13%, 66.26% and 53.13% by utilizing GRU, SVR and ST-DNN, respectively. In addition, the HTSFW model can reduce the scale of RMSE by 36.24%, 56.46% and 46.22% with GRU, SVR and ST-DNN, respectively.
In addition, we highlight the preprocessing of the missing data items in the experimental dataset. The lack or absence of data item values is inevitable, such as the malfunction of the monitoring station over a five-year duration. The handling of the missing data values is important because it is related to the experiment proceeding. In this work, the missing values of the experimental dataset are padded with zeros. The missing values with the zero-padding approach are applied to all the baseline models and the proposed model.
As shown in Tables 1 and 2, the prediction performances of the proposed model are compared with four classic deep learning models, one traditional machine learning model and one existing deep learning model. The comparison includes single-and multistep forecasting performances, and the prediction sizes are parameterized to ten time steps. In addition, MAE and RMSE are used for performance comparison. In our model, the columns of improvement show a reduction in the ratio of the prediction errors. Based on the improvement values, the proposed model is superior to the six baseline models in the ten time steps by using MAE and RMSE.
The recording period of the experimental dataset is from 01/01/2014 to 06/30/2019, and the data interval is 24 hours. There are 148,399 data samples in total. The time of the training data is from 01/01/2014 to 12/31/2017, and the number of data samples is 107,357 (72%); the time of the testing data is from 01/01/2018 to 06/30/2019, and the number of data samples is 41,042 (28%). To consider all the seasonal factors affecting the PM 2.5 concentration, we use a forecasting period between 01/01/2018 and 12/31/2018. In terms of the performance comparison of all the forecasting models, the HTSFW model can also extract the time-series data features from the seasonal interrelation.   Tables 1 and 2. According to the bar charts corresponding to MAE averages plotted in Fig. 6, the prediction errors of the HTSFW model are clearly lower than those of the baseline models. SVR has the highest MAE average of 10.76, while HTSFW has the lowest MAE average of 7.26, which means that the proposed model can reduce the prediction errors for both short-term and long-term periods.   According to ten different time step performance comparisons in Tables 1 and 2, the HTSFW model is superior to all the baseline models overall. Based on the experimental results, the contribution of this paper is stated as follows. First, the HTSFW model can accurately predict PM 2.5 concentrations in various regions concurrently, for example, urban and rural areas. In other words, HTSFW can predict PM 2.5 concentrations for monitoring stations located nationwide. Second, the HTSFW model can also accurately predict PM 2.5 concentrations in not only a short period of time but also a long period of time. PM 2.5 forecasting in those periods is superior to the existing deep learning models. Therefore, HTSFW is applicable for predicting PM 2.5 concentrations in both single time step and multiple time steps.

D. PERFORMANCE COMPARISON BETWEEN ST-DNN AND PROPOSED MODELS
In this section, the HTSFW model is further compared to the ST-DNN model [22]. First, we compare the training time for all 76 monitoring stations in ten different time steps. Next, we select four cities/counties in Taiwan and compare the prediction errors of the monitoring stations located at each of the four cities/counties. In addition to comparing the prediction errors, we further select four particular sites that are located in the four cities/counties.
As shown in Table 3  time steps indicates that ST-DNN takes 20.57 seconds and HTSFW takes 18.11 seconds. As a result, HTSFW can decrease the ratio of the average training time by 11.96%. Fig. 8 shows the geographical map of all the cities/counties in Taiwan. The upper part of the map in green is the northern region, the middle part in blue is the central region, the lower area in yellow is the southern region, and the right side of Taiwan in red is the eastern region. To predict the PM 2.5 concentration of the specified regions, we apply the developed Algorithm 1 to select the target districts. This work selects one city/county from the four regions mentioned previously indicated by the red arrow signs on the map. The four target districts are Taipei City (TPE), Taichung City (TCH), Tainan City (TAN) and Taitung County (TAT). Our Algorithm 1 can define arbitrary target district identifiers, although we select the four cities/counties in consideration of their representativeness of the four regions in Taiwan. In addition, the four selected target districts possess different weather conditions and climate circumstances. Table 4 represents the experimental comparison of prediction errors between ST-DNN and HTSFW. We train the two models in five different time steps and compare the forecasting errors of the four target districts. In the testing data of the four cities/counties, the highest maximum PM 2.5 concentration value of 74 is in TAN, while the lowest maximum value of 23 is in TAT. For the PM 2.5 forecasting one day later, the HTSFW model can decrease the ratio of MAE by 35.04% and 64.88% in TAN and TAT, respectively; additionally, HTSFW reduces the scale of RMSE by 40.62% and 57.14%.
With respect to the predicted size of 12, the HTSFW model decreases the percentage of MAE by 5.45%, 18.75%, 19.82% and 37.67% in TPE, TCH, TAN and TAT, respectively, while HTSFW can reduce the percentage of RMSE by 9.39%, 9.58%, 19.19% and 31.33%, respectively. According to the prediction comparison in the five different time steps, the HTSFW model is conspicuously better than the ST-DNN model. From the PM 2.5 forecasting of the nationwide coverage and the regional districts, HTSFW performs well in the time series with air quality data prediction.
In Table 4, the experimental results of the HTSFW model show that the forecasting errors of the four regions in five prediction sizes decrease. Since ST-DNN is an adaptive deep learning model, it is composed of traditional machine learning and classic deep learning models. Our hybrid model is further compared to the ST-DNN model by the peak error values. The peak error indicates that the peak predicted PM 2.5 values of the two models are different from the observed PM 2.5 values. Based on the experimental results, HTSFW possesses lower peak errors than ST-DNN in the TPE, TCH and TAT regions of five predicted sizes. For the TAN region, HTSFW has lower peak errors than ST-DNN of predict sizes FIGURE 8. Regional city/county for PM 2.5 forecasting in Taiwan. Note: https://eego.epa.gov.tw/english/tour1/index1.asp? Parser = 99,10,27,,,,,1. For prediction sizes three, six and twelve, HTSFW has slightly higher peak error values than ST-DNN. Fig. 9 depicts the particular monitoring station Yangming in Taipei City; the station Yangming is located in the mountain area. The maximum PM 2.5 concentration of the testing data of station Yangming is 32 in spring. The black trend represents the ground truth, the red line indicates the PM 2.5 prediction of ST-DNN, and the blue line represents the PM 2.5 prediction of HTSFW. From the plotted depiction, the one-year prediction of HTSFW is closer to the ground truth than that of ST-DNN.   10 shows the PM 2.5 forecasting of station Fengyuan, which is located in the urban area of Taichung city. The maximum value of the PM 2.5 concentration of the testing data at the Fengyuan station is 52 in the spring. According to the prediction results, the ST-DNN model [22] is worse than the HTSFW model, especially the peak points of the trend, as shown in the forecasting figure. In Fig. 11, the figure indicates the PM 2.5 prediction of the Tainan station located in an urban area in Tainan City. The maximum value of the PM 2.5 concentration of the testing data at the Tainan station is 65 in the spring. From the plotted representation, the forecasting trend of the HTSFW model is quite close to the ground truth. In contrast, the PM 2.5 prediction of ST-DNN is close to the ground truth when the values are below 20. However, the prediction of the ST-DNN model [22] is evidently different from the ground truth when the values are above 20.
In Fig. 12, the figure represents the PM 2.5 forecasting of the Guanshan station, which is located in a rural area in Taitung County. In eastern Taiwan, there is a sparse population, and most of the people in the east earn a living by farming, particularly Taitung. The maximum PM 2.5 concentration of the testing data at station Guanshan is 21 in autumn. Due to lower pollution from industry and vehicular traffic, the annual VOLUME 9, 2021  In this work, to evaluate the proposed model and the developed Algorithm 1, we first compare the baseline models with the proposed model by using the air quality data from the 76 monitoring stations in Taiwan. Furthermore, we compare the training time and the prediction errors of the existing ST-DNN model [22] with those of the proposed HTSFW model. For the comparison of the ST-DNN and HTSFW models, we compare their prediction errors of the four specified target districts and the sites belonging to each district. Moreover, we compare the prediction errors of selected particular sites belonging to each of the four target districts. According to the experimental results, the HTSFW model outperforms other models in different coverage regions and prediction periods.
Based on the experimental results, the innovation of the HTSFW model is expressed as follows. First, for air quality and geographical data training, the HTSFW model adopts the dilated CNN for feature extraction. Dilation factors are used to expand the receptive fields that lead to increased training efficiency. In addition, HTSFW applies GRU to time-series feature extraction for air quality and meteorological data learning. As shown in Table 3, the learning time decreases significantly because the GRU network is simpler than the LSTM network. Second, we develop an innovative algorithm to select the target district for PM 2.5 forecasting in Algorithm 1. By using the proposed Algorithm 1, the proposed HTSFW model can accurately predict the PM 2.5 concentration of a particular site as well as the specified regions. In Table 4 and Figs. 9-12, HTSFW can accurately predict the PM 2.5 concentration of the four cities/counties in northern, central, southern and eastern Taiwan. Furthermore, for the selected monitoring station in each of the four cities/counties, the PM 2.5 prediction of the HTSFW model is obviously better than that of the ST-DNN model [22].

V. CONCLUSION
In this paper, we propose a hybrid time-series deep learning model for daily-based PM 2.5 forecasting. The accuracy of the PM 2.5 prediction is enhanced by considering the air quality data and the meteorological data simultaneously. In addition, the PM 2.5 concentration is also related to geographical location and time frame. For the performance comparison, we first compare the MAE and RMSE of PM 2.5 prediction between the baseline and the proposed models. For ten different time steps, our model is superior to the baseline models. Moreover, to predict the air quality of the four city-/county-regions, the proposed model is further compared to the ST-DNN model [22]. The selected regions are located in northern, central, southern, and eastern Taiwan. Furthermore, we select one monitoring station in each region for the accuracy comparison. From the experimental results, the proposed model is accurate for all the regions and suitable for local regions and specified sites.
In our work, the proposed HTSFW model can accurately predict the PM 2.5 concentration of the specified monitoring station or regional district in Taiwan. In addition, in this COVID era and post-COVID period, both the administration and citizens have to avoid symptomatic infection and spread of the epidemic. For instance, timely announcements of the setup of screening stations and the confirmed case locations are both critical factors. The government needs to effectively manage and predict the COVID-19 vaccine distribution; hence, the forecasting models need to consider the spatial correlation, temporal dependency, air quality, and traffic transportation simultaneously. In other words, the forecasting models have to effectively extract the data features from variant sources and accurately predict the COVID-19 vaccine distribution in time.
In this work, we develop a hybrid deep learning model to increase the accuracy of PM 2.5 forecasting. Such hybrid models are based on the combination of multiple machine learning and/or deep learning models. The adaptive ST-DNN model is proposed to predict the PM 2.5 concentration of a single monitoring station, which is combined with machine learning and deep learning models. Our HTSFW model is composed of multiple deep learning models that enhance the prediction accuracy of the existing models. Since 2014, GANs have been widely applied to time-series data such as natural language processing [15], [17]. Air quality prediction is highly related to time-series data, and therefore, we will consider utilizing GAN for our future work.
GAN was designed in 2014 and is one of the deep learning models [15], [17]. In this model, two neural networks compete with each other based on zero-sum game theory. The generative network generates new data through unsupervised learning, and the discriminative network evaluates the new data through a competitive process. It has been widely used for image, video, and natural language processing. Inspired by the sequential data of GAN applications, we will seriously consider including GANs in future work.
For future work, we will further study the heights of each monitoring station for PM 2.5 prediction. In addition, we will also develop a new PM 2.5 forecasting model to deal with seasonal climate change to enhance the air quality prediction accuracy.


Brain Image Segmentation to Diagnose Tumor by Applying Wiener Filter and Intelligent Water Drop Algorithm

Segmentation of image has wide application in the field of medical, military, surveillance, etc. This work segments medical resonance image for detection of tumor in brain where work identify three parts in the image. First is skull, second is brain and third is tumor. Presented paper includes description of image segmentation in unsupervised manner where proposed model identify all segments of image without any training. Here, wiener filter preprocessed the input images by removing unwanted information from the image matrix. Filtered image then passed in Intelligent Water Drop (IWD) genetic algorithm for finding representative pixel value sets of image segments. Graphical water drop movement in the IWD algorithm has representative pixels value set selection accuracy. Experiment is performed in real dataset of brain tumor and detection is done by taking reference of ground truth images. Proposed model evaluated average precision value 0.98 and average accuracy 96%. Hence, when results is compared with existing methods then it is obtained that proposed segmentation work increased the segmentation evaluation parameter values.

I. INTRODUCTION
Digital image processing applications in image segmentation, provides support for medical image diagnosis, digital object detection in videos, change detection analysis, etc. Image segmentation leads to separation of image pixels into two or more cluster sets as per feature set, fitness function, etc [1]. Large number of research paper have been published on digital image segmentation in last few decades for resolving different issues. Some of papers are to identify background, some identify objects, pay per hour for medical image diagnosis [2]. So algorithms proposed in this kind of papers are under low level engineering where images are segmented as per pixel values into few clusters. Images were taken from different resources, environment so some noise were present in it then some of noise removal techniques for also proposed by scholars for enhancing digital image segmentation efficiency.
Number of techniques for image segmentation work were proposed and differentiated on the basis of their feature vectors extracted from input image and such features are edge, Manuscript  corner, histogram, wavelets, etc. Each of proposed algorithm for image segmentation reduces false pixel assignment [3]. Image processing holds an important position in the medical field having the form of magnetic resonance image, ultrasound images, X-ray, tomography study, etc. These medical images improve understanding of doctors or medical practitioners and provide treatment [4]. As per the experience of medical practitioners, identification of tumor in brain region was done but due to manual involvement, accuracy of detection varies and risk of treatment gets increases. So need of some automatic algorithm was required for the diagnosis of tumor in the brain region for improving the medical prescription accuracy, rate of development, change detection etc. Hence image segmentation for this kind of treatment have its own role. Some of MRI machines were developed with this facility for supporting the human experience [5].
Paper is structured into few sections first is introduction then second section elaborate various research work done by scholars in field of image segmentation. Further in third section paper explanation of IWD is given. Fourth section has detailed the proposed genetic based image segmentation model for detection of brain tumor with block diagram explanation. In fifth section paper compares performance of proposed model with other algorithms. Finally, paper get conclude with findings of proposed model.

II. LITERATURE SURVEY
The author [6] proposed a new rough to fine (CFM) method for brain tumor segmentation. considered hierarchy included preprocessing, deep classification of networks and post-processing. Preprocessing was used to generate image patches for all MR images and the deep learning network output was a grievous image patch. The high-level abstract function extracted from the input using the stacked auto encoder framework for identifying patches of images. After that classification results were mapped into a binary image, the final segmenting result was achieved with a morphological filter. For the analysis of the proposed method, the experiment was used for the brain tumors segment for each patient data package.
The author [7] presented the Convolutional Neural Networks in which multi-succession magnetic resonance images were grouped for tumor detection in brain region. Various CNN were given that focused on MAGNETIC RESONANCE IMAGE patches, cuts and volumetric multi-planar cuts without any preparation. Two existing ConvNets (VGGNet and ResNet) models prepared in ImageNet dataset, through a calibrating of the last scarcely any layers, likewise test the reasonableness of move examination to the mission. The Leave one tolerant out assessment plot was applied to evaluate ConvNets output. ConvNets outcomes shown that it accomplished better accuracy in all situations when arranged for a multi-planar volumetric informational collection. The author [8] built up a DLF (Deep Learning Framework) for brain tumor sections and has anticipated the endurance of glioma by methods for Magnetic Resonance Image. This paper utilized different 3 Dimension CNN engineering sets for MRI tumor detection rule. This diminished model inclination and improved execution efficiencies. They selected solid endurance properties utilizing a decision tree and cross approval. An irregular backwoods model was in the long run evolved to assess the patients by and large endurance.
The author in [9] built up a prescient Machine Learning Model for the analysis of brain tumors. Routine blood tests must require substantially more data than even the most prepared specialists by and large acknowledge. They built up an AI prescient model in which neurological patients were tried for mind tumors normally. Review assessment of 68 sequential brain tumors and 215 control patients for the nervous system science administration has affirmed the model.
In the paper [10], another digital image division/segmentation strategy was proposed by joining the FCM bunching calculation with an unpleasant set hypothesis. To start with, the quality worth table was built dependent on the division/segmentation consequences of FCM under various bunching numbers and the digital image was separated into a few little districts dependent on the undefined relationship of characteristics. At that point, the weight estimations of each quality were gotten by esteem decrease and utilized as the premise to ascertain the distinction among areas and afterward the likeness assessment of every locale is acknowledged through the identicalness relationship characterized by the distinction degree. At last, the equality connection characterized by closeness was utilized to blend areas and complete digital image division/segmentation. Paper has identified few Problems mentioned in following points 1. In binary masked image was required to remove the background. Hence work was static as the mask was different for different set of images. 2. Calculation of information table for each image was compulsory for finding the segmentation class of sub images, this increased execution time. 3. In Pre-processing , input image description was not detailed like noise removal or skull identification. 4. Given Classification accuracy is 96%, which can be further enhanced.
Our proposed model removed noises from image through wiener filter then segmented the image by using Intelligent Water Drop algorithm. It also avoided drawbacks specified for paper that is summarized in above paragraph. Our proposed model is dynamic in nature and increases adoptability of tumor detection by fulfilling objectives i.e. 1. To apply pre-processing filter to remove noise from input image. 2. To Develop dynamic skull detection and removal technique. 3. Increase the tumor detection accuracy.

III. INTELLIGENT WATER DROPS
IWD takes its motivation from common natural phenomena. IWD copies the conduct of streaming of water drops with the waterways. In light of the above property of flowing water, a keen algorithm was developed by Shah Hosseini [11]. In this every single intelligent water drop was expected to have three significant properties. First is the measure of soil that each water drop conveys with it, spilling out of one water node to another. Second is the speed of the water drop, that increment or diminishes according to the collected soil along the way. Total soil conveyed by the water drop relies on the speed of the streaming water drop. Fig. 1 shows the speed of the water drop increments marginally when moving through the stream bed with more soil. The speed of the subsequent water drop increments when moving through the stream bed or with less soil.
As Intelligent Water Drop Algorithm is inspired from nature/Biological conduct it comprises all the properties of hereditary calculation like populace age, wellness estimation, choice, hybrid and transformation while discovering arrangement of an issue.
Our proposed model is particular for Image Segmentation by utilizing Weiner channel and Intelligent Water Drop calculation by discovering delegate pixel esteem sets of a digital image section. Subtitles are depicted in segment beneath.

IV. PROPOSED MODEL
This section describes proposed ISWIWD (Image Segmentation by Weiner and Intelligent Water Drop) model which is working for brain image segmentation. It's block diagram is shown in Fig. 2. Whole work is divided into three module i.e. first is image for pre processing, second is written for skull part removal and third section elaborate working of segment representative pixels finding.

A. Image Preprocessing
In image processing, wiener filter is very prominent method for removing unwanted data from the image [12]. In our method, input image (as shown in Fig. 3(a)) was transformed into two dimensional format where pixel value ranged between 0 to 255.Image was reshaped further in order to apply matrix operations by the same approach that is used in [13]. Wiener filter removed an additive noise and reduced mean square error, as it's linear estimation of original image.
Changes as per Weiner filter was done by Eq. 3 where v 2 is noise variance.

B. Image Skull Removal
Filtered image (as shown in Fig. 3(b)) was further processed into two dimensional format where pixel values ranged between 0 and 1. Here binary conversion was done by applying otsu method. Otsu is a method to find gray threshold value for binary conversion. Binary conversion removed skull part of an image. Otsu provided threshold values for portioning the image into black and white color as shown in Fig. 4(a). Once binary image format obtained then scanning of image, left to mid in a row and right to mid in a row took placed. While scanning from left to mid in a row, proposed module point first white pixel position and then first black pixel position. So all pixels from first white to first black pixel were considered as skull part in the row as shown in Fig.  4(b). Similarly while scanning from right to mid in a row, proposed module point first white pixel position and then first black pixel position. So all pixels from first white to first black pixel were considered as skull part in the row as shown in Fig. 4 (c). Once skull part of the image was identified and removed then resetting of gray scale image values to 0 was done. So output of this module was again a two dimension matrix whose pixel value ranged in 0 to 255. Fig. 4(c) is processed brain region from image.

C. IWD Genetic Algorithm
In the IWD genetic algorithm, each pixel acts as a water drop where one drop move towards other drop and soil between the drop positions is less. So soil act as distance between the pixel values [14]. of pixel or water drop created segments in the image. Output of this module was segment representative pixel values Develop Water Graph : First phase of this intelligent water drop genetic algorithm was to develop a graph where each node is pixel and other distance between them act as weight of graph [15]. Soil of the graph acts as graph weight which was estimated by Eq. 4.
where i, j are pixel values belongs to image and S is the soil matrix of pixels. Drop Movement Probability: As image I was segmented into s class, so each representative pixel moved toward other pixel of image as per soil value. So movement probability of water drop help in this selection using Eq. 6.
where WS is Weighted Soil, FS is Feasible Solution, is constant value range between 0 to 1.
Modify Drop Velocity and Soil Values: Here i th drop velocity (DV) was updated by Eq. 8 when it moves towards i th drop soil was updated by Eq. 9 when it moves towards j th drop.
HD is heuristic durability, a constant value ranged between 0 and 1.
where in above Eq. 10 F c is fitness value of c th chromosome in the population P and P c,s is representative pixel in c th chromosome and s th segment.
IWD Crossover: As population need updating so crossover operator performed this changes where each chromosome in the population get modified as per best segment representative pixel set in a current iteration. Hence lowest fitness value of the chromosome acted as best solution and random position pixel value was replaced in this crossover operation.
This updated population chromosomes were further analyzed to identify that new chromosome solutions are better as compared to previous parent or not. following International Journal of Computer Theory and Engineering, Vol. 13, No. 3, August 2021 conditions considered for changes i.e. in case new chromosome found better, then replace parent solutions in population otherwise parent will continue and new will discard. Proposed Algorithm:

V. EXPERIMENT AND RESULTS
Experimental work of proposed model was performed on MATLAB software having machine configuration of 4GB RAM, windows 7 operating system and I3 processor. Results were compared with existing algorithm FCMRS (Fuzzy C Mean and Rough Set) proposed in [10].

A. Dataset
Experiment was performed on brain tumor Segmentation dataset (code 105) obtained from the repository maintained on URL https://ijsret.com/2017/12/14/computer-science/. This dataset consist 100 images where 50 are original image and 50 are ground truth image set. Dimension of each image are not same so proposed model work on random size image.

B. Evaluation Parameter
Evaluation parameters value of segmented image was obtained considering the basis of ground truth images of dataset. So segmented images obtained from proposed and existing algorithm are system generate pixel class (Tumor or non tumor region). Formulas of evaluation parameters are [16]: Here CC = Correct Classification and IC = Incorrect Classification. Table I and Fig. 6 shows that precision value (calculated by Eq.13) of proposed model ISWIWD is better than FCMRS model proposed in [10]. ISWIWD considered noise removal by wiener filter which increased the accuracy of work. This paper has enhanced the tumor detection by utilizing genetic algorithm for tumor pixel values identification. Proposed model has improve the work precision value by 1.97%.  Table II shows that use of Intelligent water drop genetic algorithm for segmentation has increased the recall value (calculated by Eq. 14) of ISWIWD model by 1.83%, as compared to FCMRS [10].  Table III shows that F-Score value (calculated by Eq. 15) of proposed model ISWIWD is better than previous FCMRS model proposed in [10]. Proposed model pre-processing step of noise removal by Weiner filter has increased the precision and recall which ultimately enhance f-measure parameter as well. This paper has enhanced the tumor detection by utilizing IWD genetic algorithm for tumor pixel value identification. Proposed model has improved the f-measure value by 1.98%.   Table IV and V shows that proposed model increased the positive likelihood parameter value (calculated by Eq.11) while decreased the negative likelihood value (calculated by Eq. 12) in all set of images. This is achieved by utilizing the genetic algorithm as use of rough set reduces the pixel selection variety for segment representation.   Fig. 7 shows the use of Intelligent water drop genetic algorithm for segmentation has increased the accuracy value (calculated by Eq. 16) of ISWIWD model by 3.58%, as compared to FCMRS [10]. Removal of skull has also increased the tumor and non tumor segmentation accuracy.

VI. CONCLUSION
Segmentation of an image plays an important role in medical diagnosis. This paper has resolved this issue by developing a ISWIWD model for image segmentation where three regions are identified first is skull, second is tumor and third is non tumor. Proposed model has applied a weiner filter in the input image which removes unwanted information in form of additive noise. Skull portion of input MRI image may distract the segmentation therefore scanning of image from left to mid and right to mid was done to remove skull part. Finally unsupervised intelligent water drop genetic algorithm was used for the tumor region detection. Experiment was performed on real dataset where results were compared with FCMRS method. It is shown that ISWIWD has improved the work precision value by 1.97%, while accuracy value was enhanced by 3.58%. In future one can develop machine learning method to reduce detection time and improve accuracy as well.


Personalized One-Shot Lipreading for an ALS Patient

Lipreading or visually recognizing speech from the mouth movements of a speaker is a challenging and mentally taxing task. Unfortunately, multiple medical conditions force people to depend on this skill in their day-to-day lives for essential communication. Patients suffering from Amyotrophic Lateral Sclerosis (ALS) often lose muscle control, consequently their ability to generate speech and communicate via lip movements. Existing large datasets do not focus on medical patients or curate personalized vocabulary relevant to an individual. Collecting a large-scale dataset of a patient, needed to train mod-ern data-hungry deep learning models is, however, extremely challenging. In this work, we propose a personalized network to lipread an ALS patient using only one-shot examples. We depend on synthetically generated lip movements to augment the one-shot scenario. A Variational Encoder based domain adaptation technique is used to bridge the real-synthetic domain gap. Our approach significantly improves and achieves high top-5accuracy with 83.2% accuracy compared to 62.6% achieved by comparable methods for the patient. Apart from evaluating our approach on the ALS patient, we also extend it to people with hearing impairment relying extensively on lip movements to communicate.

Introduction
: We propose a personalized one-shot lipreading framework to tackle a real-world medical challenge of lipreading a patient suffering from ALS 1 . The patient communicates primarily by mouthing the words. In our case of limited available real data, we use a combination of synthetic data augmentation technique and a domain adaptation technique called Variational Encoders to build a robust word-level lipreading model for the patient.
Lipreading is mentally taxing and can affect the communication quality. For instance, speakers with hearing impairment may lack holistic audio feedback [29] and ALS patients may have lesser control over their mouth muscles [4,16]. This may cause them to have irregular and unreliable mouth movements making it difficult for people to lipread them. Applications and automated algorithms capable of lipreading a person can thus significantly improve the day-to-day communication of people dependent on lipreading. Motivated by this need, we tackle the real-world challenge of lipreading a patient suffering from ALS and people with hearing impairment. ALS is a progressive nervous system disease that affects nerve cells in the brain and spinal cord, causing loss of muscle control [32]. An ALS patient may lose their voice and rely solely on mouth movements for communication [12,21].
Current Works and Limitations: Current deep learning techniques are inherently datahungry. Collecting large amounts of data specifically from a patient is, however, not an option. Mouthing words is a tiring maneuver for people suffering from ALS, and thus a patient undergoes physical and mental stress during such data collection exercises. Manually labeling words mouthed by a person is time-consuming. It is thus crucial to use the minimum amount of manually labeled data to build lipreading models that can work well on a person.
Recent years have seen much progress in word-level [8,17,28] and sentence-level [3,31] lipreading. Oxford's Visual Geometry Group released large-scale in-the-wild datasets such as Lipreading in the Wild (LRW) [5] and Lipreading sentences (LRS) [1,2] consisting of 1000+ speakers. LRW, the most relevant dataset to our task, is a word-level lipreading dataset made of 1000 examples for 500 English words but turns out to be somewhat limiting: (1) The speakers in the dataset do not have any speaking disability thus making perfect mouth movements. (2) It is curated by cropping words from long speech segments resulting in fastpaced speech with co-articulation in the videos. (3) It contains a large amount of head motion and variations like the different characteristics of the mouth region, both of which are unnecessary for lipreading specific medical patients. Thus, SOTA models like LipReading without Pains (LRwP) [8] and Lipreading using Temporal Convolutional Networks (LTCN) [17] trained on LRW do not directly adapt to speakers with speaking disabilities. Figure 2: Speakers in our study. In left to right order, the first speaker suffers from ALS and relies solely on lip movements. The next two speakers primarily use sign language while making imperfect lip movements. Next speaker uses deaf speech along with sign language for daily communication. The following speaker is the 46th president of the USA, Joe Biden.
LRW only supports a limited pre-curated vocabulary missing out on medically essential words like 'nauseous' or 'backache'. It also lacks a personalized vocabulary relevant to a person's daily communication. Deploying systems to enable a persons' communication would need highly accurate models on their specific lip movements for their particular vocabulary. The problem of personalized lip reading has also been explored in [22]. According to [22], lip movements vary across speakers. Observing a single speaker for an extended period could lead to better speaker-specific lipreading models. They collect ∼20 hours of data per speaker to train a personalized lipreading model generating speech purely from an individual speaker's lip movements. Collecting such a large dataset is, however, not always an option.
Lack of medical data has been studied [9,11,26] widely in the past. Taking inspiration from these, we formally tackle one-shot lipreading in a personalized setting. We first synthetically generate data using a SOTA lipsync network [23]. We then use a SOTA lipreading network [8] as our backbone and use synthetically generated data along with very limited real examples to train a word-level lipreading network. Our approach includes an important domain adaptation step using a novel network -Variational Encoders -a modified VAE for bringing a vast number of synthetic examples closer to the real domain using only one real instance per class. We specifically tackle the use case of an ALS patient and also explore the same for four other speakers (refer to Fig. 2) including the 46th President of the United States, Joe Biden, as an additional example to show that our approach can easily be extended to speakers with no disability. Our contributions in this work are threefold: 1. We tackle a real-world medical challenge of lipreading speakers with ALS and hearing impairment by developing highly accurate personalized models for each speaker.
2. We propose Variational Encoders, a novel network-based on VAE. Instead of autoencoding, they exploit the loss of the downstream task for generating task-relevant latent distributions. The learned distributions are then used for domain adaptation.
3. To the best of our knowledge, we are the first to propose lipreading in one-shot setting.
In this vein, we curate a medical dataset involving speakers with medical conditions.
2 One-shot personalized lipreading framework As mentioned previously we aim to build a personalized lip-reading model for each speaker using only single real examples. Fig. 3 presents the pipeline for personalized lipreading.  Figure 3: Personalized lipreading -We start by curating personalized vocabulary and collect unlabeled videos for each speaker. TTS models are used to generate speech utterances for the curated words. The speaker's unlabeled videos and the generated speech utterances are given as input to Wav2Lip that generates synthetic data to augment one-shot data. Variational Encoders then use the synthetic and the one-shot data to train the model for lipreading.
The use of synthetic data to augment low data in the medical domain [9,11,26] has gained traction with improving generative models. Similarly, we augment the one-shot examples collected for each speaker by generating synthetic data for each of them. To achieve this, we use a SOTA talking face generation model, Wav2Lip [23] pretrained on the largescale LRS2 [6] dataset. Given a speaker's video, Wav2Lip preserves the speaker's pose, facial expressions, and mouth characteristics like beard and skin color while modifying the speaker's lip movements according to a guiding speech. We generate word-level speech utterances using SOTA TTS models FastSpeech2 [24] and GlowTTS [14] as a replacement for the speaker's voice. Using these TTS models allows us to generate variations in the speech in terms of the speed of the spoken word, pitch, and energy. Additionally, we collect unlabeled face videos for each speaker which, along with the generated speech utterances, is used to create 1 hour of speaker-specific synthetic data on an average. The augmented dataset is then used to train an LRwP and LTCN based architecture for the classes curated per speaker.
A combination of the speaker-specific synthetic videos and a single real video per class are used to train our model. The synthetic data helps the model learn the general underlying word-level characteristics for the new classes. However, the properties of personal style of lip-movements for a word -could be because of the medical condition -is not captured in the synthetic dataset. We utilize the one-shot examples for introducing the properties of personal speaking style in the model. Single examples per class are however, not enough to capture the underlying style variations of a speaker. A person may not utter the same word, exactly, each time. To tackle this, we use our novel approach -Variational Encoders.  Figure 4: Variational Encoders -Encoder S and Encoder T denote the source and target encoders respectively. Encoder T generates a latent distribution using the target examples. The latent distribution is trained against the downstream classification task. Encoder S aims to learn robust domain invariant features by minimizing the distance between its learned embedding and points randomly sampled from the generated target distribution. latent representations preserve important domain characteristics. However, using a decoder to reconstruct the input accurately is a non-trivial task, especially for videos needing spatial and continuous temporal reconstruction. We propose Variational Encoders, a modified VAE that uses the loss of the downstream task to generate task-relevant latent distributions.

Variational Encoders: Mapping words into distributions
How do Variational Encoders differ from a standard VAE? Similar to VAEs, Variational Encoders (refer to Fig. 4) generate latent distributions given a sample. A key difference with a standard VAE is -instead of autoencoding, it uses the final multi-class classification loss of the downstream task to generate a task-relevant latent distribution that represents the "class" of input instead of a generic input feature. The learned distribution is used to sample a variation of the input example. We hypothesize that this sample introduces the missing speaker-specific style variation for the input class. The sampled variation encourages the encoder to see and align to the potential interpretation of the input class and learn robust representations. In summary, Variational Encoder retains all the benefits of a VAE, like generating domain invariant features while removing the complicated video reconstruction loss.

Network Architecture
Given a source domain S and a target domain T, domain adaptation aims to bridge the gap between the two domains by generating domain invariant features. We denote S and T as, where N S is the number of source samples and N T is the number of target samples. Variational Encoder is a paired domain adaptation network that assumes the target domain is labeled. However, the number of examples in the target domain is expected to be at max k where k ≈ 1. Thus, we assume N S >> N T . The adaptation network is made of two encoders -source and target encoders -and a single classifier. For a given inputx i belonging to either domain, the encoder learns to generates an embedding ex i for the input. A distribution p(x i ) is then generated for the input denoted by µx i and σx i from the learnt embedding ex i . A random point is sampled from the generated distribution denoted as zx i = µx i + σx i ε.
The encoder network for both domains are identical. Each encoder generates the embedding e, and the latent distribution p. To introduce domain invariance, we use KL divergence and L1 loss. The KL divergence is computed between the two generated distributions p(x i ) = (µ x i , σ x i ) and p(x i ) = (µx i , σx i ). In addition, a point zx i is randomly sampled from the target distribution p(x i ). L1 loss is then applied between zx i and the source embedding e x i . This minimizes the distance between the source embedding against several randomly sampled points from the target distribution thus acting as a pseudo for multiple target examples.
Gradient stopping: The L1 Loss between the source embedding and the sampled target embedding will force the target distribution down to a single point, losing the essence of a distribution. To prevent that, we stop the gradient from flowing back through the target encoder. This way, only the source encoder is regularized against the variations sampled from the learnt target distribution, while leaving the target encoder unaffected. We denote the sampled target embedding as zx i detach , where detach denotes that the sample zx i does not have any gradient. The combined loss to bridge the two domains is given as, where β is a hyper-parameter and i denotes both the samples belong to the same class.
For the downstream task of classification, we employ a common classifier that is trained on both domains at the same time. A single classifier allows both encoders to have a common base-loss helping them generate features relevant to the common downstream task. The downstream classifier also trains on large amounts of variations and learns robust representations. The classifier receives a randomly sampled point zx i from the learnt distribution px i as input for the target domain. For the source encoder, however, the classifier receives the learnt embedding e x i instead of a randomly sampled latent point z x i . The motivation behind a target distribution is to encourage the target encoder to hallucinate variations of the single real examples available for each class. However, during inference, we want to obtain a definite point for the input to avoid any uncertainties. The classification loss is given as, where g can be any classification loss such as cross entropy or negative log likelihood. y denote the same label for both the source x i and targetx i input. The combined loss for the entire network is then given as (see Fig. 4), where α and γ are hyper-parameters. All the network components are trained end-to-end.

Experiments
Dataset: We first collect a set of unlabeled videos for each speaker that are used later to generate the speaker-specific synthetic data using Wav2Lip. Unlabeled videos of the ALS patient are recorded without any manual intervention. For other speakers, the videos are randomly selected from their respective YouTube channels. As the next step, we curate the personalized vocabulary for each of the speakers. For the ALS patient, we curate a list of 200 words with the help of his family. We modify the existing list for the remaining speakers by removing irrelevant words and adding the most occurring keywords in the transcription of the collected videos. Splits of the curated dataset for each speaker is presented in Table 1. We obtain two sets of manually curated real data from the ALS patient's family. We use the first set for model training and the second set as the intermediate test set. We collect additional data by deploying a website that records the patient's word-level mouthings and displays the inference on our best model (see Fig. 1). The website needs external help to start and stop recording the patient. The helper can then either select one of the displayed words as the correct label or manually assign the correct label. Through this exercise, we collect an additional 320 examples. Out of these, we use 200 data points, in addition to the original 200 train data points, to train a model in a two-shot setting. We report our test results of oneshot and two-shot models on the combined set of the intermediate test set and the remaining additional 120 examples (see Table 1). To simulate the same setting, we maintain one real example for training the rest of the speakers. To generate the train and test examples, we use the transcriptions with timestamps available on YouTube for the selected videos.
Preprocessing: Our preprocessing steps are similar to LRwP. The lip landmarks are first detected using dlib [15]. The lip is then cropped out such that it is horizontally and vertically centered in the cropped image. The image is converted to gray-scale and resized to a fixed dimension of 88 × 88. A maximum sequence length of 64 frames is used. A batch size of 16 on a multi-GPU NVIDIA GeForce RTX 2080 setup is used. The models are trained up to 200 epochs using a cosine scheduler and Adam optimizer, with a 3e − 5 learning rate and a weight decay of 1e − 4. The encoders in the experiments are adopted from LRwP that uses Resnet18 and BiGRU, and LTCN that uses Resnet18 and Temporal Convolutional Networks. Table 2 presents a comprehensive overview of all the experiments conducted on the 5 speakers. Spk-1 uses deaf-speech and sign language, Spk-2 and Spk-3 use sign language as their primary mode of communication while mouthing words with imperfect lip movements. Table 3 presents additional experiments performed on the dataset of the ALS patient. We initialize our models with the weights of LRwP or LTCN, both of them pretrained on LRW. We observed that using the pretrained weights leads to faster convergence.

Training strategy
Baseline (Exp-cl−r): We begin by training our model directly on the one-shot examples. Since we are the first to perform one-shot lipreading on a personalized vocabulary, we treat this model as our baseline. Table 2 Exp-cl−r presents the performance of the baseline model on each of the 5 speakers. The average accuracy of the speakers at top-1 and top-5 is only 33.0% and 51.6% respectively. The accuracy of the current SOTA lipreading model on the LRW dataset is 88.5% at top-1. This presents us with a huge scope for improvement.  Table 2: Evaluation of our models against each speaker reported in %. All metrics are evaluated on the curated test set made of only real-data. Spk-1 uses a combination of deafspeech and sign language, Spk-2 and Spk-3 use sign language for communication. Patient denote the ALS patient in our study. r, s, and aug indicate real, synthetic, and augmented-real datapoints. cl indicates standard classification while VE is the proposed technique.  Table 3: Accuracy of additional experiments performed on the ALS patient in %. The test dataset used for both, One-shot and Two-shot experiments is the same. FastSpeech2 and GlowTTS are represented by fs tts and glow tts, respectively. All experiments are conducted using LRwP as the backbone on the combined real, synthetic, and augmented real datasets.
Data Augmentation using Synthetic Data (Exp-cl−r+s): We augment the one-shot examples with a potentially unlimited number of synthetic examples. We train our models with varying amounts of synthetic data. We plot a graph of accuracy against the combined synthetic and one-shot examples to determine the optimal amount of synthetic data needed for each speaker against the one-shot examples as shown in the supplementary, Fig. 1. The optimal number of synthetic examples per speaker is reported in Table 1.
As shown in Table 2 Exp-cl−r+s, we observe a significant jump in the accuracy consistently for every speaker with an overall improvement of 9% and 7% at top-1 and 5, respectively on LRwP, and 7% and 12% at top-1 and 5, respectively on LTCN. Although the accuracy improved over the baseline model, we observe that the model overfits on the synthetic data after a few epochs. Since we use only one example per class for the real domain, the model cannot foresee the target (real) variations that it may encounter during testing. We, therefore, try to introduce variations using Variational Encoders.
Variational Encoders (Exp-ve−r+s): To train the network, we use the one-shot examples as the source domain. For the target domain, we combine the real and synthetic dataset. We observed that using only the synthetic data in the source encoder makes the training highly unstable and the network does not converge. Instead, we allow the source encoder to first fit and then improve.
As seen from Table 2 Exp-ve−r+s, Variational Encoders consistently achieves the highest accuracy at top-5 across LRwP and LTCN. There is an overall improvement of ∼ 19% over the baseline and ∼ 12% over Exp-cl−r+s. The improvement at top-1 accuracy is comparatively much lower, ∼ 15% over the baseline and only ∼ 6% over Exp-cl−r+s. We use PCA visualizations to analyze the latent representations learnt by the LRwP based source encoder, as shown in Fig. 5 (right). We observe better separation for each class compared to the separation of Exp-cl−r+s (middle). We also observe better disentanglement between nonhomophenes such as 'bathroom', 'appreciate', and 'coughing' compared to Exp-cl−r+s. For Varitional Encoders we observe that homophenes 'coughing', 'cooking', 'something' are closer together. This suggests that the model trained on Variational Encoders can get confused for homophenes bringing the accuracy at top-1 down and at top-5 higher. For Exp-cl−r+s however, all the classes seem equally apart.
Ad-hoc Data Augmentation (Exp-cl−r+s+aug and Exp-ve−r+s+aug): In addition to the implicit variations introduced by Variational Encoders during the model training, we introduce explicit real-domain variations by augmenting the one-shot dataset. First, we use moviepy [20] library to speed up and speed down the one-shot videos by a factor of 1.2× and 0.8×. The videos in our work are unconstrained, that is, the actual mouthing could be spread across several frames placed temporally anywhere in the video. Thus, we increase the video frames sequence length to 85 and add temporal variations during training by padding the videos with random number (between 0 to 20) of empty frames at the start and end.
We observe improvement in the performance across speakers for both, classification (Exp-cl−r+s+aug) and Varitional Encoders (Exp-ve−r+s+aug). We observe that for Spk-1 and Spk-2, the performance degrades. Upon further analysis, we find that the videos maintain constant pace and have fewer overall variations. Thus, the added variations during training behave as noise driving the performance down. The overall performance improves by 7% and 3% at top-1 on Exp-cl−r+s+aug and Exp-ve−r+s+aug over Exp-cl−r+s and Expve−r+s respectively. We observe the best performance on one-shot setting with these ad-hoc additions for classification. For Variational Encoders, the improvements are less significant, especially at top-5, indicating the technique itself makes up for these ad-hoc additions.
Additional Experiments on the ALS Patient's Dataset: To observe the affect of using different TTS models for generating synthetic data, we perform an ablation by eliminating one TTS model at a time for data generation and compare its performance against the data generated by combining both the TTS models. As shown in Table 3, the performance with both the TTS combined gives us the best performance. This indicates that the variations introduced by different TTS models are important for generalization.
Lastly, we train an additional model with the additional data obtained from the patient's family to evaluate the performance difference between one-shot and two-shot setting. As seen from Table 3, the performance improves by ∼ 7% at top-5 against the best performing one-shot model on both training methodologies. Specifically for Variational Encoders, the performance improves to 71.64% and 89.36% at top-1 and top-5. This suggests that the network learns a better real-domain latent distribution using the extra real examples.
Evaluation on Joe Biden: Although Joe Biden represents speakers without disabilities, he speaks in an American accent, while LRW is composed of British speakers. As can be observed from Table 2 Exp-cl−r, the performance for Joe Biden on the baseline model is similar to the performance for the deaf speakers. A different accent can also be thought of as a different style of speaking. Thus the pretrained model does not directly adapt to Joe Biden. We observe in his case, adding synthetic data leads to significant improvements in the accuracy over the baseline with an average gain of 21% at top-1 and top-5 (Exp-cl−r+s). Variational Encoders fail to bring expected improvements compared to the other speakers. With LRwP, at top-1, the accuracy drops by 3%, and we see a marginal improvement at top-5. We note that the TTS used to generate synthetic data is of an American accent. This suggests that the synthetic data captures the variations of the real domain exceptionally well in his case. Variational Encoders, on the other hand, adds more noise than valuable variations.

Conclusion and Future Work
In this work, we leap from previous lipreading approaches and propose a one-shot personalized lipreading framework to aid patients suffering from ALS or hearing disabilities. Due to the extreme scarcity of personalized data available for a medical patient, we generate synthetic data to augment our training process. We train our network with Variational Encoders, a domain adaptation technique, to bridge the gap between the synthetically generated examples and the available one-shot real examples for each class. Our method proves to be highly effective, and we achieve over 83% top-5 accuracy for the ALS patient. We also report the performance of speaker-specific models trained for multiple speakers with hearing impairment and a speaker with no disability. In the future, we would like to improve the accuracy of our model while also increasing the vocabulary it can handle. We believe our work achieves essential milestones for the lipreading community and can enhance the communication between people dependent on lipreading. It encourages research in the direction of few-shot lipreading that is an important real-world challenge that can have far-reaching applications including and beyond medical lipreading. Although, Variational Encoders has been proposed for few-shot lipreading, its utility can be explored in other areas with a few-shot settings.

Ablation Experiments
In this section, we report ablation experiments to inspect and gain further insights on the different components of our network. Our experimental setup is identical to the setup mentioned in Section 3 of the main paper. For better readability during comparisons, we have copied the results of

Variational Encoders on Only Single Real Examples
As reported in Section 3.1 of the main paper, our network is trained on the combined data of synthetic and one-shot examples. In this section, we remove the synthetic data to see its benefit. Instead of passing synthetic and one-shot examples, we only use one-shot examples as input to both encoders. Since the Variational Encoder itself acts as a data augmentation network, we assume the network to improve over the baseline (Table 1 Exp-cl−r). As shown in Table 1 Exp-ve−r, the performance of the reduced network improves by an average of 6% and 4% at top-1 and top-5 respectively over the baseline. However, the performance is still lower than the rest of the experiments mentioned in the main paper (Table 2 Exp-cl−r+s and Exp-ve−r+s). This suggests that Variational Encoders with just one example cannot surpass methods with additional data augmentation techniques. But, even with only one example, Variational Encoders is capable of bringing significant improvements to the model. Thus, our approach can be a potential data augmentation technique impacting other tasks requiring a low-data setting.

Effect of ∆ dist
As reported in Equation (4) of the main paper, ∆ ve is a combination of ∆ dist that introduces domain invariance by reducing the distance between the feature space of source and target encoder, and ∆ entropy that allows the encoders to generate task-relevant latent embeddings and distributions using the loss of the downstream task. In this section, we remove ∆ dist and observe the behavior of the network using only ∆ entropy . This allows us to determine the effect of introducing domain invariance. Loss of the reduced network is thus given by, As can be seen from Table 1 Exp-ve−r+s−mod, the accuracy of the reduced network drops by 1% and 5% across the speakers at top-1 and top-5 respectively over Exp-ve−r+s. We observe, ∆ entropy plays a substantial role in our network. Generating variations using Variational Encoders lets the downstream classifier see various examples of the real domain making it robust. We observe the accuracy drop at top-5 is higher than top-1. The drop suggests that ∆ dist may introduce domain invariance across related mouth movements (such as homophenes). 2 Need for Variational Encoders

Other Data Augmentation Techniques
In this section we report additional approaches that we tried to augment the one-shot examples. An expert human lipreader can only lipread 40% of a given sentence [? ]. Thus, manually labeling unlabeled videos is challenging. We therefore try to automatically label the unlabeled videos.
Best Match using Syncnet -Syncnet [? ] is an audio-video synchronization model to synchronize mouth movements and a given speech utterance. It generates a confidence score for a given audio and video segment. We exploit this method to find segments of unlabeled videos closely matching the generated TTS audios. We use different confidence thresholds to label the videos and vary the length of video segments based on the audio length.
Pseudo Labeling -Pseudo Labeling [? ? ] is a technique that iteratively labels the unlabeled data in a semi-supervised setting. A model is trained on the labeled data and is used to label the unlabeled data known as pseudo labels. The model is trained further on the pseudo labels. These steps are repeated until the pseudo labels converge. It assumes the first model trained on the labeled data to be accurate so the pseudo labels are close to the actual labels. Pseudo labels can be noisy, but as the training progresses, labels converge to stable and accurate predictions. We first segment the continuous unlabeled videos into smaller word-level video segments. We use lip-landmarks detected by dlib to measure the rate of change of lip-movements and determine the pauses between words. We then segment the videos on these pauses. We then iteratively label these videos using the pseudo labeling technique.
We observed that these methods led to noisy labels. The performance for each model mentioned Table 2 of the main paper dropped by an average of 15% and 7% on adding pseudo labels and syncnet labels respectively.

Test Loss on One-Shot Examples
Training a classifier on one-shot examples restricts the model from seeing many variations of the real domain and thus tends to underfit the real domain. We show this phenomenon in Fig. 2 that plots the test loss of Exp-cl−r+s (of the main paper) and Exp-ve−r+s. In Expcl−r+s, we see loss on the test set flattens and increases after Epoch-12. In Exp-ve−r+s we see the loss keeps decreasing slowly. This suggests that Variational Encoders introduces meaningful variations of the real domain.