s2fieldsofstudy:
Computer Science
Psychology

text:
Early Prediction of Autism Spectrum Disorder by Computational Approaches to fMRI Analysis with Early Learning Technique

The neuro imaging developmental classification studies are undergone with small amount of samples from the brain activity samples. It promises the inspiring complications in high dimensional data analysis. Autism prediction methodologies are based on behavioral function alone previously which provides good precision but repossession will be unfortunate. We address those problems for early prediction of autism with neural development modern techniques and compared with older. Moreover, visualization of brain activities is quite important in neuro imaging. We believe in better visualization and classification of neuro images in early month captures and appended of Mullen Scales of Early Learning (MSEL). Functional magnetic resonance imaging (fMRI) is one of the controlling tools for measuring non-invasively measure brain activity and it provides with good resolution. For high resolution of brain activity, fMRI gives better than electro encephalon graph (EEG). Visualization of brain activity very clearly is first step to recognize the faults of autism. We have taken into the account for predicting in early Autism Spectrum Disorder (ASD) with help of multiple behavioral activities and development measures using machine learning algorithm. The prediction methods are examined with mostly many prediction methods start to examine the neuro imaging with ultra-high risk factors. The prediction of ASD is moderate accuracy in 14 month development measures from multiple time points. In this proposed work, Mullen early prediction is appended for early prediction and it is examined with computational approach to fMRI analysis with adaptive functioning classifier for machine learning algorithm. This proposed algorithm provides improved version of classification in machine languages with MSEL and high accuracy with conservative methods.

INTRODUCTION
One of the complex organs in the human body is brain which connects with almost all the organs through many veins and nerves (1). Mainly it divides with some parts as follows, frontal lobe, parietal lobe, occipital lobe, temporal lobe and cerebellum. It is shown in figure 1 in below.

Figure 1 Human Brain with primary parts division
Dissociation of neural in the brain and behavior development leads to inability to the human is called autism. The autism symptoms may vary child to child. But the symptoms of autism can be detected after 12 to 18 months. It is very hard to detect before this period of children (2). The autism person has problem with verbal and nonverbal communication; hence there is absence of their eye contact with us. The autism brain contains face processing networks is very weaker. The treatment will be more effective and it's curable sometimes if autism is detected as early as possible. The over connectivity of the nerves in brain says autistic function as shows in figure 2. Recently, clinical laboratory categories an identifying in early the autism spectrum disorder is a top priority. Infant at high risk in an early age stage, the brain will be in developing stage before core autism signs in behavioral or in the biological mark have emerged. There is possible to alter the early intervention strategies (3,4,5).
In a brain, autism neural function is affected easily in cerebral cortex, amygdala, basal gandla, corpus callosum and cerebellum. Also we can estimate in those parts with mullen scale parameters. Usually, amygdala region in brain is responsible for aggressive behavior. Also hippocampus contains the nerves system that remembering recent events and information. Mainly, those areas are started to affect for autistic parameters. We believed this logic and bio marked in those places for early prediction of autism behavior. We succeed it too.

RELATED WORKS
Vaishali R et al investigate the optimum behavior sets for autism which consists of 21 features getting from UCI machine learning. It says that possible to achieve better accuracy with less number of features (6). Fadi thabatah et al has come across machine learning method with ASD screening model. They highlight the problems that arise in DSM manuals and solve (7). Mohamed shanavas et at propose the research idea to detect the autism problems and stages of autism with a help of SVM in the neural networks. Also they show students behavior and their social interaction (8). J.A. Kosmickil et al study on evaluation of clinical assessment of ASD with different machine learning approach. They are evaluating in their paper with those 28 behaviors parameters from module, and calculated ASD risk with good accuracy (9). Suman raj et al investigated detection of ASD using different techniques of machine learning approaches. It clinically proves high accuracy than exist methods (3). Li B, et al proposes the imitation method in machine learning algorithm. Here discriminative test conditions are investigated with kinetic parameters on the adolescent dataset. Also, consideration of kinetic parameters for detection of autism, the sensitivity rate is high compared to others (10). Arbabshirani et al studies focus that in order to predict individual outcome of ASD, uses behavioral measure alone. They emphasis was depends on theory that by identifying infants at risk at early age when brain is growing up. The linear regression method is very efficient when linear progression between input and output (11).
Most of the works are done and detected ASD and it gives different accuracy and sensitivity levels. These works miscarries in early prediction of ASD in all. The prediction algorithm is Mullen scales of early learning and fMRI analysis and FC to detect autistic behavior in child. In this work, comparing linear regression with Mullen scales of early learning prediction and approaches for scaling up advanced fMRI analysis for model analysis.

Motivation
The early diagnosis or intervention of ASD is quite challenging task in clinical field. Generally, "autism is a lifelong developmental disability". It is degenerative an irreversible process during the young age or early stage of disorder and when their brain plasticity is much more that easier to prevent ASD. So the detection of ASD is not providing perfect solution for their life. With the intention of avert, naming the children as "autistic" in their community and for the rest of their lives. This reason is the motivation of this research article. Mostly many researchers are showing decrease in brain connectivity between default node and network node in brain bio marks.

ORGANIZATION OF THE RESEARCH
This research article covers introduction, related work, and motivation, theoretical analyses for early prediction for autism, results and discussion, conclusion in a row.

THEORETICAL ANALYSES
Functional magnetic resonance imaging (fMRI) is one of the standard tools is to characterize the brain function in health and disease (12,13,14,15). We can understand that how ASD exhibits functionally in the successive interval. Using fMRI data, we can early predict autism behavior function with the help of detecting sensitive and particular biomarkers for ASD (4,5). There are lots of studies to understand and predict the specific biomarkers for ASD due to local functional or operational brain abnormalities in ASD (16). Our research suggests that there can be found relevant markers for ASD whereas irregularities in the brain function connectivity with the help of functional connectivity. Figure 3 shows design of our proposed algorithm that adaptive functioning classifier will done after selected measuring samples and features. Those features are clustered and predicted with Mullen scale early learning. The scale function is measuring from cognitive functioning and provides score for overall intelligence for infants and preschool-age children. The overall intelligence is estimating by this composite score.  The fMRI connectivity is the useful biomarkers for ASD though in the complex connectivity relation of different parts of the brain. Also fMRI connectivity involves in good very complicated pattern of changes in biomarks system in brain (17,18,19). Functional MRI data collected from ABIDE data set. This data set needs preprocessing procedure to analysis and classification purpose due to some variance of signal due to noise environment and physiological disturbance like sudden respiration, hand motions etc. Sometimes non neural variability will be occurred during the process (20,21). To rectify those problems, blind deconvolution method can be carried out in this process. Initially, in order to make mean zero normalization of every ROI, preprocess should be done. In every pair of ROIs, FC will be computed and it gives us connectivity matrix to perform graphs for each subject. In order to differentiate between two groups of features these ROI approach is used. The Functional Connectivity (FC) analyses used to synchronize ROIs. The complex network constructed with graphs from FC. The data is divided into group or individual by linear model for statistical analyses (22).

Mullen scale of early learning
This complex network measures are used to identify the node or vertex for machine learning classification. This identification has contained that inclusive of node degrees, node strengths, clustering co efficients, centrality of node and edge both as well (23,36). The centrality of edges provides the number of all shortest paths between the given nodes. These nodes are divides further in the network and named as modules. The form of degree centrality can be limited the module degree. Inside the module connection of individual nodes are measured by harmonizing participation coefficient (HPC). This CPC will facilitate the functional segregation and integration to do clustering for classification in the complex network (24,36). This clustering of brain region ROIs will be measured in the given networks. There is a calculation of the average clustering co-efficient of the node in the network (25,26).
The clustering construction is necessary to compute in the complex network for optimization purpose. With the intention of maximize the intra ROI links and reduces the connection between localize nodes; the clustering construction is being constructed. There are classified into low and high degree vertices. Our proposed technique is clustering the coefficients in the entire graph with the help of transforming regularization (TR) method. This combined version of clustering and segregation of inter and intra community nodes called integration. The calculation of average distance between the vertices is defined as;

Where N is Number of vertex and every vertex ∈ ( )
In a complex network, the calculation of correlation coefficient between the nodes in the community is necessary for measuring the resilience of it. In order to measure the assort coefficient function for the network, the resilience factor calculation is essential. In our proposed algorithm, the input data is splitting into two which is known as training and testing data. The training data can be clustering the features F in to "n" clusters by k means algorithm. There is a computation of performance of every cluster. After computation of performance of every cluster, remove 20% of clusters with assigned scores (27). The remaining features are shifted into single community region. This will be evaluated classification accuracy with the testing data. This is the process known as SVM classification (21,28,29,30).

Importance of higher classification accuracy
For early prediction, the classification and computational approach should be effective. Due to getting the higher classification accuracy, our functional MRI data should be rich for cognitive neuroscience. Many modified progression will be considered to yield for early diagnosis from the conservative methods. The process modifies by often involving concrete and improves the visual perception. Then manipulate the model localizing model variables in the brain; after locate the variables appropriately and it is estimate parameters independently of behaviors like errors and loss (31,32,33). This behavior is red and computed decision variables step to final stage. It provides better result compared to the fMRI analysis. The fMRI data will be combined in group by mean value at each vertex by the cross validation (34). Additionally, finding the most important voxels for the classifier, the evolutionary algorithm is used in which different voxel clusters are used to train and evaluate the classifier (35).

RESULTS AND DISCUSSIONS
In Classifiers, our proposed study shows great performance in accuracy level which is showed in figure 4 (a) & (b). The vertical axis in (a) figure is that accuracy of our proposed study. Moreover, the accuracy is measured with important features shown in the graph.

(a) (b) Figure 4 (a) & (b) Our proposed method accuracy between base classifiers & important features
The linear regression based algorithms are predicted in coronal slice of scanned image as "autistic" function after computed the feature of thicker cortex and more folds. Even though absence of thicker cortex in frontal lobe of the brain, our proposed algorithm identifying autistics behavior in the image which shown in the figure 5b.

Figure 5 (a) & (b) Presence of Neural wrinkles in top sliced view of brain
The autism brain image is having more folds and thicker cortex in its neuro function which is shown in figure  6 (a). The figure 6 (b) tells that our algorithm predicted well autistic function at early stage while other algorithm is unsuccessful.

(a) Predicted region with normal neuro functioning brain (b) Autism neuro functioning brain
Due to increase the parameters to predict the autism functioning in the brain nervous which will be responsible for the early prediction of autism. Our method is providing good effects in higher level aspects of cognition as early prediction during decision, valuation, control (DVC) and social interactions. The early prediction results highlight that MSEL is useful technique to assist medical physician to predict autism in a high accuracy. This tool is considering the main parameters receptive and expressive language, visual and fine motor impairments during early stage of children. Our proposed method appears to be sensitive to behavior activities. In the linear regression method, deviates output coefficients when inputs increase more complexity. Our integrated SVM algorithm is suitable and effective in high dimensional space. The sensitivity of the linear regression prediction gradually increases with our proposed study for early prediction which is shown in figure 8. The horizontal axis of figure 8 is from 10 th month of child. Our proposed study has proved good sensitive in predicting autism in early stages and shown in figure 8.

CONCLUSION
The proposed study is aimed well with combination of adaptive functioning classifier and early learning techniques and works well. Since our process is based on high dimensional spaces, it shows the effective work. Also here appropriate kernel function will be hard to choose. The computational approach will address the complexity of theoretical models with less number of empirical data. Our proposed method considers this computational approach that dimension of variation at every voxel. The space will expands widely with the voxel dimension variation. This challenging problem solved by our computational approach which may improve the ability to predict the cognitive signals in earlier. Since we are having less data set our proposed algorithm was suited for the process. In case of analysis the large data; our algorithm has to revise. We were preprocessed our algorithm to perform and handle easy less amount of noisy data set. Our proposed technique is improved process for number of features for each data point exceeds the number of training data samples. The execution time for advanced scaling up fMRI analysis method is greater than model based analysis without Mullen scale of early learning. The revised algorithm has to implement.

ACKNOWLEDGEMENT
We are most grateful to Mr. Mussie Estifanus, Department of Electrical & Electronics Engineering, Mainefhi College of Engineering, for his assistance to collect the data and complete this research article.


Electroencephalography Artifact Removal using Optimized Radial Basis Function Neural Networks

Electroencephalography (EEG) is a major clinical tool to diagnose, monitor and manage neurological disorders which is mostly affected by artifacts. Given the importance and the need for an automated method to remove artifacts, in this paper some intelligent automated methods are proposed which are composed of three main parts as extraction of effective input, filtering and filter optimization. Wavelet transform is utilized to extract the effective input, and the wavelet approximation coefficients are used as an effective input signal. In addition, Radial Basis Function Neural Network (RBFNN) has been used for filtering. The appropriate number of RBFs has been selected using extensive simulations, and the optimal value of spread parameter has been achieved by Bees algorithm (BA). Finally, the proposed artifact removal schemes have been evaluated on some real contaminated EEG signals in Mashad Ghaem hospital database. The results show that the proposed artifact removal schemes are able to effectively remove artifacts from EEG signals with little underlying brain signal distortion.

INTRODUCTION
1EEG is a major tool which is used to record the electrical activity on human brain. It is recorded using 10-20 electrode placement system on the scalp of a person with low spatial and high temporal resolution [1]. EEG is mainly affected by different artifacts which reduce its clinical usefulness. There exist two types of artifacts, namely, biological and external, where the former is mostly caused by electromyogram (EMG) or muscular activities, Electrooculogram (EOG) or ocular activities and electrocardiogram (ECG) or cardiac activities, however, the latter is mostly caused due to technical factors as electrode leads and lineinterference.
Most EEG artifact processing algorithms are developed to reduce biological artifacts, however, the effect of artifacts with technical origin is greatly reduced by improving technology and paying extra attention to electrode attachments to body surface [2]. Artifact removal is the most appropriate approach introduced to identify and remove artifacts from brain signals and keep the related neurological phenomenon intact. There exist different schemes to remove EEG signal artifacts in clinical studies. Examples include linear filtering, linear combination and regression, Blind Source Separation (BSS), Principal Component Analysis (PCA), Wavelet Transform (WT), regressionbased techniques, adaptive filters and neural networks.
Linear filtering is used to remove artifacts which are located in certain frequency bands and do not overlap with those of the neurological phenomena of interest [3]. In this regard, high-pass and low-pass filtering is utilized to remove EOG and EMG artifacts, respectively. In early clinical studies, linear filtering was commonly used to remove EEG signal artifacts [4]. Linear combination of EOG signal and EOGcontaminated EEG signal can be considered as one of the most common techniques to remove ocular artifacts from EEG signals [5]. BSS techniques are used to separate the EEG signals from their components. The aforementioned techniques are used to identify the components which are attributed to artifacts. Moreover, the EEG signal is reconstructed without them [6].
Recently, great attention has been paid to Independent Component Analysis (ICA) which can be considered as one of the most widely used techniques among BSS techniques. In ICA, mixtures of independent source signals are blindly separated and the components are forced to be independent. In [7], eye-blink artifacts are automatically removed from the EEG data. In this scheme, the EEG raw data is decomposed into independent components by ICA followed by Peak Detection Algorithm of Independent Component which is suggested to identify eye-blink artifact components. In [8], ICA and Empirical Mode Decomposition (EMD) are combined where ICA is used to obtain the independent components, however, EMD is applied to remove the ocular artifacts with larger amplitude in independent components. In [9], a hybrid approach is presented which is based on ICA and Adaptive Noise Cancellation (ANC). It is noted that ICA decomposition is utilized to extract the artifact source signal. 2 In PCA, data is transformed to a new coordinate system using the eigenvectors of signal covariance matrix. Afterwards, signal components are extracted by projecting the signal onto the eigenvectors. In [10], a wavelet-based threshold scheme and a PCA based adaptive threshold scheme are used to remove the ocular artifacts from EEG signals. In [11], ICA and PCA are applied to intracranial recordings, and three methods are proposed to remove the reference signal and line noise.
Wavelet Transform is a time-frequency analysis method which is suitable for non-stationary signals as EEG. It is based on different statistical characteristics of signal and noise [12]. In [13], a wavelet enhanced ICA method (WICA) is presented which uses wavelet thresholding to de-mixed independent components instead of the observed raw EEG. In [14], the wavelet-ICA and Support Vector Machine (SVM) are used to remove target artifacts. In this paper, SVM is used to identify artifactual components which are separated by wavelet-ICA. In [15], an automatic removal method is presented for Ocular Artifacts (OAs) to overcome EEG data interference. In this method, the Discrete Wavelet Transform (DWT) is applied to every recorded signal to achieve multiple scale coefficients.
There exist some regression-based methods including time and frequency domain [16][17]. These approaches involve calibration tests to determine the propagation coefficient between EOG channel and each EEG channel. In this case, eye blink artifact is removed by subtracting each EEG channel from the separately recorded EOG. It is worth nothing that the EOG signal which is subtracted from the recorded EEG signal contains some EEG information which introduces loss in the desired information [16][17].
Recently, adaptive filters have achieved widespread applications in different areas. Because of the time varying nature of signals arising from the human body, adaptive filtering is suggested for EEG artifact removal and many new approaches are presented to design adaptive nonlinear filters for noise cancellation [18]. In [19], ocular artifacts are removed from EEGs using DWT and ANC. In [20], an adaptive filtering technique is presented to remove ocular artifacts from EEG recordings using the forgetting factor and the filter length parameters. In [21], the combination of EEG decomposition with Adaptive Filters (AFs) is addressed to improve the overall denoising process.
Neural networks are an attractive approach in adaptive signal processing [22]. In [23], ocular artifacts are removed using JADE Algorithm and neural networks. In this regard, independent components are gained using JADE, however, neural networks are used to classify the obtained independent components. Also, there exist two neural network schemes to learn classification rules from the EEG data. In [24], a combination of adaptive noise cancellation and adaptive signal enhancement is presented in a single recurrent neural network for the adaptive removal of ocular artifacts from EEG.
In this paper, an intelligent method is proposed which is based on RBFNN. It can remove artifacts from EEG signals and is composed of three main parts as extraction of effective input, filtering and filter optimization. Also, WT is used to extract the effective input, and wavelet approximation coefficients are utilized as effective input signal.
The rest of the paper is organized as follows: Section 2 briefly presents the related preliminaries, Section 3 details the proposed method based on optimized RBFNNs with wavelet approximation coefficients as input, afterwards in section 4, the performance of our approach is assessed by a simulation set up, thereafter, section 5 provides the conclusion of this paper.

PRELIMINARIES
In this section, concepts on RBFNNs and WT are presented.

Concepts on RBFNNs
In this subsection, the structure of a neural network is presented [25]. In total, there are three layers in RBFNNs, i.e. input layer, hidden layer and output layer.

135
The importance of RBFNNs is due to the following main features: 1. Universal Approximation Property 2. Learning Capability The aforementioned features make the weights and adjustable parameters to constantly update and improve the total performance. The output of an RBFNN with n input and m neurons is as follows [25]: In the next subsection, WT is briefly presented.

Concepts on WT
EEG signals can be appropriately analyzed by reducing the signal noise before any further analysis [26]. An efficient method for de-noising the EEG signal is WT which is used to solve the resolution problem by using multi-resolution analysis. It is a tool which cuts up data, functions or operators into different frequency components, and then studies each component with a resolution which is matched to the corresponding scale [27].
There exist two types of WT, i.e. Continues Wavelet Transform (CWT) and Discrete Wavelet Transform (DWT). CWT is as follows [27]: Where, () xt is the original signal,  is the complex conjugation and a and b are the scaling factors. Also, , () ab t   is achieved by scaling the wavelet at time b and scale a . In CWT, the inner products of analyzing function and the original CWT signal is used and the similarity between these two functions is measured by integration. In CWT, it is assumed that a and b are continuously changing. The CWT drawback is that the wavelet coefficients shall be calculated for every possible scale which ends in a large amount of data. The aforementioned drawback can be overcome by using DWT.
DWT is another type of WT which uses the mutually orthogonal set of wavelets defined by choosing the scaling and translation parameters ( a and b ). DWT is as follows [27]: Remark1-In the current study, RBFNN is utilized because of its stability, good generalization ability, its easy design, online learning ability, and good input noise tolerance. Also, WT is used as an efficient method for de-noising the EEG signal.

THE PROPOSED METHOD BASED ON OPTIMIZED RBFNN WITH WAVELET APPROXIMATION COEFFICIENTS AS INPUT
In this paper, an intelligent method based on an RBFNN is presented to remove artifacts from EEG. The proposed scheme is composed of three main parts as extraction of effective input, filtering and filter optimization. WT is utilized to extract the effective input and wavelet approximation coefficients are used as an effective input signal. Afterwards, effective inputs are filtered for complete artifact removal. By applying WT, a two-level filtering system is achieved where the first level is wavelet approximation coefficients which remove some noise signals and the second level is the neural network which removes the remaining ones. RBFNNs have high potential in estimation and filtering. The number of RBFs and the spread parameter highly affect the performance of networks.
The Bees algorithm [28] is an optimization algorithm which is introduced in recent years and its ability to solve complex nonlinear problems has been proven. It is inspired by the natural foraging behavior of honey bees to find the optimal solution. It is a population-based search algorithm which mimics the food foraging behavior of swarms of honey bees. The basic version of BA performs a sort of neighborhood search which is combined with random search and can be used for both functional and combinatorial optimization. Bees algorithm shows remarkable robustness. It is able to converge to the maximum or minimum value without being trapped at local optima and it usually outperforms other methods in terms of optimization speed and accuracy [28]. In this paper, BA is used to find the optimal spread value. The number of RBFs is chosen the same as the training data, and the neural network is trained on the training data, so it may have poor performance on the test data. A solution for this problem is to construct an RBFNN with a single RBF. If an optimal solution is extracted from this RBFNN, the network is built. Otherwise, the number of RBFs is added by one and the network is tested again. This process is continued until the final optimal solution is achieved. The Flowchart of the proposed method based on optimized RBFNN with wavelet approximation coefficients as input is given in Fig. 1. The algorithm of our scheme is as follows: Step 1. Getting the EEG.
Step 2. Applying WT to EEG taken in step1.
Step 3. Constructing a neural network with an RBF.
Step 4. Using BA to find the optimal spread value for the constructed network regarding the following steps:  Setting BA parameters. Setting BA parameters as the total number of bees, the number of elite bees and the number of scout bees.
 Creating the initial population of bees.  Calculating eligibility. The mean square error (MSE) is used as eligibility criterion.
 Local searching. Scout bees must search around bees in elite groups one and two, until the final optimal solution is extracted.
 Global searching. Nonelite bees are distributed in the whole search area to prevent the optimization algorithm from being trapped in the local minimum.
 Stop criterion. If the stop criterion is reached, the algorithm stops and the network is built. Otherwise, go to step 4-C. In this algorithm, the stop criterion is chosen as the maximum number of iterations.
Step 5. If the network has an optimal performance, stop searching and the algorithm is finished. Otherwise, add one to the number of RBFs and go to step 4.
Continue this procedure until the number of RBFs becomes the same as the number of training data.

SIMULATION RESULTS
The proposed artifact removal schemes are evaluated on some real contaminated EEG signals in Mashad Ghaem hospital database with a sampling frequency of 256.There exist 3 artifact-free EEG data signal (baseline) and 355 artifact contaminating signal corresponding to 3 people. Also, three different artifacts are produced as eye blink, teeth grinding and swallowing. For each artifact, 61time trial is recorded and the recording time is considered 10 seconds.
In our proposed methods, the RBFNN is used as a filter. In this scheme, artifact-contaminated signals and baseline signals are considered as neural network input and output, respectively, and the MSE criterion is used as a performance criterion for neural network. In the simulations, 355 artifact-contaminated signals are used. 70% of data is used to train the neural network while the remaining data is utilized for the testing purpose.
In the following, two tests without and with wavelet approximation coefficients as input are performed. In each test, our proposed schemes are evaluated based on four types of RBFNNs: A. The proposed scheme based on precise nonoptimized RBFNN B. The proposed scheme based on precise optimized RBFNN C. The proposed scheme based on non-optimized RBFNN D. The proposed scheme based on optimized RBFNN Test1. Using the original data (without wavelet approximation coefficients as input)

A. The proposed scheme based on precise nonoptimized RBFNN
In this structure, the number of RBFs is chosen the same as the training data (250 RBFs are considered in this experiment). Also, the spread parameter value is obtained by trial and error. The results obtained using From Table 1, the impact of spread value on system performance becomes obvious and it can be concluded that it is essential to find a proper spread value. Fig. 2 shows the proposed scheme performance based on precise non-optimized RBFNN. It is clear that the performance is satisfactory.

B. The proposed scheme based on precise optimized RBFNN
In this structure, the number of RBFs is chosen the same as the training data (250 RBFs are considered in this experiment). Also, the spread value is obtained using BA. The spread parameter highly impacts the network performance. Table 2 indicates the BA parameters. The results obtained using this neural network are given in Table 3. Comparing Tables 3 and 1, the impact of spread value can be realized. Using BA, the optimal spread 3.11 is obtained which leads to the breakdown of MSE from 0.055 to 0.038 in precise optimized RBFNN. Finally, the performance of the proposed scheme based on precise optimized RBFNN is shown in Fig. 3.

C. The Proposed scheme based on non-optimized RBFNN
In this structure, the number of RBFs is optional and is set as an arbitrary value between1 to 250, where 250 is considered as the total number of the training 138 data. The best value for the number of RBFs is chosen by testing all possible values. Furthermore, for any network with a given number of RBFs, a good spread value can lead to a satisfactory performance. So, a neural network with an RBF is constructed and the spread value which leads to the lowest MSE for the network is achieved using trial and error. If the result is not satisfactory, the number of RBFs is added by one and the network is tested again. It is recommended to increase the number of RBFs to get the best possible result. The best spread value is achieved by trial and error. The flowchart of the proposed method based on non-optimized RBFNN is presented in Fig. 4 and the results obtained using this neural network are given in Table 4.   Tables 1, 3 and 4, the impact of choosing a suitable number of RBFs arbitrarily between 1 to 250 can be realized which leads to the breakdown of MSE from 0.055 and 0.038 in Tables 1  and 3, respectively, to 0.031 in Table 4. In Fig. 5, the proposed scheme performance is shown based on nonoptimized RBFNN.

D. The proposed scheme based on optimized RBFNN
In this structure, the number of RBFs is optional and is set as an arbitrary value between 1 to 250, where 250 is considered as the total number of the training data. Furthermore, for any network with a given number of RBFs, a good spread value can lead to a high performance. So, BA is used to find the optimal spread value in each step which enhances the neural network performance. The results obtained using this neural network are given in Table 5. Comparing Tables 5, 4, 3 and 1, the impact of choosing a suitable number of RBFs arbitrarily between 1 to 250 and using BA to find the optimal spread value can be realized which leads to the  Tables 1, 3 and 4, respectively, to 0.01 in Table 5. Fig.  6 shows the proposed scheme performance based on optimized RBFNN.

Test2-Using wavelet approximation coefficients as input
In this experiment, the effect of wavelet approximation coefficients as the input of neural network is investigated. Similar to the previous test, the proposed schemes based on four types of RBFNNs are evaluated using wavelet approximation coefficients. Exhausted simulations are performed and wavelet approximation coefficients for contaminated signals with different artifacts as eye blink, teeth grinding and swallowing are achieved, however, the wavelet approximation coefficients for only two contaminated signals with eye blink and swallowing artifact are presented in Fig. 7 for simplicity. WT acts as a filter for the original signal and eliminates artifacts. In this scheme, wavelet approximation coefficients are used as an effective neural network input. Fig. 7. a) Two contaminated signals with eye blink and swallowing artifact b) Wavelet approximation coefficients for the two signals at level five decomposition.
In Fig. 7-a, two artifact-contaminated signals are shown. It is clear that these two signals interfere with each other for many times because of artifacts. Using neural network learning algorithm, the main signal is achieved after artifact removal. However, because of the interference caused by artifacts, the incorrect training of neural network is possible which results in insufficient artifact removal. In Fig. 7-b, wavelet approximation coefficients at level five decomposition is shown for the two signals. Comparing Figs. 7-a and 7-b, it is clear that the interaction rate in Fig. 7-b is lower than Fig.7-a. It is noted that the higher level of decomposition does not mean better neural network performance. However, increasing the level of decomposition, renders more signal information to be disappeared and from a certain level, the neural network performance is degraded. Also, Haar wavelet is used in the simulations.

A. The proposed scheme based on precise nonoptimized RBFNN with wavelet approximation coefficients as input
In this structure, the number of RBFs is chosen the same as the training data (250 RBFs are considered in this experiment). In addition, the spread value is obtained by trial and error. Table 6 shows the results obtained using this neural network.  Table 6 shows that the best result is achieved at level one decomposition. Fig. 8 shows the proposed scheme performance based on precise non-optimized RBFNN with wavelet approximation coefficients as input.
From Table 6 and Fig. 8, it is clear that the RBFNN with wavelet approximation coefficients as input, proper number of RBFs and proper spread value at level one decomposition, renders a satisfactory performance. Moreover, comparing Fig. 8 with Fig.2, it can be realized that error is reduced and the neural network performance is improved.

B. The proposed scheme based on precise optimized RBFNN with wavelet approximation coefficients as input
In this structure, the number of RBFs is chosen the same as the training data (250 RBFs are considered in this experiment). In addition, the spread value is obtained using BA. Table 2 indicates the BA parameters. The results obtained using this neural network are given in Table 7.  Table 7, it is clear that the RBFNN with wavelet approximation coefficients as input, proper number of RBFs and proper spread value at level one decomposition, has better performance comparing with other levels of decomposition. Comparing Tables 7 and  6, the impact of spread value can be realized. Using BA, the optimal spread value is 2.93 which ends to the breakdown of MSE from 0.043 to 0.029 at the first level of decomposition. Fig. 9 shows the proposed scheme performance based on precise optimized RBFNN with wavelet approximation coefficients as input. Comparing Table 7 and Fig. 9 with Table 6 and Fig.  8, it can be realized that the former shows better performance comparing with the latter, since the spread value is achieved using BA in the former, however it is achieved by trial and error in the latter. Also, comparing Fig. 9 with Fig. 3, it can be realized that the error is reduced and the neural network performance is improved which is due to the use of wavelet approximation coefficients as input. In addition, the proposed scheme based on precise optimized RBFNN with wavelet approximation coefficients as input renders better performance comparing with the previous methods proposed in this paper.

C. The proposed scheme based on Nonoptimized RBFNN with wavelet approximation coefficients as input
In this structure, the number of RBFs is optional and is set as an arbitrary value between 1 to 250, where 250 is considered as the total number of the training data. In order to find the best value for the number of RBFs, all possible values are tested and the best value is selected. Furthermore, for any network with a given number of RBFs, a good spread value ends in a satisfactory performance. So, a neural network with an RBF is constructed and the spread value which leads to the lowest MSE for the network is achieved using trial and error. If the result is not satisfactory, the number of RBFs is added by one and the network is tested again. It is recommended to increase the number of RBFs to get the best possible result. Also, the best spread value is achieved by trial and error. Table 8 shows the results obtained using non-optimized RBFNN and using wavelet approximation coefficients as input.  Tables 8, 7 and 6, the impact of choosing a suitable number of RBFs arbitrarily between 1 to 250 can be realized which leads to the breakdown of MSE from 0.043 and 0.029 in Tables 6 and 7, respectively, to 0.009 at the first level of decomposition in Table 8. Fig. 10 shows the performance of the proposed scheme based on nonoptimized RBFNN with wavelet approximation coefficients as input.  Table 8 and Fig. 10 with Table 7 and Fig. 9 and Table 6 and Fig. 8, it can be realized that the former shows better performance comparing with the latter which is due to the impact of choosing a suitable number of RBFs arbitrarily between 1 to 250. Also comparing Fig. 10 with Fig. 5, it can be realized that the error is reduced and the neural network performance is improved which is due to the use of wavelet approximation coefficients as input. Also, the proposed scheme based on non-optimized RBFNN with wavelet approximation coefficients as input renders better performance comparing with the previous methods proposed in this paper.

D. The proposed scheme based on Optimized RBFNN with wavelet approximation coefficients as input
In this structure, the number of RBFs is optional and is set as an arbitrary value between 1 to 250, where 250 is considered as the total number of the training data. Furthermore, for any network with a given number of RBFs, a good spread value can lead to high performance. So, BA is used to find the optimal spread value in each step which enhances the neural network performance. The flowchart of the proposed method based on optimized RBFNN with wavelet approximation coefficients as input is presented in Fig.  1. Table 9 shows the results obtained using the proposed method based on optimized RBFNN with wavelet approximation coefficients as input.  7 and 6, the impact of choosing a suitable number of RBFs arbitrarily between 1 to 250 and using BA to find the optimal spread value can be realized which leads to the breakdown of MSE from 0.043, 0.029 and 0.009 in Tables 6, 7 and 8, respectively, to 0.001 at the first level of decomposition in Table 9. Fig. 11 shows the proposed method performance based on optimized RBFNN with wavelet approximation coefficients as input. Fig.11. Performance of the proposed method based on optimized RBFNN with wavelet approximation coefficients as input a) Comparison of the wavelet approximation coefficients of original signal and neural network prediction b) The neural network error.
Comparing Table 9 and Fig.11 with Table 8 and Fig. 10, Table 7 and Fig. 9 and Table 6 and Fig. 8, it can be realized that the former shows better performance comparing with the latter which is due to the impact of choosing a suitable number of RBFs arbitrarily between 1 to 250 and using BA to find the optimal spread value. Comparing Fig. 11 with Fig. 6, it can be realized that the error is reduced and the neural network performance is improved which is due to the use of wavelet approximation coefficients as input. Also, the proposed scheme based on optimized RBFNN with wavelet approximation coefficients as input renders better performance in comparison with the previous methods proposed in this paper.

Performance evaluation of various wavelet types
In this subsection, the proposed method is evaluated using various wavelet types. The results with various wavelet types at different decomposition levels are presented in Table 10. Table 10 shows that Haar wavelet at first level of decomposition performs better than other wavelet types.

CONCLUSION
In this study, novel artifact removal schemes based on RBFNNs are established. The proposed schemes are consist of three main parts as the neural network, the optimization and WT. In the first step, precise nonoptimized RBFNN is used to remove artifacts where the number of RBFs is equal to the number of training data and the spread value is obtained by trial and error. With this method, the neural network performance is satisfactory and the artifacts are almost removed. In the next step, BA is utilized to find the optimal spread value of precise RBFNN to enhance the performance. The simulation results show that using the optimization algorithms could significantly raise the performance of neural network. Afterwards, the number of RBFs are determined by trial and error to achieve better performance and less mean square error. Simulation results show that determining the proper number of RBFNNs and the use of BA can enhance the performance of neural networks. Next, WT is used to improve the RBFNN performance and reduce the computing time. Using wavelet approximation coefficients could significantly raise the performance of neural network. The impact of using wavelet approximation coefficients is shown by comparing the results with previous approaches where the original data is utilized as the neural network input. Finally, the optimized RBFNN with proper RBFs, optimal spread value and appropriate approximation wavelet input coefficients achieves the least mean square error among all schemes proposed in this paper.


extfieldsofstudy:
Computer Science
Mathematics

text:
Capacity of Continuous Channels with Memory via Directed Information Neural Estimator

Calculating the capacity (with or without feedback) of channels with memory and continuous alphabets is a challenging task. It requires optimizing the directed information rate over all channel input distributions. The objective is a multi-letter expression, whose analytic solution is only known for a few specific cases. When no analytic solution is present or the channel model is unknown, there is no unified framework for calculating or even approximating capacity. This work proposes a novel capacity estimation algorithm that treats the channel as a `black-box', both when feedback is or is not present. The algorithm has two main ingredients: (i) a neural distribution transformer (NDT) model that shapes a noise variable into the channel input distribution, which we are able to sample, and (ii) the directed information neural estimator (DINE) that estimates the communication rate of the current NDT model. These models are trained by an alternating maximization procedure to both estimate the channel capacity and obtain an NDT for the optimal input distribution. The method is demonstrated on the moving average additive Gaussian noise channel, where it is shown that both the capacity and feedback capacity are estimated without knowledge of the channel transition kernel. The proposed estimation framework opens the door to a myriad of capacity approximation results for continuous alphabet channels that were inaccessible until now.

Abstract-Calculating the capacity (with or without feedback) of channels with memory and continuous alphabets is a challenging task. It requires optimizing the directed information rate over all channel input distributions. The objective is a multiletter expression, whose analytic solution is only known for a few specific cases. When no analytic solution is present or the channel model is unknown, there is no unified framework for calculating or even approximating capacity. This work proposes a novel capacity estimation algorithm that treats the channel as a 'black-box', both when feedback is or is not present. The algorithm has two main ingredients: (i) a neural distribution transformer (NDT) model that shapes a noise variable into the channel input distribution, which we are able to sample, and (ii) the directed information neural estimator (DINE) that estimates the communication rate of the current NDT model. These models are trained by an alternating maximization procedure to both estimate the channel capacity and obtain an NDT for the optimal input distribution. The method is demonstrated on the moving average additive Gaussian noise channel, where it is shown that both the capacity and feedback capacity are estimated without knowledge of the channel transition kernel. The proposed estimation framework opens the door to a myriad of capacity approximation results for continuous alphabet channels that were inaccessible until now.

I. INTRODUCTION
Many discrete-time continuous-alphabet communication channels involve correlated noise or inter-symbol interference (ISI). Two predominant communication scenarios over such channels are when feedback from the receiver back to the transmitter is or is not present. The fundamental rates of reliable communication over such channels are, respectively, the feedback (FB) and feedforward (FF) capacity. Starting from the latter, the FF capacity of an n-fold point-to-point channel P Y n |X n , denoted C FF , is given by [1] In the presence of feedback, the FB capacity C FB is [17] C FB = lim n→∞ sup P X n Y n−1 where, is the directed information (DI) from the input sequence X n to the output Y n [8], and P X n Y n−1 := n i=1 P Xi|X i−1 Y i−1 is the distribution of X n causally-conditioned on Y n−1 (see [21], [24] for further details). Built on (3), for stationary processes, the DI rate is defined as As proved in [8], when feedback is not present, the optimization problem (2) performed over the marginals P X n is equivalent to the optimization in (1). This casts DI as a unifying information measure for representing both FF and FB capacities.
Computing C FF and C FB requires solving a multi-letter optimization problem. Closed form solutions to this challenging task are known only in several special cases. A common example for C FF is the Gaussian channel with memory [14] and the ISI Gaussian channel [15]. There are no known extensions of these solutions to the non-Gaussian case. For C FB , a solution for the 1st order moving average additive Gaussian noise (MA(1)-AGN) channel was found [12]. Another closed form characterization is available for auto-regressive moving-average (ARMA) AGN channels [11]. To the best of our knowledge, these are the only two nontrivial examples of continuous channels with memory whose FB capacity is known in closed form. Furthermore, when the channel model is unknown, there is no efficient method for numerically approximating capacity.
Some recent progress related to capcity computation was made based on deep learning (DL) techniques [9], [19]. In a novel work [9], mutual information neural estimator (MINE) [2] was used to learn a modulation for a memoryless channel. In [19], a capacity estimator was proposed based on reinforcement learning algorithm that iteratively estimates and maximizes the DI rate, but only for discrete alphabet channels with a known channel model. Inspired by the above, we develop the framework for estimating FF and FB capacity of arbitrary continuousalphabet channels, possible with memory, without knowing the channel model. Our method does not need to know the channel transition kernel. We only assume a stationary channel model and that channel outputs can be sampled by feeding it with inputs. Central to our method are a new DI neural estimator (DINE), used to evaluate the communication rate, and a neural distribution transformer (NDT), used to simulate input distributions. Together, the DINE and NDT lay the groundwork for our capacity estimation algorithm. In the remainder of this section, we describe DINE, NDT, and their integration into the capacity estimator.

A. Directed Information Neural Estimation
The estimation of mutual information (MI) from samples using neural networks (NNs) is a recently proposed approach [2], [3]. It is especially effective when the involved random variables (RVs) are continuous. The concept originated from [2], where MINE was proposed. The core idea is to represent MI using the Donsker-Varadhan (DV) variational formula The supremum is over all measurable functions T for which both expectations are finite. Parameterizing T by an NN and replacing expectations with empirical averages, enables gradient ascent optimization to estimate I(X; Y ). A variant of MINE that goes through estimating the underlying entropy terms was proposed in [3]. The new estimators were shown empirically to perform extremely well, especially for continuous alphabets.
Herein, we propose a new estimator for the DI rate I(X → Y). The DI is factorized as where Applying the approach of [3] to the entropy terms, we expand each as a Kullback-Leibler (KL) divergence and a cross-entropy (CE) residual and invoke the DV representation. To account for memory, we derive a formula valid for causally dependent data, which involves RNNs as function approximator (rather than the FF network used in the independently and identically distributed (i.i.d.) case). Thus, the DINE is an RNN-based estimator for the directed information rate from X n to Y n based on their samples.
DI estimators were recently presented in [25]- [27]. Also, an estimator of the transfer entropy using FF networks was proposed [16], which upper bounds the DI in the special case of a jointly Markov process with finite memory. DINE is the first method based on RNN and hence does not assume any parametric model such as discrete alphabets, or Markovity. Further details on the DINE algorithm are given in subsection II-A.

B. Neural Distribution Transformer and Capacity Estimation
DINE accounts for one of the two tasks involved in estimating capacity, it estimates the objective of (2). The remaining task is to optimize this objective over input distributions. Generally, sampling from an arbitrary distribution is a complex task. To overcome this, we design a deep generative model of the channel input distributions, namely the NDT. The idea is similar to ones used for generative modeling tasks, e.g, generative adversarial networks [23] or variational autoencoders [22]. The designed NDT maps i.i.d. noise into samples of the channel input distribution. For estimating FB capacity, in addition to the i.i.d. noise, the NDT also receives channel FB as inputs. Together, NDT and DINE form the overall system that estimates the capacity as shown in Fig 1. The capacity estimation algorithm trains the DINE and NDT models together via an alternating maximization procedure. Namely, we iteratively train each model while keeping the (parameters of the) other one fixed. DINE estimates the communication rate of a fixed NDT input distribution, and the NDT is trained to increase its rate with respect to fixed DINE model. Proceeding until convergence, this results in the capacity estimate, as well as an NDT generative model for the achieving input distribution.
We demonstrate our method on the MA(1)-AGN channel. Both C FF and C FB are estimated using the same algorithm, using the channel as a black-box to solely generate samples. The estimation results are compared with the analytic solution to show the effectiveness of the proposed approach.

II. METHODOLOGY
We give a high-level description of the algorithm and its building blocks. Due to space limitations, full details are reserved to the extended version of this paper. The implementation is available in github. †

A. Directed Information Estimation Method
We propose a new estimator of the DI rate between two correlated stationary processes, termed DINE. Building on [3], we factorize each term in (6) as: where h CE (P, Q) and D KL (P Q) are, respectively, the cross entropy (CE) and KL divergence between P and Q, and P Y is uniform reference measure over the support of the dataset. To simplify notation, we use the shorthands Subtracting both elements in (7) and observing that the difference of CE terms equals the DI at the former time step, we have Note that the difference of KL divergences equals I(X n ; Y n |Y n−1 ). For stationary data processes we take the limit and obtain (10) Each D KL is expanded by its DV representation [4] as: To maximize (11), each DV potential is parametrized by a modified LSTM and expected values are estimated by empirical averages over the dataset D n := {(x i , y i )} n i=1 . Thus, the optimization objectives are: whereỹ n i.i.d. ∼ P Y and T θY , T θ Y X are the parametrized potentials.
The estimator is given by: By universal approximation of RNNs [6] and Breiman's theorem [7], the maximizer of (13) approaches I(X → Y) as the number of samples grows, provided the neural networks are sufficiently expressive.
To capture the time dependencies in D n we introduce a modified LSTM network model for functional approximation. LSTM [5] is an RNN that receives a time series {y i } T i=1 as input and for each i, performs a recursive non-linear transform Algorithm 1 Directed Information Rate Estimation input: Samples of the process D n . output: I Dn (X → Y), estimated directed information rate.
Initialize networks parameters θ Y , θ Y X .
Step 1, Optimization: repeat Feed the network with the examples and compute Step 2, Perfrom a Monte Carlo estimation over D n and subtract loss evaluations to obtain estimation : to calculate its hidden state s i . We denote the LSTM function by F : (y i , s i−1 ) −→ s i . The full characterization of F is provided in [5].
We modify the structure of the LSTM to perform the calculations: A similar modification is introduced for D Y X by substitution of y i with (y i , x i ) and y i with ( y i , x i ), we have: A visualization of a modified LSTM cell (unrolled) is shown in Fig. 2. The LSTM cell's output is the sequence {(s i , s i )} n i=1 , which is fed into a fully-connected layer to obtain T θY and T θ Y X . As demonstrated by Algorithm 1 and Fig. 3, in each iteration we draw D B , a subset on D n , of size B. We feed the NN with D B to acquire T θY , T θ Y X . Those enter the NN loss function (12), and gradients are calculated to update the NN parameters θ Y , θ Y X .
...   Fig. 3. End-to-end architecture for the estimation of D Y (θ Y , Dn). Each batch of time sequences enters the system, a batch of the same size is sampled from the reference measure and those enter the NN to compute T θ Y and T θ Y X .

B. Neural Distribution Transformer
The DINE model is an effective approach to estimate the argument of (2). However, finding the capacity comprises maximization of the DI with respect to the input distribution. For this purpose we present the NDT model that represents a general input distribution of the channel. At each iteration i = 1, . . . , n the NDT maps an i.i.d noise vector N i to a channel input variable X i . When feedback is present the NDT maps (N i , Y i−1 ) −→ X i . Thus, NDT is represented by an RNN with parameters µ as shown in Fig. 4. The NDT model is used to generate the channel input X n , and the DINE estimates the DI between X n and Y n . Fig. 4. The NDT. The noise and past channel output (if feedback is applied) are fed into an NN. The last layer performs normalization to obey the power constraint, if needed.

C. Complete Architecture Layout
Combining DINE and NDT models into a complete system enables capacity estimation. As shown in Fig. 1, the NDT model is fed with i.i.d. noise and its output is the samples X n . These samples are fed into the channel to generate its output. Then, (X n , Y n ) are fed both to the DINE model that outputs I Dn (X → Y). To estimate the capacity, DINE and NDT models are trained together. The training scheme, as shown in Algorithm 2, is a variant of alternated maximization procedure. This procedure iterates between updating the DINE and NDT models parameters sets θ, µ, where each iteration the parameters of one model are fixed and the other ones are updated. By the end of training a long Monte-Carlo evaluation of ∼ 10 6 samples is done in order to estimate the expectations in (12) accurately.
Applying this algorithm to channels with memory estimates their capacity without any specific knowledge of the channel underlying distribution. Next, we demonstrate the effectiveness of this algorithm on continuous alphabet channels.
Step 2: Train NDT Generate B sequences of length T of i.i.d random noise with NDT and channel compute the objective:

III. NUMERICAL RESULTS
We demonstrate the performance of Algorithm 2 on the AWGN channel and the first order MA-AGN channel. The numerical results are then compared with the analytic solution to verify the effectiveness of our method.

A. AWGN channel
The power constrained AWGN channel is investigated as an instance of memoryless continuous alphabet channel for which analytic solution is known. The channel model is given by where Z i ∼ N 0, σ 2 are i.i.d RVs, and X i is the channel input sequence bound to the power constraint E X 2 i ≤ P . Its capacity is given by C = 1 2 log 1 + P σ 2 . In our implementation we chose σ 2 = 1 and estimated the capacity for a range of P values. The numerical results are compared to the analytic solution in Fig. 5

B. Gaussian MA(1) channel
The calculation of capacity of linear Gaussian channels with memory can be divided into two cases, feedback (C FB ) and feed-forward (C FB ) capacity. We will focus on the MA(1) Gaussian channel model, which is given by: ∼ N (0, 1), X i is the channel input sequence bound to the power constraint E X 2 i ≤ P , and Y i is the channel output.

1) Feed-forward capacity:
For the LTI Gaussian channel with input power constraint, C FF can be obtained by applying the water-filing algorithm [14]. We applied Algorithm 2 to estimate C FF and compare with results of the water-filling algorithm. Results are in Fig. 6. 2) Feedback capacity: In general, C FB of the ARMA(k) Gaussian channel can be formulated as a dynamic programming problem, which can be solved by an iterative algorithm [11]. For the particular case of (17), C FB is given by − log(x 0 ), where x 0 is a solution of a 4th order polynomial equation. We applied Algorithm 2 for the feedback capacity to obtain an estimate of C FB . The results and compared with the analytic solution as shown in Fig. 7).

IV. CONCLUSION AND FUTURE WORK
We have presented a methodology to estimate FF and FB capacity using the channel as a "black-box". The estimator is designed by a novel DI estimator (DINE) and NDT model, both based on RNNs. The performance of the estimator are demonstrated on the AWGN and MA(1)-AGN channels, and estimation agrees with the analytic solution.
We wish to further generalize our method of information rate estimation for multi-user communication channels, a field with many unsolved problems and to find theoretical guarantees of the estimator. In addition, information theory (e.g, channel capacity) give us a rigorous mathematical framework where analytical solution are known due to Shannon theory hence this can be a good problem for evaluating machine learning approaches.


Hybrid Improved Document-level Embedding (HIDE)

In recent times, word embeddings are taking a significant role in sentiment analysis. As the generation of word embeddings needs huge corpora, many applications use pretrained embeddings. In spite of the success, word embeddings suffers from certain drawbacks such as it does not capture sentiment information of a word, contextual information in terms of parts of speech tags and domain-specific information. In this work we propose HIDE a Hybrid Improved Document level Embedding which incorporates domain information, parts of speech information and sentiment information into existing word embeddings such as GloVe and Word2Vec. It combine improved word embeddings into document level embeddings. Further, Latent Semantic Analysis (LSA) has been used to represent documents as a vectors. HIDE is generated, combining LSA and document level embeddings, which is computed from improved word embeddings. We test HIDE with six different datasets and shown considerable improvement over the accuracy of existing pretrained word vectors such as GloVe and Word2Vec. We further compare our work with two existing document level sentiment analysis approaches. HIDE performs better than existing systems.

Introduction
Sentiment analysis is a technique to classify document based on latent opinion expressed in the textual contents. It used across a various domain of research, such as marketing decision making, social media analysis and many more (Deng, Sinha, & Zhao, 2017;Mostafa, 2013;Pantano, Giglio, & Dennis, 2019). With recent development in Natural Language Processing (NLP) techniques, word embedding is frequently used in many NLP tasks such as -syntactic parsing, question answering etc. (Iyyer, Boyd-Graber, Claudino, Socher, & Daumé, 2014;Socher, Bauer, Manning, & Ng, 2013). Word embedding is a robust approach to capture semantic of a target word by finding conditional probability from its context (Shuang, Zhang, Loo, & Su, 2020). These embeddings represent words in a continuous low dimensional vector. As context of word is involved in generation of these vectors, the words with similar semantics are represented in near proximity in the embedding space. However, gerneration of word embedding require large corpora and involve costly processing time. This drawback compel researchers to turn their focus towards pre-trained word embeddings. Among pre-trained embeddings, Word2Vec and GloVe are used successfully (Corrêa & Amancio, 2019;Kameswara Sarma, 2018;Mikolov, Sutskever, Chen, Corrado, & Dean, 2013;Olatunji, Li, & Lam, 2020;Pennington, Socher, & Manning, 2014;Saumya, Singh, & Dwivedi, 2019). In spite of the success of pre-trained embeddings in various NLP tasks, it is not considerably perform well in sentiment analysis. This performance degradation of pre-trained embeddings in sentiment analysis tasks is driven by certain shortcomings. Firstly, pre-trained embeddings are generated predominantly to capture the semantics of a word using its context. However, many times a positive and a negative word appear in similar word context. Secondly, words in a particular domain have a different distribution than the generic textual contents which is used to generate pre-trained embedding. Due to lack of domain-specific information, the performance of pre-trained embeddings in sentiment classification tasks is degrading (Blitzer, Dredze, & Pereira, 2007;Sarma, Liang, & Sethares, 2018). Thirdly, although word embeddings represent the semantics of a word based on its context, it does not incorporate the entire information concerning a word such as parts of speech (POS) (Petroni, Plachouras, Nugent, & Leidner, 2018).
In this work, we address the problems discussed above and present a Hybrid Improved Document-level Embedding (HIDE) by incorporating domain-specific, sentiment and POS information. Firstly, we adopt an approach to enrich word level embeddings with domain specific information. We consider pre-trained GloVe and Word2Vec embedding. Here, Word2Vec is generated from a domain-specific corpus. The combined vector of both word embeddings captures generic representation of words along with the domain information. Secondly, to address the word polarity related issue, we use lexicon-based sentiment analysis dictionaries to find the polarity of a word. We vectorized the lexicon information and embed it into word vectors. Thirdly, we integrate the parts of speech (POS) information of each word in a vectorized format. The document vector is generated by combining the embeddings of each word appeared in the document. Next, to incorporate document level domain information we use Latent Semantic Analysis (LSA) which represent the documents in low dimensional vector format. Combination of these two document level vector produces HIDE. We have tested our approach through several experiments with six benchmark datasets and Support Vector Machine (SVM) classifier with Gaussian radial basis kernel. The result shows an improvement over the pre-trained vectors as well as in sentiment analysis performance.
The work of Rezaeinia et al., (2019) motivates our approach of enriching word embeddings only; however, we differ in the following ways. First, we take combination of GloVe and Word2Vec embeddings and avoid random vector generation. This combination improves the word vectors with more relevant embedding than a random one. Second, all the word in a document are represented by their corresponding embeddings and document embedding is computed using average embeddings of participant words. Due to this, the position of a word in the document becomes irrelevant. Third, we have used less number of lexicon dictionaries to capture word sentiment scores; we discuss this in the following sections. Lastly, in contrast to word level embedding, we implement a domain-specific document level embeddings. Our main contribution in this work is as follows - • We propose HIDE, an approach to develop a document level embedding with sentiment, POS and domain specific information.
• We improve word embedding by incorporating domain specific, POS and sentiment information using less number of lexicon databases.
• To incorporate document level domain information Latent Semantic Analysis (LSA) has been used. To best of our knowledge this is first time LSA is combined with improved word embeddings.
• HIDE supersedes many of the existing word embedding based sentiment classification approaches. We present our work with six different benchmark datasets and compare it with other approaches of word embedding improvement and domain-specific sentiment analysis.
The next portion of the paper is organised as follows -section 2 related works, section 3 methodology, section 4 experiment, section 5 results and discussion and section 6 conclusion respectively.

Related Works
The related literature concerning word and document level embedding is reported in the next sub-sections.

Word representation using Embeddings
In natural language processing, word representation is considered as one of the vital tasks (Yu, Wang, Lai, & Zhang, 2018). Word embedding represent a word as a distributed vector using its context in a large corpus with the help of a neural network. The earliest work, in this area, suggests, generation of embedding using neural network language model (NNLM) leveraging the context of each word (Bengio, Ducharme, Vincent, & Jauvin, 2003;Yu, et al., 2017). Later on, word embeddings are generated using models like a continuous bag of words (CBOW) and continuous skip-gram (Stein, Jaques, & Valiati, 2019). Both of these methods are concerned with the context of the word. In skip-gram, the current word is used to predict the context of the word and CBOW model, predict the word based upon its context (Pham & Le, 2018).
Word2Vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) embeddings are used in many text classification tasks (Joulin, Grave, Bojanowski, & Mikolov, 2017;Rezaeinia et al., 2019;Yu, et al., 2017). Word2Vec is estimated by maximizing the log conditional probability, occurred within a window of context (Nalisnick, Mitra, Craswell, & Caruana, 2016). Whereas, GloVe works on global word co-occurrence and local context window (Pennington et al., 2014;Yu, et al., 2017). Generation of these custom embeddings requires large corpora, which is not always feasible, hence pre-trained Word2Vec and GloVe are used (Corrêa & Amancio, 2019;Kameswara Sarma, 2018;Mikolov et al., 2013;Pennington et al., 2014;Saumya et al., 2019). Pre-trained embedding computes the word vectors from general-purpose corpora such as Wikipedia (Rezaeinia et al., 2019). Although pre-trained embeddings are very efficient in textrelated tasks, its performance is not appreciable in case of sentiment analysis (Kameswara Rezaeinia et al., 2019;Sarma et al., 2018). This happens because these embeddings are created by prioritising the context of a word, and in that way, if two mutually opposite sentiment words occur in the same context then both of the words get similar embedding (Fu, Sun, Wu, Cui, & Huang, 2018;Tang et al., 2016). For example, words like happy and sad may appear in the same context (Yu et al., 2017). Unable to separate words based upon polarity degrades sentiment classification accuracy (Giatsoglou et al., 2017;Yu et al., 2017).
To address this problem, the sentiment embedding technique is proposed to refine the pretrained word vectors using the lexicon sentiment intensity scores ( Yu et al., 2017). Giatsoglou et al., (2017) proposed a hybrid feature taking lexicon values along with word embeddings together. These works focus on incorporating sentiment information into word embeddings, but it does not archive parts of speeches (POS) information of the words. Att2vec method learn word embeddings, including the POS tag information (Petroni et al., 2018). The drawback here is it does not include lexicon information. In this context, IWV (Improved Word Vector) incorporates both POS tags and lexicon information into pre-trained word embeddings to improve the sentiment analysis accuracy (Rezaeinia et al., 2019).
Apart from the sentiment and POS related issues, pre-trained word embeddings also lack domain-specific information as they are generated out of general-purpose corpora (Dragoni & Petrucci, 2017;Sarma et al., 2018). To assimilate domain information word embeddings are to be computed from domain-specific corpora. Moreover, documents are built upon words; the existing approaches to combine word embeddings into the document is discussed in the next subsection.

Document-level Embedding
In supervised sentiment analysis techniques, term frequencies and count vectors are used as features in many of the cases (Tripathy, Agrawal, & Rath, 2016). These two methods have their own disadvantages. The vector space model generated from term frequency or by count vectors is high dimensional and heavily sparse. Moreover, it represents documents statistically rather than semantically and ordering of the words are not maintained (Huang, Qiu, & Huang, 2014;Stein et al., 2019). Word embeddings are useful in this context, but representing documents using word embedding is tricky as the frequency of words varies in different textual contents (Huang et al., 2014). To pacify this, an average of word vectors of each word present in the document is adopted by many researchers (Huang et al., 2014;Kameswara Sarma, 2018;Stein et al., 2019). Le & Mikolov, (2014) proposed paragraph vector, which is a unsupervised approach to representation fixed-length vector from sentences, paragraphs, and documents preserving word ordering. However, in paragraph vector the necessity of domain knowledge is not embedded. Domain knowledge plays a significant role in sentiment analysis. So, accumulation of domain knowledge and integrate into sentiment classification feature is important (Dragoni & Petrucci, 2017). Training of word vectors demands vast corpus. However, embedding with smaller dimension can capture a legitimate amount of information from a small-sized domain-specific corpus (Stein et al., 2019).
On the other hand, Latent Semantic Analysis (LSA) is used to produce a document level vector which is more semantically enriched way (Mitra & Jenamani, 2018;Stein et al., 2019). LSA transform the document term matrix in the denser low dimensional representation of the document using Singular Value Decomposition (SVD). When applied on documents of the same domain, LSA can capture domain-specific semantics in terms of vector representation (Fernández-Reyes & Shinde, 2019; Kameswara . It is shown that aligning pretrained vectors along with domain-specific vectors generated using LSA, performs better compared to individual ones . However, the lexicon-based sentiment scores of each word is not considered in this method. Moreover, word-level improvement of pretrained embedding is not considered here. Table1 discussed key literature of this study. Table1 represents literature based on their proposed methods and used information to generate word embeddings.
Table1: Few recent works to improve word embedding

Paper Proposed Method Used information Domain
Sentiment POS (Mikolov et al., 2013) Word2Vec model where word vectors represented by maximizing the log conditional probabilities within a context window.
X X X (Pennington et al., 2014) The GloVe uses the global co-occurrence matrix as well as local contexts.
X X X (Giatsoglou et al., 2017) Proposed a hybrid vector with vector produced by lexical scores and pre-trained word embedding.
X X (Dragoni & Petrucci, 2017) NeuroSent tool has been proposed for domainspecific polarity classification. A domain-specific corpus is used to generate word embeddings. X X (Yu et al., 2017) Sentiment embeddings are generated using word embeddings and lexicon scores of words. X X (Petroni et al., 2018) Used POS tag information to contextually enrich word embeddings for better classification performance.
X X  Propose domain adapted word embedding where LSA based domain-specific embedding and pretrained vectors are aligned using canonical correlation analysis. X X (Rezaeinia et al., 2019) Focused on improving the pre-trained word vectors for better accuracy in sentiment analysis tasks.

X
Our Approach LSA based domain-specific embeddings combined with improved pre-trained and domain-specific word embeddings to analyse the sentiment of a document

Methodology
In HIDE we improve the performance of word embeddings by applying natural language processing techniques, lexicon score of words, parts of speech information of the words. We configure document level embedding from these word embeddings and combined with domainspecific document level vectors to improve the sentiment classification tasks. In Fig.1, we show our proposed methodology. We improve word embeddings with domain-specific corpora and sentiment information. Then we generate document embedding from improved embeddings. We perform LSA to capture document level domain adaptive representation. Finally, we combined these two document embeddings into HIDE model.

Generation of word embedding
Pre-trained word embedding do not have domain-specific information as it is trained on large corpora of diverse topics, mostly on Wikipedia corpus. To address this issue we have used Word2Vec on domain-specific corpora. Use of domain specific texts reveals domain-specific semantics which embed into the word vectors (Ferrari, Donati, & Gnesi, 2017;Hu et al., 2018). Gensim python library has been used in this purpose. This library provides modules to generate Word2Vec and related algorithms. Now, In comparison with other general-purpose corpora, domain-specific corpora is not huge. Small dimensional word vectors works better to capture semantics, if small size dataset has been used for training. Hence, we generate 50-dimensional domain-specific embedding. We use separate domain specific corpora to generate domainspecific Word2Vec embeddings. The description of domain specific corpora is given in the next section. Along with domain specific Word2Vec, we incorporate 50 dimensional GloVe pre-trained embeddings.
Every word in the classification process is converted into its corresponding embedding by taking average of domain-specific Word2Vec and pre-trained GloVe. However, there may exist some of new words in the classification dataset which do not appear in the domain specific corpora.
For example, suppose a sentence present in the corpus as "It's a great movie to watch". So, after removing the stop words, i.e. "it's", "a", "to" there remains three words "great", "movie" and "watch" which can polarize the sentence. Table2 shows the method of combining Word2Vec and Glove.
Table2: Example of combining word embedding

Improving word embedding
The improvement of word embedding is done using sentiment lexicon scores; POS tag information.

Lexical Score to Vector (Lex2vec)
Sentiment lexicons are the words which are vital for predicting the sentiment of a document (Medhat, Hassan, & Korashy, 2014;Turney, 2001). The lexicon-based approach identifies the polarity of a document using the semantics of the phrases present (Taboada, Brooke, Tofiloski, Voll, & Stede, 2011). Here a dictionary of sentiment words is used along with the score for each lexicon. Many lexicon dictionaries are available. From them, choosing a lexicon dictionary is vital as because some of the dictionaries are more accurate than the other.   which contains a total of 26,577 lexicons and value ranges from -5.27 to +3.702. Each of these lexicon databases has a word or a phrase along with its sentiment scores. We normalized and scaled the sentiment scores of all the words between [0, 1] and formed a vector by concatenating the lexical values from both these databases. The lexical values of the words not present in the databases are considered to be zero. The lexical score vector of running example sentence is given in Table3.

Parts Of Speech to Vector
POS tagging is a technique of assigning the words with their Part of Speech tags. This information is not captured in the pre-trained word embedding. However, the recent study shows it improves the classification accuracy of pre-trained embedding (Petroni et al., 2018).
In case of sentiment analysis, involving POS tag information in feature vector gives better results (Pasupa & Seneewong Na Ayutthaya, 2019). The POS tag helps the model to figure out the actual meaning of the word (whether the word is a noun, adverb, adjective etc.). We figure out the POS tags of a word using a POS tagger and converted the POS tag of each word into a vector having one-hot representation, i.e. the index of the corresponding tag was given a value one and all others as zero. Nltk toolkit (Navarre & Steiman, 2002)  . Fig.2 shows the final embedding of the running example. Fig.3 displays the entire process of improving word embeddings. As both Word2Vec and GloVe pretrain embeddings are of same length of 50-dimesion the resultant average combination is of same size. However, after appending the lexical and POS vector the final embedding increase in length. We rescale the dimensions of embeddings in Fig.3 for the ease of understanding. These improved embeddings are used further to generate document level embeddings.

Document Representation using Word Embedding
Documents are collection of words. With word embedding technique we get embeddings of each word. As we want to incorporate the domain information, we generate our document embedding by combining two types of vector representation. First, document embedding is generated from the improved word vectors. Second, another document representation is computed by applying LSA on the training corpus. The final document embedding is generated by combining these two document representation. GloVe and Word2Vec do not provide any optimized embedding for sentences or documents, so taking the average of embeddings corresponds to all words present in a document can give a document level embedding (Renter, Borisov, & De Rijke, 2016

Experiments
This section is about the experimentation conducted to analyse the performance of HIDE on six benchmark datasets. The description of the datasets, our approach and setting of our experimentation has been discussed. We primarily estimate the advantage of HIDE over the use of word vector and lexicons. Different combination of word embedding and lexicon database has been used to generate hybrid document level vectors. We also compare HIDE with existing method to understand its efficiency.

Description of Datasets
We use total of six benchmark datasets. We take datasets from disjoint domains such asmovies, electronic products etc. The datasets are -MR: This dataset contains 1000 positive and 1000 negative movie reviews categorised based on their star ratings (Pang & Lee, 2005). Rating less than 3, labelled as negative and rating four and five marked as positive. SST: Stanford Sentiment Treebank dataset consisting 11,855 sentences of label positive and negative (Socher, Perelygin, et al., 2013).

SST-1:
This dataset is similar to SST having train, test and dev splits and includes fine-grained labels such as -very positive, positive, neutral, negative, very negative .

Amazon Review Dataset (ARD):
This dataset contain 14.2 million entries (He & McAuley, 2016;McAuley, Targett, Shi, & Van Den Hengel, 2015). Although, the reviews are not labelled as per positive or negative sentiment but in the light of Pang & Lee, (2005) we labelled these reviews in positive and negative categories based on their star rating. Rating below three considered as negative and above three marks as positive.
The description of dataset has been given in Table5. We use benchmark datasets from the movie and electronic domain to generate Word2Vec embeddings. In this purpose, we use Large Movie Review dataset for binary sentiment classification (Maas et al., 2011). It contains 25,000 training and 25,000 for testing reviews. For electronic domain Amazon review dataset (He & McAuley, 2016;McAuley et al., 2015) has been used. The electronic category contains around 1,689,188 reviews. We randomly select 25000 reviews for domain specific Word2Vec generation. The detail description of datasets used in both experiments and domain specific Word2Vec generation is given in Table6.

Evaluation protocol
We present HIDE with Support Vector Machine (SVM) classifier. It is a discriminative classifier which realized by a separating hyper-plane. In many of text classification tasks along with sentiment analysis SVM has been used (Catal & Nangir, 2017;Dey, Jenamani, & Thakkar, 2017;Tripathy et al., 2016). Accuracy is used to assess the performance of the method which is defined as -T P + T N A ccu racy = (5) T P + T N + FP + FN Where, TP is True positive, which represent reviews labelled correctly, as positive. True Negative represents by TN shows negative reviews labelled with negative. The other conditions are represented by False Positive ( FP ) and False Negative ( FN ) when negative reviews are labelled as positive and positive reviews are labelled as negative respectively. Let's consider, two sentences from MR dataset, "I recommend this movie for children and adults who are a child at heart" and "The film had no script to test any actors acting skill or ability". It is clear that the first sentence is belong to the positive class and second one belongs to the negative class. Now, if the model classifies the first sentence as positive, it will be consider as True Positive (TP) and if the second sentence is classified as negative then it will be True Negative (TN). However, False Positive (FP) happens when the second sentence is labelled a positive. It is also similar to type I error. Similarly, if the first sentence marked as negative by the model it will be a False Negative (FN) incident, which may refer as type II error as well. We can represent the classification scenario using a 2x2 confusion matrix. Fig.4 shows the corresponding confusion matrix. We use Precision (P), Recall (R) and F-scores measures based on the confusion matrix to get more precise results. Precision, recall and F-score defined as -T P P r e c i s i o n = ( 6 ) T P + F P T P R e c a l l = ( 7 ) T P + F N 2 x ( P x R ) F -s c o r e =

Results and Discussion
In next subsections, we report the results obtained by applying the evaluation protocol stated above on six datasets. We experiment with three popular machine learning classifiers and choose SVM. As HIDE is a document level representation and the basic intention is to estimate its performance over the pre-trained vectors, we experiment with document vector generated using only GloVe and domain specific Word2Vec separately. Moreover, HIDE considers two types of lexicons, hence, we experiment with individual lexicon databases along with their combinations to see how performance varies. We compare HIDE with two existing models. First, domain adapted sentiment analysis model proposed by Kameswara Sarma, (2018). Second, with improved word vectors (IWV) (Rezaeinia et al., 2019). Table8 shows the performance scores. Cross validation is applied whenever is possible. SST and SST-1 datasets are already clubbed into train and test set, so we do not use further cross validation. However, to find best classification parameter we perform 10-fold cross validation. To find suitable parameter for SVM we use grid search algorithm with parameter C and gamma. Best possible performance has been chosen based on performance accuracy in the grid search.
We have used python as the basic programming language for experimentation. We have used 50 dimensional GloVe embeddings. To keep the consistency in vector length the dimension of the Word2Vec is remain 50. To represent the POS information 36 dimensional vectors has been used. Further for lexicon information for each word a 2 dimensional vector is computed. Lastly, document representation using LSA is computed into a 50 dimensional space to keep uniformity. Finally, after combining all these vectors we get a 138 dimensional HIDE embedding. These vectors are considered as final feature vector and use it for classification.

Experiment with other classifiers
SVM is used as base classifier in HIDE. To understand the linearity of the final HIDE embedding we implement t-SNE plots with randomly selected 70 documents from positive and negative data. TSNE, t-Distributed Stochastic Neighbour Embedding, is a technique to visualize high dimensional data. Fig.5 shows the t-SNE plot of HIDE for four datasets. The t-SNE plot indicates that the HIDE is not linearly separable. Hence, we use Gaussian radial basis function as SVM kernel instead of linear one. However, before choosing SVM as classifier, we conduct experiments with other popular machine learning classifiers like -Naïve Bayes, Decision Tree. These classifiers are experimented upon MR and RT datasets. Neural networks and deep learning models are not used in our approach mainly because of two reasons. Firstly, neural networks works well with large amount of data. HIDE is a document level representation and as the dataset use in the process are small in size, we drop the implementation of neural networks. Secondly, neural networks are costly in terms of time and resource utilization. The evaluation of SVM with other two classifiers shown in Fig.6. Table7 shows the results of different classifiers. We use SVM parameter gamma is kept in "scale" and C=1 for this experiment. It is evident that the accuracy of SVM is higher than rest of the two classifiers with a steady precision recall value. Hence, we choose SVM as classifier for this study.

Experiment with Word Vectors
We experiment with three type of approaches to generate a document level embedding in context of HIDE. The first case involve only GloVe pre-trained vectors to generate document level embedding. The second case involve only Word2Vec, where it is trained on domain specific corpora. Lastly, we consider combination of both the embedding and generate document level embedding. Table8 shows the comparison based on accuracy. The precision, recall estimations are shown in Fig.7 and Fig8. Table9 represents the comparison based on fscores. As per the grid search results we fix the SVM parameter for these experiment. For ARD the C value is 0.1 and for SST-1 gamma is 0.9 except that all the experiments give best results with C=1 and gamma= "scale".
Table8 clearly shows HIDE performs better than document embedding generated from GloVe, Word2Vec and their combination. It is also noticeable that combination of domain specific Word2Vec and GloVe does not provide better result than GloVe. It is because of two reasons. Firstly, Word2Vec is trained on domain specific corpora in comparison to the general purpose corpora, on which GloVe has been trained. As a result, Word2Vec captures essential domain specific information from the context around the target word. However, it significantly change the embedding of a particular word with respect to pre-trained GloVe. Secondly, due to the small size of domain specific corpora, it is not always possible to get embeddings of all the words present in the test dataset. This drawback is eminent in the performance of domain specific Word2Vec where its accuracy is way less than GloVe. HIDE eradicates this flaw by taking advantage of both the embeddings in association with lexical and POS information. Furthermore, HIDE combines document representation generated using LSA which contributes to its performance and makes it more befitting for sentiment analysis. HIDE performs maximum with IMDB (MR) dataset and Amazon Review Dataset (ARD) with 87% accuracy.
MR and RT are balanced datasets whereas CR is not a balanced dataset. GloVe performs better in CR with respect to other datasets. However, HIDE outperform GloVe with an increase of 3.5% in accuracy. Another interesting observation can be made from SST-1 dataset. This dataset contain multiple classes. As HIDE is modelled from the perspective of binary classification, it is expected it will perform with less accuracy with multiclass classification. Other vectors also suffers from similar drawback. The accuracy score of SST-1 for GloVe, Word2Vec and their combination remain close to HIDE. However, HIDE marginally performs better. F-scores for HIDE is better than GloVe and domain specific Word2Vec embeddings. Fig.5 shows the comparison of precision scores for different datasets. HIDE performs better than other method in terms of precision. Fig.6 shows the recall values for each dataset.

Experiment with Lexicon databases
The goal of this experiment is to analyse the effect of lexicon databases use to generate HIDE.

Comparison with other Methods
To analyse the performance of HIDE, we compare HIDE with existing word embedding improvement algorithm IWV (Rezaeinia et al., 2019) and Domain Adapted (DA) embeddings . In IWV, pre-trained vectors, GloVe and Word2Vec has been improved with lexical information, POS tag and word position information. In contrast to IWV, in HIDE, document is represented by the average of word embeddings which makes word position information redundant. Moreover, domain specific Word2Vec is not implemented in IWV. In DA  method, document embedding has been generated by LSA, then it combined with GloVe and Word2Vec pre-trained embedding to get higher accuracy in sentiment analysis. However, DA does not incorporates any lexical information.
In Table11  where each sentence is marked as positive and negative. So, with HIDE when we are applying LSA considering each sentences as a document the generated document embedding is not able to capture much information at document level. This is the cause of the visible drop in the accuracy of HIDE in both CR and SST dataset. In case of SST-1 dataset the accuracy of both IWV and DA method is low. It is due to multiclass classification issue as we discussed earlier.
From the comparison of different method it is evident that, lexical information plays a significant role in document sentiment classification. DA method incorporates domain information. However, it does not include lexical information. In our experiment DA does not make any significant score and outperformed by HIDE and IWV.
The cumilative analysis of comparision discussed above, gives an insight about each individual components of HIDE. For instance, dataset with larger vocabulary contain more domain specific information which helps HIDE to capture precise domain contexts. The domain information is also embedded into document level by LSA. The performance of HIDE in CR and SST dataset indicate that HIDE is relatively dependent on the nature of the training dataset. It also estabilishes the significance of document level domain information in HIDE. Lastly, lexical information corresponding to a word is as necessary as domain information to analyze the sentiment orientation of a document.
We further tallied between HIDE, IWV and DA to show the amount of improvement HIDE incur upon pre-trained GloVe embeddings. Table12 shows the comparision. Improvement using IWV is positive for MR, SST and CR datasets but negative for RT,SST-1 and ARD dataset. Whereas, HIDE shows consistent and positive increase over GloVe embedding for all the datasets.

Conclusion
Pre-trained word embeddings, which are used in various NLP tasks, suffers from poor accuracy in sentiment analysis tasks. Pre-trained word embeddings are trained on general purpose corpora which results in lack of domain and sentiment information in each word vectors. To address this problem, we propose an approach, Hybrid Improved Document level Embedding (HIDE). In HIDE, we improved word level embeddings with domain, sentiment and POS information. Each word of a document is combined into document level representation. Similarly, using LSA we generate document level embedding and combine both of these document level vectors for sentiment analysis. HIDE is tested using six benchmark datasets and it shows considerable improvement over pre-trained GloVe vectors and domain specific Word2Vec. We have tallied HIDE with two existing methods IWV and DA. HIDE comes as better solution for document level sentiment analysis in most of cases.
Although, experiments using HIDE gives good outcomes, it is not free from limitations. Firstly, we have used only GloVe and Word2Vec, there are few more well-known embeddings are available such as -Fasttext, Elmo etc. HIDE can be implemented using one of those embeddings as well. Secondly, the datasets involved in this experiment are not very large in size. Hence, experimentation can be done using large datasets. Thirdly, we conduct our experiment with movies and electronics domain, other domain datasets should also be tested. Moreover, deep learning techniques is not used in our experiments. With larger dataset and good deep learning models embeddings can be improved.