extfieldsofstudy:
Computer Science
Medicine

text:
Comparison of classification algorithms for predicting autistic spectrum disorder using WEKA modeler

Background In healthcare area, big data, if integrated with machine learning, enables health practitioners to predict the result of a disorder or disease more accurately. In Autistic Spectrum Disorder (ASD), it is important to screen the patients to enable them to undergo proper treatments as early as possible. However, difficulties may arise in predicting ASD occurrences accurately, mainly caused by human errors. Data mining, if embedded into health screening practice, can help to overcome the difficulties. This study attempts to evaluate the performance of six best classifiers, taken from existing works, at analysing ASD screening training dataset. Result We tested Naive Bayes, Logistic Regression, KNN, J48, Random Forest, SVM, and Deep Neural Network algorithms to ASD screening dataset and compared the classifiers’ based on significant parameters; sensitivity, specificity, accuracy, receiver operating characteristic, area under the curve, and runtime, in predicting ASD occurrences. We also found that most of previous studies focused on classifying health-related dataset while ignoring the missing values which may contribute to significant impacts to the classification result which in turn may impact the life of the patients. Thus, we addressed the missing values by implementing imputation method where they are replaced with the mean of the available records found in the dataset. Conclusion We found that J48 produced promising results as compared to other classifiers when tested in both circumstances, with and without missing values. Our findings also suggested that SVM does not necessarily perform well for small and simple datasets. The outcome is hoped to assist health practitioners in making accurate diagnosis of ASD occurrences in patients.

Introduction
Autistic Spectrum Disorder (ASD), which was first described in 1940, is a group of neuro-developmental disorders that commonly occurs in boys [1]. Children with ASD will commonly develop a number of peculiar behavioural changes and will have difficulties in social interaction and communication which will become more apparent when the children reach six months of age [2][3][4]. There is a need to ensure that ASD is diagnosed at the earliest stage in order to facilitate and prepare the parents or caregivers of the affected children with appropriate interventions as soon as possible [4]. Early interventions for patients diagnosed with ASD are important so as to ensure that the patients and caregivers are able to deal with the environmental circumstances during the growing-up phase of the children. Thus, prompt classification of ASD in children is very important which makes the integration of Open Access *Correspondence: fairuzradzi@usm.my screening tool with data mining tool crucial. A systematic review conducted by Marlow et al. [5] showed that the screening tools that have been used by healthcare practitioners have yet to incorporate machine learning intervention intensively. According to Pinto-Martin et al. [6], there is a need to ensure that a tool is incorporated in the development of ASD screening tool for rapid and accurate diagnosis since medical practitioners may probably have difficulties to keep up with the screening process all by themselves which can lead to human errors and in this case, machine learning method can help to solve the issue [7].
There is an urgent need to ensure that machine learning is incorporated in healthcare area to assist the practitioners, however, not many studies have been conducted to examine the sensitivity, specificity, and accuracy of machine learning techniques for ASD screening process which makes the integration nearly impossible. This has been mentioned by Thabtah [8] who compiled a number of ASD screening-related datasets which can be analysed through machine learning for improving the sensitivity, specificity, and accuracy in diagnosing ASD patients. In other similar research, Thabtah [9] found that the algorithms in machine learning can assist practitioners in expediting the process of screening and diagnosing a patient with ASD. Thus, we attempt to fill in the gap by conducting an experiment to compare the best classification algorithms to be used for ASD screening process. This paper is organized as follows; "Research Method" section explains the approaches that we used to classify the ASD screening dataset including the classification process, classification techniques, performance matrix, and dataset involved in this study, "Result and discussion" section presents the results of the classification process using WEKA Explorer and WEKA Experimenter to the dataset in two conditions; with and without missing values, "Conclusion" section concludes the findings and suggests several potential studies that can be conducted in the future.

Motivation
Not many studies have utilized classification algorithms to particularly analyse ASD screening dataset. From the literature review, most researchers utilized other most common datasets such as breast cancer and heart disease datasets, thus there exists difficulties in analysing and comparing the performance of classifiers being tested to ASD screening dataset. This is important in order to assist in selecting the best classification methods for screening and diagnosing a patient with ASD, thus we attempted to fill in the gap by analysing and classifying ASD screening dataset by using six classifiers.

Related works
There have been a number of studies conducted to investigate the role of data mining or data science in assisting healthcare practitioners in analysing and diagnosing a particular disease or illness in an effective and efficient manner. Many countries have been utilizing health informatics as a way of reducing the government spending on healthcare data management. In the United States, health informatics has helped the government to save the medical management industry up to $450 billion each year annually [10]. This is due to the fact that health informatics is capable of integrating big healthcare data from different sources which is useful to assist the healthcare practitioners particularly in detecting or diagnosing a disease or illness more accurately.
The utilization of machine learning for data mining can assist professionals to examine medical or health related problem that can lead to better understanding of healthcare related issues [11]. In data mining, several methods are utilized to analyse healthcare dataset especially for classifying a particular disease; statistical analysis, decision tree, k-nearest neighbour, artificial neural network, and many more [12].
There is an urgent need to ensure that machine learning is incorporated in assisting the practitioners, however, little studies were conducted to examine the sensitivity, specificity, and accuracy of machine learning techniques for ASD screening process which makes the integration nearly impossible. This has been mentioned by Thabtah [8] who compiled a number of ASD screening-related datasets which can be analysed through machine learning for improving the sensitivity, specificity, and accuracy in diagnosing ASD patients. This study attempted to fill in the gap by conducting the experiment in comparing the best classification algorithms to be used for ASD screening. In other similar research, Thabtah [13] found that the algorithms in machine learning can assist practitioners in expediting the process of screening and diagnosing a patient with ASD.
Ramotra et al. [14] conducted a study to predict the presence of heart disease by utilizing classification techniques namely Decision Tree, Naïve Bayes, Support Vector Machines, and Artificial Neural Networks classifiers. The study concluded that Naive Bayes classifier presented the highest accuracy among other classifiers being tested. A study conducted by Kibis et al. [15] utilized artificial neural network (ANN), classification and regression tree (C&RT), Logistic Regression, and Bayesian Belief Network (BBN) to investigate breast cancer survivability with respect to the classifiers' accuracy, sensitivity, specificity, and area under the curve (AUC) metric. The study found that Logistic Regression method presented better performance with regard to AUC metrics. Alaiad et al. [16] tested five classification algorithms, namely, Naive Bayes, Decision Tree, Support Vector Machine, K-Nearest Neighbour (KNN), and JRip in predicting chronic kidney disease. The study found that KNN achieved high performance in term of the accuracy as compared to other classification algorithms. In a study conducted by Dawngliani et al. [17] to classify benign and malignant tumors for breast cancer screening dataset, J48 decision tree outperformed other decision tree algorithms in terms of the accuracy. The study tested J48 with other decision tree algorithms namely Decision Stump, Random Forest Tree, REP tree, Hoeffding Tree and Logistic Model Tree (LMT). In other study conducted by Mun and Jumadi [18], Naïve Bayes, Decision Table and Random Forest were tested to compare which algorithm produced the best statistical performance in classifying dyslexia dataset. The study has shown that Random Forest produced 100% accuracy and specificity in classifying dyslexia subject. Raj and Masood [19] classified ASD screening dataset by using Naïve Bayes, Support Vector Machine, Logical Regression, KNN, Neural Network, and Convolutional Neural Network (CNN). The study concluded CNN to be the best algorithm for handling classification as it has the highest accuracy as compared to other algorithms being tested.
The related works mentioned above showed that different classification techniques outperformed other classification techniques when tested to health related datasets. In relation to the mentioned related works, this study is intended to test the best classification technique from each existing study to the the ASD screening dataset in order to investigate which classification technique can produce better performance in screening the ASD class. We propose to investigate the performance of Naive Bayes, Logistic Regression, KNN, J48, Random Forest, and Deep Neural Network (DNN) algorithms by comparing their accuracy, specificity, sensitivity, receiver operating characteristic (ROC), and AUC results. We argue that Deep Neural Network (DNN) is most suitable to be used in replacement of CNN due to the fact that the ASD screening dataset contained no visual images or text embedding which does not fulfil the requirement for utilizing CNN. It is evident that CNN is mainly used to analyse and recognize patterns in images and text embeddings of a dataset by utilizing the input data's collocated grids [20][21][22], thus, we replaced CNN with DNN instead.
Missing values are common in data analysis especially when big data from different sources is involved. Missing values or data inconsistency can potentially affect how data is analysed, and thus, it should be addressed by embedding several methods; deletion or data imputation [23][24][25][26]. Data imputation is a method whereby missing values are replaced with meaningful values such as the mean of the available records. This study aims at investigating the effect of ignoring missing values in the ASD dataset to the sensitivity, specificity, and accuracy of each classification algorithm used in this study since there exists a lack of studies being conducted to investigate this matter. Data imputation method has been utilized to address the missing values and to compare the sensitivity, specificity, and accuracy before and after the method was applied to the ASD dataset.

Method
The main purpose of this study is to analyse a set of ASD screening data by using six classification algorithms to help in improving the diagnosis process of ASD in healthcare practices. This section demonstrates the approaches that we used to classify the ASD screening dataset including the classification process, classification techniques, performance matrix, and dataset involved in this study.
Since the output of the training dataset is known which is the screening class, classification technique is the most suitable one to be used in this case. Classication can help to improve screening process and assist at reducing any possible errors caused by inexperienced health practitioners [7,27]. Figure 1 illustrates the process of classifying the dataset by using six chosen classifiers. The process began by classifying the ASD screening dataset in two conditions; with and without missing values, using six classifiers individually with Weka Explorer. As proposed by Tsai and Chang [28], data imputation method was used to deal with incomplete data by replacing the missing values with reasonable values. Once the results were obtained, we furthered the experiment by running all six classifiers simultaneously in Weka Experimenter and the results were analysed and compared for both conditions.

Classification techniques
For the purpose of this study, six classifiers techniques have been chosen to analyse their performances in classifying ASD screening dataset. The classifiers used include Naive Bayes, Logistic Regression, KNN, J48, Random Forest, and DNN. Each of these classifiers have shown the highest performance when tested with other classifiers in previous studies.
Naïve Bayes uses Bayesian techniques to build a simple network in which it is assumed that the probability that one new example belongs to a class based on the assumption that all attributes are self-reliant from each other [29,30]. According to Lewis [31], Naive Bayes algorithm can produce an optimal classifier in different situations where the assumptions are wildly violated. Logistic Regression is a multivariate statistical method which requires less assumption. The method is useful for evaluating the relationship between independent variables and dependent variables as well as for predicting the risk of a disease based on predictor variables built within the model [32]. K-nearest Network (KNN), or known as IBk in WEKA modeler environment, is an instance-based learning algorithm that is capable at classifying many types of datasets as well as capable of performing distance weighting. In KNN, when k samples belong to a type of category and it is found that the samples of the dataset are closely identical to other k samples, the samples will be classified into the category [33]. The classifier will examine the distance of the samples to the sample of the nearest neighbouring training in the feature space [34]. J48 is one of the decision tree techniques implemented from C4.5 algorithm. It consists of four main programmes; the decision tree generator, production rule generator, decision tree interpreter, and production rule interpreter. The decision tree produced by C4.5 is utilized for classification and grouping purposes. This algorithm works by producing a decision tree based on the dataset by recursive division of the data followed by the development of the decision tree using Depth-First strategy [35].
Random Forest, used for classifying and regressing an observation, is one of the decision tree techniques which is constructed by using a different bootstrap sample from the original data. The technique which roots from decision tree algorithm consists of a huge number of individual decision trees which work as an ensemble [36]. Approximately one-third of the data will be left out of the bootstrap sample and not utilized in the construction of the kth tree [37]. Similar to other classifiers that have been previously mentioned, Deep Neural Network (DNN) is one of the types of neutral network algorithms which can also be used for both classification and regression cases. It has been widely used as a tool to classify data in many crucial fields which can be extended from speech recognition to computational medicine [38]. The neural network incorporates the training of neural network with an amplified cross-entropy error function [39].

Dataset
This study utilized a set of ASD screening dataset provided by Dr. Fadi Fayez Thabtah from the Department of Digital Technology, Manukau Institute of Technology, New Zealand 13]. The researcher developed a screening app used to screen the symptoms of ASD in patients and the data of the patients, which have been proven important in screening ASD occurrence 40], were collected and stored in a designated database.
The dataset is made up of 21 attributes, which includes 292 instances or records. The attributes or data fields include age, gender, ethnicity, jaundice status, existence of autism among family member, relation, country of residence, used screening app before, type of screening method, questions about the performance or ability of the patient (A1_Score, A2_Score, A3_Score, A4_Score, A5_Score, A6_Score, A7_Score, A8_Score, A9_Score, A10_Score), screening score, and screening class. The diagnosis result from the screening test is categorized into "1" for a patient diagnosed with ASD and "0" for a patient not diagnosed with ASD and due to the categorization, the dataset is categorized as asymmetric dataset. The dataset contains 90 or 30.82% missing values in the age, ethnicity, and relation attributes. Age is one of the most significant information needed in ASD screening as it assists practitioners to determine the type of early treatment that the patients should be undergoing if they are diagnosed with ASD [13,41,42]. Thus, it is important to address the missing value. Table 1 shows the list of the attribute names and types contained in the ASD screening dataset. When the ASD screening dataset is loaded to WEKA modeler environment, the modeler will recognize the attributes and during the scanning process of the dataset, it will compute descriptive statistical analysis (Table 2) on each attribute in the dataset. The left panel highlighted in green (Fig. 2) shows the list of attributes for the dataset. The modeler shows that the dataset contains 21 attributes and 292 instances in total.

Performance metrics
In this study, the accuracy result of each classifier is used as the performance metrics for the purpose of analysing each classifier's performance more systematically. The AUC metric is also examined for all classifiers used in this study. The metric is used to measure the performance of each classifier over the full range of sensitivities and specificities and is not affected by the trade-off between accuracy and specificity [30].

Confusion matrix
Confusion matrix is used to evaluate the performance of the classification algorithms. It is a visualization tool used to illustrate the accuracy, specificity, and sensitivity of the classifiers in classification process. The matrix illustrates the relationships between the outcome and prediction of classes shown in Table 3.
The description of the confusion matrix shown in Table 2 is shown as follows: • a is True Positive (TP) where it is predicted as having ASD and having ASD in actual situation. • b is False Negative (FN) where it is predicted as not having ASD but having ASD in actual situation. • c is False Positive (FP) where it is predicted as having ASD but not having ASD in actual situation. • d is True Negative (TN) where it is predicted as not having ASD and not having ASD in actual situation.
In data mining, accuracy, calculated using the formula; (TP + TN)/(TP + FP + TN + FN), is defined as the proportion of the total number of data that is correctly classified. It is computed by dividing the total sum true positive and true negative with the total sum of all positive and negative data. Sensitivity or recall or true positive rate, calculated using the formula; TP/ (TP + FN), calculates the proportion of actual positives that are correctly classified, for instance, the percentage of patients that are correctly identified as having ASD. To calculate accuracy from the confusion matrix, the total number of data categorized as true positive is divided with the total sum of true positive and false negative data. Specificity or true negative rate calculates the proportion of actual negatives that are correctly classified, for instance, the percentage of patients correctly identified as not having ASD. The formula used is TN/(TN + FP). Sensitivity is calculated from true negative and false positive data. Specificity also allows practitioners to determine the probability of false alarm.

ROC and AUC
The performance of each algorithm can also be measured by comparing the result of ROC and AUC. ROC plots the classification algorithm's sensitivity and specificity at different classification thresholds in a visual format. Meanwhile, AUC measures the area below the ROC curve which is depicted as the probability that the classification algorithm ranks a random positive data more highly than a random negative data.

Pre-processing and processing measures
The ASD screening dataset contains missing values in three attributes; age, ethnicity, and relation, thus the study tested the classifiers on the dataset with and without missing values to investigate the effect of both conditions on the classifiers' performance result. The missing values were addressed during the pre-processing phase by deploying the most used imputation method which is imputation using mean values. Imputation using mean values is relevant in this study since the missing values in the ASD screening dataset are categorized as missing completely at random (MCAR), thus reducing the biasness to the classification result [43]. Moreover, the imputation method is performed to the dataset as it is easy and less time consuming as well it is suitable for small dataset such as the ASD screening dataset used in this study. We also performed normalization technique to numeric attributes namely age and result by using "Normalize" function on WEKA Explorer. After the pre-processing stage, cross-validation mode was performed to the dataset by splitting the data into  Gender F 84 n/a n/a n/a n/a M 208 n/a n/a n/a n/a Ethnicity Asian 46 n/a n/a n/a n/a Black 14 n/a n/a n/a n/a Hispanic 7 n/a n/a n/a n/a Latino 8 n/a n/a n/a n/a Mid-Eastern 27 n/a n/a n/a n/a Pasifika 2 n/a n/a n/a n/a South Asian 21 n/a n/a n/a n/a Turkish 2 n/a n/a n/a n/a White-European 108 n/a n/a n/a n/a Others 14 n/a n/a n/a n/a Jaundice No 212 n/a n/a n/a n/a Yes 80 n/a n/a n/a n/a Autism No 243 n/a n/a n/a n/a Yes 49 n/a n/a n/a n/a Relation Parent 214 n/a n/a n/a n/a Self 5 n/a n/a n/a n/a Relative 17 n/a n/a n/a n/a Healthcare professional 13 n/a n/a n/a n/a Used_app_before No 281 n/a n/a n/a n/a Yes 11 n/a n/a n/a n/a A1_Score 0 107 n/a n/a n/a n/a 1 185 n/a n/a n/a n/a A2_Score 0 136 n/a n/a n/a n/a 1 156 n/a n/a n/a n/a A3_Score 0 75 n/a n/a n/a n/a 1 217 n/a n/a n/a n/a A4_Score 0 131 n/a n/a n/a n/a 1 161 n/a n/a n/a n/a A5_Score 0 75 n/a n/a n/a n/a 1 217 n/a n/a n/a n/a A6_Score 0 84 n/a n/a n/a n/a 1 208 n/a n/a n/a n/a A7_Score 0 115 n/a n/a n/a n/a 1 177 n/a n/a n/a n/a A8_Score 0 147 n/a n/a n/a n/a 1 145 n/a n/a n/a n/a A9_Score 0 148 n/a n/a n/a n/a 1 144 n/a n/a n/a n/a A10_Score 0 80 n/a n/a n/a n/a 1 212 n/a n/a n/a n/a Result n/a 292 0 10 6.24 2.285 Class/ASD No 151 n/a n/a n/a n/a Yes 141 n/a n/a n/a n/a k-block or a chunk of objects and this process was performed before classification process took place. Once the modeler split the dataset, the data was tested by each classifier in which it was trained to use K-1 block and the process was reiterated for all the blocks. For this study, the ASD screening dataset was split into ten blocks.

Results and discussion
This section presents the results of the classification process when we tested the classifiers using WEKA Explorer and WEKA Experimenter to the dataset in two conditions; with and without missing values by which the results are presented and discussed accordingly.

Testing classifier performance using WEKA explorer
The study tested six different types of classifiers which include Naïve Bayes, Logistic Regression, KNN, J48, Random Forest, DNN, and SVM. As for DNN classifier, we utilized two-layer technique. Table 4 illustrates the performance of the classifiers tested in WEKA Explorer by deploying cross-validation test mode. For classification with missing values, the area under ROC curve (AUC) of each classifier has also been illustrated for comparison as shown in Fig. 3. We furthered the study by classifying the ASD screening dataset through the removal of the missing values in the dataset. This was conducted by using one of the imputation methods which is imputation using mean values. In WEKA modeler environment, the said imputation method is represented as "Replace Missing Value" filter. In the pre-processing phase, the missing values were replaced with the mean value of the attribute that contained missing values, before being classified using the classifiers. The result is organized in Table 5. We also illustrated the AUC of each classifier for comparison in Fig. 4.
From the experiments, all classifiers obtained different results when classifying the ASD screening dataset with and without missing values. Naïve Bayes classifier produced slightly different AUC score when classifying the dataset with and without imputation, dismissing the previous studies [44,45] which claimed that Naïve Bayes is insensitive to missing values. Based on the classification result of all the classifiers being tested in Table 4, when missing values were ignored, J48, Random Forest, and SVM have shown 100% performance as compared to other classifiers in terms of the accuracy, sensitivity, and specificity. However, J48 only required 0.08 s to classify the dataset as Further, we compared the ROC score to study the performance of each classifier. ROC enables the demonstration of the diagnosis ability of a classifier in which the main goal is to have the curve closer to the value 1 on the Y-axis of the graph which indicates the ability of a classifier to classify a given dataset accurately. Simply put, the closer the curve to the value 1, the better the performance of the classifier at classifying a dataset. J48, Naïve Bayes, Random Forest, and SVM produced the best ROC curve which indicates that the classifiers can predict ASD occurrences accurately as opposed to Logistic Regression, KNN, and DNN when classifying the dataset in both conditions. From the ROC curve, we further illustrated the AUC of each classifier shown in Figs. 3 and 4. The experiment showed that when the dataset with missing values is classified, J48, SVM, and Random Forest produced the best AUC with 1.000 score followed by Naïve Bayes (0.9996), Logistic Regression (0.9893), DNN (0.9282),  The result indicates that the classifiers that could not produce a perfect score of 1.000 for AUC are unable to predict ASD occurrences accurately, this in turn, may affect the life of a patient being wrongly diagnosed.

Verifying classifier performance using WEKA experimenter
We further the experiment by classifying the dataset with and without missing values by utilizing WEKA Experimenter in order to validate the results produced by the classifiers when tested in WEKA Explorer environment. WEKA Experimenter allows more than one classifier to be tested simultaneously to classify the dataset. We tested the dataset by running tenfold or 10-epoch crossvalidation test mode. Corrected Paired T-Test mode was utilized to verify the performance of each classifier by comparing the accuracy and AUC results of each of the classifiers when tested simultaneously. The test works by comparing each classifier in pairs and making reasonable assumptions with regards to the distribution of the results collected. The test used 0.05 two-tailed confidence level and the result is shown in Table 6.
Based on the test conducted using WEKA Experimenter, 600 data were loaded since each classifier was evaluated 100 times (tenfold cross validation multiplied by 10 repetitions). We compared the accuracy and AUC score of each classifier when tested in both WEKA Explorer and WEKA Experimenter. The result shows that J48 and SVM classifier outperformed other classifiers in terms of its accuracy and AUC readings; 100% and 1.00 respectively. On the other hand, the accuracy of Naïve Bayes and Random Forest decreased while Logistic Regression, KNN, and DNN improved their accuracy especially DNN that produced 7.5337% of improvement. We furthered the test by using J48 as the test base since J48 has better training runtime than SVM.The result shows that no classifiers outperformed the test base, except Random Forest, which generated 1.00 of AUC result, akin to J48.
In addition, the accuracy of Logistic Regression (95.51%), DNN (94.52), and KNN (88.42%) as well as the AUC of KNN (0.89) and DNN (0.98) have shown to be significantly different than that of J48.
Based on the test using both WEKA Explorer and WEKA Experimenter, it is evident that J48 is the best classifier among the other five classifiers with 95% of confidence level when tested with the dataset containing missing values. This shows that J48 is capable at handling   [46] tested Naïve Bayes, J48, and Random Forest to classify a non-medical dataset and the classification results were compared. Based on the result, they found that Naïve Bayes produced the best accuracy in classifying the instances as opposed to this study. Thus, we can suggest that a classifier's classifying performance is affected by the type of dataset as well as the number of instances involved in the experiment. We then addressed the missing values by using imputation method and tested the dataset in WEKA Experimenter. We tested the dataset without missing values by deploying the same method through which we used Corrected Paired T-Test by using 0.05 significance level and compared the performance of each classifier in terms of the accuracy and AUC score. Since J48 turned out to be the best classifier among other classifiers being tested, we used the classifier as the test base for the experiment. The result of the accuracy and AUC of the classifiers is illustrated in Table 7.
As shown in Table 6, we found that Random Forest and SVM, both of which have 100% accuracy and 1.00 AUC score, have outperformed other classifiers when missing   We also found that Naive Bayes improved its accuracy when missing values were replaced with the mean value of the attribute. This study has somewhat debunked the study which showed that Naïve Bayes has low sensitivity towards the existence of missing values in a dataset [47]. However, a further study is sought to test its sensitivity using other datasets.

Confusion matrix of the best classifier
As mentioned previously, confusion matrix is a visualization tool that allows researchers to illustrate the accuracy, specificity, and sensitivity of a given classifier. From the experiment, confusion matrix is produced to illustrate the accuracy, specificity, and sensitivity of J48 classifier, the best classifier among other classifiers being tested. The confusion matrix of J48 classifier is shown in Figs. 5 and 6 for both conditions; ASD screening dataset with and without missing values, respectively. The confusion matrix which is produced when J48 is used to classify ASD screening dataset with and without missing values illustrates that in both conditions, J48 could classify 151 instances with 'yes' class category and 141 instances with 'no' class category correctly. The matrix also shows that neither false positive nor false negative classification was produced by J48, which proves that J48 can perform better in classifying the database in both conditions as compared to other classifiers especially Random Forest which also produced the same accuracy score as J48 in WEKA Explorer but can only produce 99.97% of accuracy when tested in WEKA Experimenter environment when missing values were present in the ASD dataset.

Future work and conclusion
Not many studies have utilized classification algorithms to particularly analyse ASD screening dataset. From the literature review, most researchers utilized other most common datasets such as breast cancer and heart disease datasets, thus there exists difficulties in analysing and comparing the performance of classifiers being tested to ASD screening dataset. This is important in order to assist in selecting the best classification methods for screening and diagnosing a patient with ASD, thus we attempted to fill in the gap by analysing and classifying ASD screening dataset by using six classifiers. Based on the result obtained through the tests with both Weka Explorer and Weka Experimenter using tenfold cross validation, we can conclude that J48 outperformed other classifiers in terms of the time taken to classify the class as well as the accuracy and AUC produced by the classifier when tested to the ASD dataset with and without missing values. On the other hand, KNN performed poorly in classifying the dataset as compared to other  classifiers even though the classifier took 0.00 s of runtime to classify the dataset. The accuracy produced by J48 may assist health practitioners to make better decision when analysing datasets in both circumstances, with and without missing values. Making a right decision at the right time is imperative in crucial industries especially health-related field for it to perform effectively and efficiently as well as to reduce human errors which can affect the life of a patient. Moreover, we compared our results with the previous works mentioned in this paper and found that a classifier's classifying performance is affected by the type of dataset as well as the number of instances involved in the experiment. However, a specific study is required to test the classification algorithms to classify other health related datasets with different volume of instances in order to investigate the effect of the volume on the performance of the algorithms. Future work can include utilizing other classification algorithms such as Simple Cart, Ada Boost, Logit Boost, Bagging, and Decision Stump. This study can also be further extended by deploying multi-modal deep learning to predict ASD occurrences as well as the abnormalities of other related neurological diseases such as amnestic mild cognitive impairment and multiple sclerosis [48,49]. Simulating missing values existing in other significant attribute fields should also be considered for future work. Moreover, other imputation methods are sought to be used for addressing the missing values as the current imputation method used in this study may lead to inaccuracy and uncertainty when testing the dataset. Lastly, an extensive study is suggested to be carried out to further investigate the kappa value of each classifer being tested in this study.


Bounding Boxes Are All We Need: Street View Image Classification via Context Encoding of Detected Buildings

Street view images classification aiming at urban land use analysis is difficult because the class labels (e.g., commercial area), are concepts with higher abstract level compared to the ones of general visual tasks (e.g., persons and cars). Therefore, classification models using only visual features often fail to achieve satisfactory performance. In this paper, a novel approach based on a"Detector-Encoder-Classifier"framework is proposed. Instead of using visual features of the whole image directly as common image-level models based on convolutional neural networks (CNNs) do, the proposed framework firstly obtains the bounding boxes of buildings in street view images from a detector. Their contextual information such as the co-occurrence patterns of building classes and their layout are then encoded into metadata by the proposed algorithm"CODING"(Context encOding of Detected buildINGs). Finally, these bounding box metadata are classified by a recurrent neural network (RNN). In addition, we made a dual-labeled dataset named"BEAUTY"(Building dEtection And Urban funcTional-zone portraYing) of 19,070 street view images and 38,857 buildings based on the existing BIC GSV [1]. The dataset can be used not only for street view image classification, but also for multi-class building detection. Experiments on"BEAUTY"show that the proposed approach achieves a 12.65% performance improvement on macro-precision and 12% on macro-recall over image-level CNN based models. Our code and dataset are available at https://github.com/kyle-one/Context-Encoding-of-Detected-Buildings/

I. INTRODUCTION
U RBAN land use records how people use the land with social-economic purposes, such as residential, commercial, and recreational purposes [2]. Land use classification using satellite images have been extensively studied in remote sensing community. With the rise of geo-data commercial services (e.g., Google maps) and crowdsourced projects (e.g., OpenStreetMap) [3], urban spatial data of different modalities are used [4]. As their representative, Google street view (GSV) [5] provides abundant street-level details which have been increasingly used in urban land use classification. Street view images are accurately geo-located, updated regularly, , which led to wrong predictions. Right: By using a trained detector, the proposed approach obtained bounding boxes of buildings in the input image, which contain semantic labels and their context information such as the cooccurrence patterns and layout. Correct results were obtained by encoding these information using the proposed "CODING" algorithm and an RNN. easy and free to access. Moreover, they contain richer visual information which makes it easier to be distinguished (see Fig. 2). Therefore, visual models that perform well in common computer vision tasks, such like CNNs have recently been widely used to extract visual features of street view images for urban land use and urban functional zone analysis [1], [2], [6]- [12]. However, the performance so far has been less than satisfactory partly due to the high-level abstractness of urban land use labels, which makes it hard to represent the concepts directly using visual features. In addition, street view images contain many of the same visual elements (e.g., sky and ground) which interfere with distinguishing different usages of land. When using the whole images directly, the most distinguishable visual elements are underutilized.

A. Motivation
We consider street view image classification as a finegrained outdoor scene analysis problem. The keys of this task are, firstly, acquiring the most significant objects in street view images aiming at land use, and then, effective modeling of their contextual information. Based on the above viewpoints, a "Detector-Encoder-Classifier" framework is proposed to replace the common CNN based architecture, as shown in Fig 1. On the first point, significant objects change with specific tasks. Buildings are the main places where people engage in social and economic activities. Urban functional zones also consist mostly of buildings of different categories. Therefore, individual buildings with fine-grained labels should be considered as "significant objects" in street view images for task of urban land use and functional zone analysis. Fig. 3 demonstrate the importance of "significant objects".
Unfortunately, existing open datasets with outdoor scene for common visual tasks [13]- [16] and specific visual tasks (e.g., autonomous driving [17], [18]) are all lack of systematic, finegrained class definition for buildings. As a milestone work for street view image classification, BIC GSV [1] classifies individual buildings into 8 categories. However, its image-level annotation may cause ambiguity when a street view image contains multi-class buildings. In fact, currently there is no dataset using object-level annotations of fine-grained multiclass buildings for street view images.
On the second point, the dominant model for street view image classification is CNN. Recent works either use CNNs directly for end-to-end image-level classification from the same source [1], [6], [8], or CNNs with a two-stream network structure to fuse visual features of images from different sources [2], [7], [9], [10]. As we mentioned, classifying highlevel abstract labels directly using visual features may lead to performance bottlenecks. To break the bottlenecks, not only "visual semantics" is needed, but also "visual syntax". The  former can be obtained by encoding the visual features. The later should be learned from the context releations of formers.

B. Contributions
The contributions of the paper lie in three aspects as follows.
• Based on BIC GSV, a dual-labeled dataset named "BEAUTY" (Building dEtection And Urban funcTionalzone portraYing) with a 19,070 street view images and 38,857 individual buildings by combining automatic labels acquisition from OpenStreetMap (OSM) and expert annotation. It can be used not only for street view image classification aiming at urban land use analysis, but also for multi-class building detection. We also provide baselines for image classification and object detection running on this dataset. • Based on BEAUTY, a "Detector-Encoder-Classifier" framework is proposed to replace the common CNN based architecture. As shown in the right column of Fig. 3(b), without "looking" at the whole image, our approach can infer the land use by only using the bounding boxes of detected buildings. In our approach, object detector is regard as a plug-and-play module that can be arbitrarily replaced, which allows the performance of our approach to easily improve synchronously with the improvement of object detection technology. • We explored the effect of co-occurrence pattern of multiclass buildings and, further, their spatial layout on urban functional zone analysis. Based on this, we proposed "CODING" (Context encOding of Detected buildINGs) algorithm to encode the contextual information of bounding boxes into metadata which make it easier to further encoding and classifying using RNN or other models.

C. Section Arrangement
The rest of the paper is organized as follows. In Section II, we review related work on land use classification using streetlevel images and current research progress on scene context modeling. Section III introduces our dataset "BEAUTY". The proposed approach is expatiated in Section IV. Section V shows the experimental setup, results and discussions. Section VI concludes the paper.

II. RELATED WORK
Urban land use classification has been a growing research field as more data from different sources are available. For example, satellite and aerial images data have been mostly used by the remote sensing community [19], while streetlevel images were mainly studied by the computer vision community [4]. In the latter, social media images and street view images are the two main sources. Both of them are often referred to a scene analysis problem. In this section, we briefly review the research progress on land use classification using street-level images and context modeling for scene analysis.

A. Land Use Classification Using Social Media Images
Leung and Newsam [20] first used social media images from Flickr for land use classification. They used the bag of visual words (BOVW) with a soft-weighing scheme to represent image features and then classified them into 3 categories with support vector machine (SVM). Zhu and Newsam improved Leung's work by using two groups of Flickr images: indoor and outdoor [21], and replacing BOVW features with pretrained CNN features [15]. Antoniou et al. [22] extracted geotagged images from Flickr, Panoramio and Geograph for an area of London, and discussed their usefulness for land use classification. Based on Antoniou's work, Tracewski et al. [23] used Places205-AlexNet [15] to classify social media images with volunteered geographic information (VGI) for land use classification. Zhu et al. coupled images from Google Places and Flickr with a two-stream CNN to predict the land use [7]. By using ResNet101 as backbone of each branch, they reported 49.54% classification accuracy on 45 categories. Hoffmann et al. [24] classified building instance into 5 land use categories by training a VGG16 using Flickr images.
Social media images provide more street-level details for land use classification. However, they also have shortcomings. First of all, they are often not accurately georeferenced. What's more, they usually portray highly personalized content (e.g., touristic viewpoints, selfies or zoomed objects) from a subjective, fickle perspective, rather than urban objects from a relatively objective, fixed perspective. Last but not least, they tend to cover the city unevenly (e.g., most images are taken in touristic areas). These problems make such street-level images less suitable for reliable urban land use mapping.

B. Land Use Classification Using Street View Images
Services like Google Street View (GSV) make it is possible to acquire street-level images with urban objects shot from a relatively objective perspective, which are accurately geolocated, updated regularly and densely available in many cities all over the world. Recently GSV is being increasingly used in land use classification. Movshovitz et al. [26] used CNN to classify store fronts into 13 business categories from single GSV images. Kang et al. [1] classified urban buildings into 8 categories using GSV images with labels from OSM. Their model predicts one label for each image corresponding to one urban building. Srivastava et al. fused multiple GSV images of a building using a Siamese-like CNN [25] and showed an overall accuracy of 62.52% on 16 OSM label prediction. Noticing that each land use category is made of different objects present in a set of images, they then extended their approach to multi-label prediction [6] to avoid the ambiguity caused by single-label image classification.
Researchers also try to fuse street view images with overhead images by multi-modal strategies. Combination of both modalities was initially used in image geo-localization. Lin et al. [27] matched HRO from Bing Maps with street view images from Panoramio by using four handcrafted features and adding land cover features as the third modality. To extend their approach, they used a Siamese-like CNN to learn deep features between GSV images and 45-degree oblique aerial images [28]. Workman et al. [29] fused overhead images and GSV images by an end-to-end deep network which outputs a pixel-level labeling of overhead images for three different classification problems: land use, building function and building age. They reported a top-1 accuracy of 77.40% and 70.55% for land use classification task on Brooklyn and Queens datasets respectively. Zhang et al. [30] combined airborne light detection and ranging (LiDAR) data, HRO and GSV images for land use classification. In their study, thirteen parcel features were chosen as input variables in a Random Forest classifier which achieves an average accuracy of 77.50%. Cao et al. [2] used images from Bing Maps and GSV for land use segmentation with a two-stream encoder and one decoder architecture which evolved from SegNet [31]. Hoffmann et al. [9] used a twostream CNN model for building functions classification. They predicted four class labels namely commercial, residential, public and industrial for overhead images by fusing deep features of overhead images and street view images. Their model increases the precision scores from 68% to 76% with a decision-level fusion strategy. Srivastava et al. [10] extend their early work [25] to a multi-modal strategy by leveraging the complementarity of overhead and street-level views. They deal with the situation of missing overhead imagery by using canonical correlation analysis (CCA) based on their twostream CNN model. By using VGG16 as the backbone, their model achieves an overall accuracy of 73.44% and an average accuracy of 70.30%.
Although the usage of multi-modal strategies gets better results to some extent, the performance so far has been less than satisfactory partly due to the high-level abstractness of urban land use labels which were hard to be abstracted directly using visual features. To break the bottlenecks, some new point of view is needed.

C. Context Modeling for Scene Analysis
Image context contains a wealth of information about how objects and scenes are related. Cognitive science studies [32], [33] have shown the importance of contextual information in human visual recognition. Typical contextual information including global context [34], visual context [35], object cooccurrence [36] and layout [37], are now exploited to improve the performance of various visual tasks. Pathak et al. [38] proposed a context encoder to generate the contents of an arbitrary image region conditioned on its surroundings. Choi et al. [39] present a graphical model that combines different sources of context information to detect out-of-context objects and scenes. Izadinia et al. [40] encoded the scene category, the context-specific appearances of objects and their layouts to learn scene structures. Chien et al. [41] built a CNN to predict the probability of observing a pedestrian at some location in image. Wang et al. [42] used a variational auto-encoder to extract the scale and deformation of the human pose and thus predict opportunities of interaction in a scene. Qiao et al. [43] proposed an encoder-generator model that encodes the properties of input objects and generates a scene layout representing the scene context. We consider street view image classification as a fine-grained outdoor scene analysis problem. The proposed context encoder will be detailed in Section IV.

III. DATASET
As we mentioned in Section I-A, currently there is no dataset using object-level annotations of fine-grained multiclass buildings for street view images. Most existing street view datasets use the single-label image-level annotation which contains only global semantics but no descriptions of content or context. Srivastava et al. [25] used a multilabel image-level annotation dataset namely "BAG" 1 which contains object co-occurrence information that could be used to describe contextual relations of the image scene to some extent. However, the labels are for individual buildings such as "office" and "shop", which lack global semantics of the land use. Furthermore, image-level annotation cannot provide spatial information of objects (e.g., size and position). Thus it contains no richer context information such as layout.
To explore the context relations between street view scene and urban objects in it, a street view image dataset with a dual-label system is made based on the existing BIC GSV dataset [1]. On one hand, each image has a land use label to describe the functional zone it portrays, such as "commercial". On the other hand, each urban object (mostly individual building) in the image is annotated by a bounding box with an object-level label such as "retail". Thus, the proposed dataset named "BEAUTY" can be use both in land use classification task and in individual building detection task.
BIC GSV obtained geo-tagged GSV images located over several cities of the US and Canada (e.g., Montreal, New York City and Denver) and their associated ground truth building labels extracted from OSM. BEAUTY makes the following improvements over BIC GSV.
• The remaining invalid samples are further removed.
Although BIC GSV has removed some outliers with VGG16 trained on Places2 [15], some invalid samples were still found during the manual inspection. As shown in Fig. 4(a), we further remove four types of remaining • Object-level annotations are given for each building in an image. In combination with the building labels 2 automatically obtained from OSM, we manually annotate each individual building in each image under the guidance of architecture experts. In object-level annotations, we use the 8 class labels used in BIC GSV, namely apartment, church, garage, house, industrial, office building, retail and roof. Object-level annotations avoid the ambiguity when buildings in different classes are in the same image and also afford the layout information of buildings in the same scene of land use. An example is shown in Fig. 4(b).
• Image-level labels are further abstracted into land use categories. In combination with the land use labels 3 automatically obtained from OSM and the Land Based Classification Standards (LBCS) Function Dimension with Descriptions 4 , we manually annotate each image under the guidance of urbanist. We fuse the OSM land use labels and LBCS urban function descriptions into 4 highly compact classes namely commercial, residential, public and industrial, which have been used in [9], because such a classification has a very high value to urban geography being correlated with socio-demographic parameters such as population density and income. The correspondence between the proposed four land use categories and OSM, LBCS labels is shown in TABLE I. The BEAUTY dataset consists of 19,070 street view images with 38,857 individual buildings. As can be seen from Fig. 5, both the sample distributions of land use classes and building classes are long-tailed, which are in line with the situation in the real world. Fig. 6 shows samples of proposed dataset. Fig. 7 shows the pipeline of proposed approach. The inputs are street view images and the outputs are their predicted land use categories which would be mapped to the geographic information systems according to their geo-location.

NUMBERS OF INDIVIDUAL BUILDINGS
(b) Sample numbers of each building class. As we mentioned in Section I-A, the first key of our task is acquiring the most significant urban objects in street view images. The building detector plays the key role to do it. Two off-the-shelf detectors were used in this paper namely Faster R-CNN [44] and Cascaded R-CNN [45]. The detectors were trained using the object-level ground truth of training samples in BEAUTY. The outputs of the detector are bounding boxes of each building with their classes and confidence scores, which would be transferred into metadata through the context encoder. In addition to being an intermediate module of the street view classification task, we also consider building commercial industrial public residential   7. Pipeline of proposed approach using a "Detector-Encoder-Classifier" framework. The core algorithm "CODING" encodes the input bounding boxes into scene metadata containing contextual information. detection as a separate task and conduct the corresponding baseline tests. This part will be detailed in Section V-B. As shown in Fig. 7, two different kinds of contextual information are optional in the proposed "CODING" module, which are co-occurrence patterns and layout.

A. Context Encoding Using Object Co-occurrence Only
Outputs of a detector are bounding boxes of detected object regions, each of which consists of the following data: predicted confidence score, class label and position in the image. The class and confidence score are concrete representations of semantic information. We integrate them together into a feature vector with the form of the hot-one vector of predicted class whose none-zero value was replaced by the confidence score. These "semantic vectors" are used directly to the classification task. The position data are often with the form of To put it simply, we first select the bounding box with highest confidence score and largest size (Line 5) to be the leading box (Line 10), and then ascending sort the rest ones (Line 16) by the centroids distance (Line 14) between them and the leading box. Finally, sequences of vectors with hotone like form (semantic structure), and their order (syntax structure) constitute the scene metadata.
To further encoding and classifying the metadata obtained from "CODING", two RNN architectures are used, namely last-layer-concatenated single-directional RNN and all-concatenated bidirectional RNN (BRNN) [46], both with two hidden layers. The inputs of RNNs are the semantic vectors with the size of 8, representing the detected bounding boxes of 8 class buildings in an image. The size of a hidden layer neuron h i is 16 and the dimension of the parameter matrix A in the first hidden layer is 16 × (16 + 8). The basic units follow a simple RNN structure. It can also be replaced by a gated recurrent unit (GRU) [47] or a long short-term memory (LSTM) [48] unit, which will be compared in Section V-C.
In the first architecture, all hidden neurons in the last layer are concatenated. In order to reduce the weights of zero vectors generated by padding, vectors of the input metadata should be arranged in reverse order. The concatenation layer ensures that no feature of a single bounding box would be forgotten by the directionality of the RNN. Well in the BRNN architecture, all neurons in both hidden layers are finally concatenated to connect to a full connection (FC) layer and output the predicted probability of four classes of land use scenes after a softmax operation.

V. EXPERIMENTS AND ANALYSIS
In order to verify the validity of the proposed approach, we ran it on BEAUTY dataset which has been introduced in detail in Section III. In this section, we will first conduct baseline tests for the tasks of street view image classification and building detection on this dataset, then compare the performance of proposed approach with baseline, and draw some useful conclusions through analysis.

A. Data Preparation and Experimental Setup
We randomly selected 75% of the samples from each category as the training/validation set and the remaining 25% as the test set. The training/verification set was then randomly divided into training set and verification set according to the ratio of 9:1. From Fig. 5 we can learn that there is a class imbalance problem [49] in BEAUTY. To reduce the impact of class imbalance and achieve better performance, we carried out class rebalance for the training samples using a random minority oversampling strategy. Specifically, the samples of public and industrial were expanded by 2 times and 2.5 times respectively. Since all training samples will be flipped randomly in the horizontal direction in the data augmentation stage before training, only random copy is needed for minority oversampling, which is a common strategy in the industry. We also tried random geometry transformation and random color jittering [50] for training data augmentation. Unfortunately, the performance is not as good as random horizontal flip considering both the tasks of street view image classification and building detection. The sample distributions after rebalance are shown in Fig. 8. The total number of training images and individual buildings are 16,871 and 32,439 respectively after class rebalance before data augmentation. Compare Fig. 8 with Fig. 5 we can learn that not only the image-level samples of land use classes are rebalanced, but also the object-level samples of building classes.
All experiments are based on the same hardware and software conditions as follows. GPU: GeForce GTX 1080 × 2; OS: Ubuntu 18.04.3 LTS; CUDA Version: 10.0.130; PyTorch Version: 1.4.0 for cu100; TorchVision Version: 0.5.0 for cu100. We set the number of RNN input l = 25, which ensures that most bounding boxes are involved and avoids too much zero vectors. The pre-trained models and the training hyperparameters will be presented in detail in later sections. All the results were averaged after 10 runs.

B. Baselines
To facilitate the evaluation of model performance on street view image classification and building detection on BEAUTY, we selected the corresponding baseline models for both tasks. Considering the leading role of CNN-based end-to-end model in image classification and object detection tasks in recent years, we chose the most representative and most widely used ResNet [51] model and two detectors based on it as the baseline models.
1) Baseline Test for Street View Image Classification: For the task of street view image classification, ResNet50 and ResNet101 are selected as the candidate baseline models. We finetuned the pre-trained models 5 on BEAUYT with the learning rate of 0.01, which was multiplied by a factor of 0.1 after every 10 epochs. The training was pursued for 100 epochs with Adam [52] as an optimizer. The training and validation losses are drawn in Fig. 9.
Although the training set has been rebalanced, serious class imbalance remained in the test set of BEAUTY. Therefore, as a commonly used metric for classification, overall accuracy is not suitable for the evaluation on our dataset. Instead, the macro-average of the per-class metrics are used, namely macro-precision (M-P), macro-recall (M-R) and macro F1-Score (M-F1) [53]. As shown in TABLE II, the performance of ResNet50 beats ResNet101 in all macro-average metrics. Thus, ResNet50 is selected as the baseline model for street view image classification on BEAUTY. The confusion matrices in Fig. 10 also show that ResNet50 performs better than ResNet101 in all other categories with the exception of public.
2) Baseline Test for Building Detection: For the task of building detection, Faster R-CNN [44] and Cascaded R-CNN [45] with the backbone of ResNet50 and ResNet101 are selected as the candidate baseline models. We finetuned the 5 https://pytorch.org/docs/stable/torchvision/models.html  pre-trained models 6 of MMDetection [54] using their default hyperparameters on BEAUTY. Part metrics used in COCO 2020 Object Detection Task 7 are used and extended as our metrics for building detection, which are "average precisions" (AP) at "intersection over union" (IoU) of different values. For example, AP IoU=.50:.05:.95 refers to the AP of all classes when the detections and ground truth bounding boxes were matching according to the least IoU value to be 0.5 to 0.95. It is a relatively strict metric, because a high IoU lower limit represents a high requirement for the position accuracy of the prediction box. In contrast, AP IoU=.50 is a metric with relatively loose requirement for position accuracy. In order to compare the effectiveness of visual feature extraction between the detector and the end-to-end classifier, we extend this set of metrics to AP IoU=.00 , which means that the location of an object detection is not considered, but only whether its category is correct for evaluation. In essence, this metric is equivalent to the macro-precision of a multi-label classifier.
The performances of candidate detectors with candidate backbones on AP IoU=.50:.05:.95 to AP IoU=.00 are shown in TABLE III, where Fa-50 refers to Faster R-CNN with the backbone of ResNet50, and Ca-101 refers to Cascaded R-CNN with the backbone of ResNet101. As the metrics became looser, the detectors scored higher. When consider the accuracy of bounding box position, Ca-101 was the best detector. In contrast, Fa-50 becomes optimal when only the accuracy of the class prediction is considered. We select Ca-101 to be the default detector for our system because our approach encodes the layout of buildings by using both the class and position information. If not specified, the detectors used in the subsequent experiments are all Ca-101. TABLE III is also considered as the baseline for an independent visual task namely multi-class building detection on BEAUTY. Compared with the best M-P (69. 16) in TABLE II, the best AP (79.33) in TABLE III is significantly improved with the same backbone CNN architecture (ResNet50). Although this comparison is not rigorous, we can still roughly observe that the individual buildings in images are easier to be visually characterized and abstracted more effectively than the whole street view image by the same visual feature extractor. This conclusion is the cornerstone of the superiority of our approach over image-level end-to-end CNN models. More details about the effectiveness of visual extraction are presented in Section V-D1.

C. Comparison of Different Settings
After Ca-101 is selected as the default detector, Section IV provides two contextual encoders, namley the co-occurrence encoder and the layout encoder and two RNN network structures, namely single-directional RNN and bidirectional RNN, with three basic network units, namely simple-RNN unit, GRU and LSTM unit. In the following sections, we will discuss these options and try to find the best combination.
1) Performance of Co-occurrence Encoder: TABLE IV shows the performance of the co-occurrence encoder combined with different RNN classifiers. For classifiers using simple-RNN and LSTM units, the single-directional structures are better than the bidirectional ones. The reverse is true when GRU is used. Simple-RNN units in both structures clearly outperform the others, which makes it to be the default network unit. The best combination belongs to simple-RNN units with single-directional structure, which is regarded as the best classifier for co-occurrence encoder. 2) Performance of Layout Encoder: TABLE V shows the comparison between co-occurrence encoder and layout encoder combined with single-directional and bidirectional structure using simple-RNN units. Layout encoder clearly beats cooccurrence encoder, indicating that the spatial arrangement of the building reflects a certain structural context and is useful for distinguishing street view images with different types of land use. For layout encoders, the structure of RNN does not matter much. This also indicates that the spatial arrangement of buildings has a certain robustness for distinguishing different land use scenes. 3) RNN Training Using Ground Truth Bounding Boxes: From Fig. 7 we know that our RNN is trained by the bounding boxes output from detectors. Why don't we use the ground truth bounding boxes to train the RNN and use outputs of detectors during test stage? TABLE VI shows the comparison between using and not using the ground truth bounding boxes during training stage. The results are disappointing.
The "standard answer" seems to be helpless might due to the mismatch during training and test stage. Comparing the co-occurrence coding and layout coding, it can be found that the influence of training-test mismatch on the latter (0.89% in M-F1) is significantly lower than that on the former (6.42% in M-F1). This observation once again demonstrates the robustness of spatial structure. So far, the optimal performance of the proposed approach is generated by layout coding combined with bidirectional RNN structure.

D. Comparison with Baseline
Finally, the upper limit and the optimal performance of proposed approach and baseline are compared in TABLE VII. In Section V-C3, the poor performance has been shown when using ground truth bounding boxes as training samples but the outputs of a detector as test samples, due to the trainingtest mismatch. How about we use the ground truth bounding boxes also in the test stage? It is impossible for a classification system to know some intermediate results of test samples in advance (e.g., ground truth bounding boxes of buildings in the test sample images), but the hypothesis could help us to find out the performance upper limit of the proposed encoder-classifier system. Upper limit means that a perfect detector is used, whose outputs during training and test stages are ground truth bounding boxes which make the proposed encoder-classifier system perform best.
The M-F1 of the perfect detector shows a 12.45% higher than the current optimal combination of the proposed approach, which means that our approach has a lot of room to improve with the progress of object detection. On the other hand, the M-F1 of the proposed approach presents a 12.32% higher than the baseline (ResNet50 image classification model), which is a significant improvement. More details could be obtained in the confusion matrices of the four classes of land use scenes (Fig. 11). The category with the most room for improvement of the proposed method is industrial (16%). While the category with the most improvement over the baseline is public (39%). These are discussed in more details in the following parts. 1) Typical Case Analysis in Visual Feature Extraction: In this part, we try to explain why the significant improvement by proposed approach over common image-level end-to-end visual models in terms of the effectiveness of visual feature extraction. In order to demonstrate the effectiveness of visual features more intuitively, we use visual feature heatmap [55], which is often used for interpretability analysis of neural networks. For baseline (ResNet50), feature maps before the last average pooling layer are used to generate the heatmaps. The regions that contribute to the prediction of each class are marked by warm color regions in the heatmap of the class. For the proposed approach, the approximate heatmaps are generated by the outputs of detector (Fa-50). For each bounding box b i , its approximate heatmap is assumed to be a two-dimensional Gaussian distribution that described by (1), where T x,y is the temperature of point (x, y) in an image, w i , h i are the width and height of bi and (x i0 , y i0 ) are the center coordinates of b i . Since the detector was not directly used for scene classification, we overlaid and normalized the heatmap of each detection to show the regions that were potentially helpful for the final classification. Heatmaps of typical cases are shown in Fig. 12.
We chose typical cases from the categories with the most improvement over the baseline namely public (39%) and industrial (5%) to do an in-depth analysis. Fig. 12(a) shows a case of public (image ID: public 383 in BEAUTY). The activated regions of the heatmap for the correct class (public) include large areas of sky and ground. While the activated regions of the heatmap for the prediction (residential) do not cover the whole buildings and miss the cross of the church in left, which carries key information about land use. It also contains lots of areas of ground. In contrast, the heat map generated by detector covers all key regions tightly. A similar situation can be clearly demonstrated in the industrial example ( Fig. 12(b)).  Fig. 11. The confusion matrices of upper limit (left), optimal performance of proposed approach (middle) and baseline (right).  2) Typical Case Analysis in Context Information Extraction: Cases in previous part demonstrate that the proposed approach can obtain more effective visual features than CNNbased image-level end-to-end model by using detectors specially trained for buildings. Is the good performance of the proposed approach entirely dependent on the detector? In this part, several cases will show that the proposed encoder and RNN-based classifier can obtain correct scene classification according to the learned context information, even if the detector incorrectly predicts the class of some buildings. Outputs of the detector are firstly encoded by the proposed "CODING". Then a growing number of bounding boxes with high confidence scores have been tampered with as the ones of wrong categories. During this process, the results of the final scene classification are observed all the time. Cases are shown in Fig. 13.
For illustration purposes, only bounding boxes with a confidence score greater than 0.4 are shown. The actual number of detections of the first case "commercial 5548" is 12, showing only 7 in Fig. 13(a). Parts of the bounding boxes overlap, such as "retail: 0.95" and "office building: 0.56" on the far right, and "office building: 0.56" and "garage: 0.50" next to them. Several bounding boxes are tampered with to the wrong class one by one. When two bounding boxes were tampered with (office building: 0.56 → apartment: 0.56, office building: 0.89 → garage: 0.89), the proposed approach could still make correct prediction about the scene class. When the third bounding box was tampered with (office building: 0.96 → house: 0.96), the prediction jumped from commercial: 0.82 to residential: 0.88 without a gradient. In the second case "residential 5173", there are some errors in the original detections. The ground truth of the largest bounding box in the lower left corner  is garage, but the detection is apartment: 1.00. The ground truth of the small bounding box on the left is apartment, but the detection is garage: 0.94. This kind of detection errors could cause minor changes to the layout encoding and will be ignored in RNN-based classifier. And since the co-occurrence relationship between building classes does not change, it did not affect the final classification result (residential: 1.00). In Fig. 13(b), class of the detections are tampered with one by one from right to left. The prediction jumped from residential: 0.98 to public: 0.98 after four bounding boxes were tampered with. Some bounding boxes that were not drawn because their confidence scores were less than 0.4 still contributed to the context relationships such as co-occurrence and layout, which resulted in the prediction of scene class being maintained when three detections were tampered with. The last straw was the manipulation of the bounding box with the largest size and confidence score (apartment: 1.00 → church: 1.00).
The case analysis above can give a glimpse of why the proposed approach achieved better performance over imagelevel end-to-end CNN models such as ResNet50. The general conclusions are given in Section VI.

E. Use on Open World Street View Image Data
To further verify the performance of the proposed approach on an open world data set, land use maps of Calgary are generated using open world GSV images provided by [1]. Land use maps of Calgary based on 6,124 street view images are shown in Fig. 14. Geo-tagged street view images were classified by the proposed approach. The results were then drawn on CesiumJS 8 according to the geographical locations of the input images. Four land use classes residential, commercial, industrial and public are marked by dots of blue, red, purple and yellow respectively. Regions where dots with the same color clustered in a city-scale map are zoomed in to see if the classification is correct.
Generally speaking, the distribution of residential area, commercial area and industrial area in Calgary is relatively balanced. The commercial areas (red dots) are relatively concentrated, while the other two are scattered. In Fig. 14, a commercial area, an industrial area and a residential area that was shown as red, purple and blue dots cluster respectively were zoomed in. As can be seen from the zoomed over-head image on the top right, the buildings of commercial area are dominated by tall buildings. In the middle right zoomed image, the industrial area is dominated by large flat-roofed buildings with low floors, which is a typical feature of the industrial area. In zoomed over-head image on the bottom right, the characteristics of residential areas are also obvious for a large number of small well-arranged low-rise buildings. The public areas represented by yellow dots rarely form clusters.

VI. CONCLUSION AND FUTURE WORK
As the CNNs gradually show an overwhelming advantage in common visual tasks, various image-level CNN models are increasingly favored in street view image classification in recent years. In this paper, a dataset "BEAUTY" is presented, which can be used for both street view image classification and building detection. We used ResNet50, which performs steadily and well on common visual classification tasks as a baseline model to represent the current mainstream imagelevel CNN models. However, the macro-precision of the ResNet50 was only 69.16%. After analysing large number of street view image samples, we find that the approaches based on image-level CNN models have the following fatal problems.
• The undifferentiated use of the whole image leads to the extraction of common visual factors that confuse classification. • Street view image labels for land use classification are often concepts with a high level of abstraction and cannot be described directly and effectively with visual features.
As can be seen from the example in Fig. 12, although CNNs have the ability to extract regions conducive to classification through autonomous learning, these regions are often not accurate when classification labels cannot be directly and effectively described with visual features. In addition, the only use of visual semantics (e.g., the recognition results of objects) can no longer well represent highly abstract landuse concepts, which must be done with context-describing visual syntax. Based on the above considerations, this paper proposes a "Detector-Encoder-Classifier" architecture. Object detectors extract visual features that are more recognizable by learning the annotations specifically for buildings. The proposed "CODING" method encodes the context relations such as co-occurrence and layout of these highly recognizable visual objects. At last, RNNs are very suitable for accurately classifying the combination patterns of visual elements with structural relations. The proposed approach performs 81.81% on macro-precision, an improvement of 12.56% over the baseline model.
The first row of TABLE VII shows the performance of the proposed approach using a "perfect detector", which gives the upper limit of the proposed approach and two ideas for improving the performance under the current architecture.
• To achieve the upper limit, better detectors are needed.
With the development of object detection, this plug-and play-module can be upgraded continuously. • To exceed this limit, more powerful context encoders need to be proposed. Self-attention [56] or transformers model [57] might be used.
In addition to models, data form different sources are also an important way to improve the performance. A more accurate description of the land use may be obtained by matching the layout of the building in street view images to one in overhead images.


In-Memory Hamming Error-Correcting Code in Memristor Crossbar

— This paper proposes a in-memory Hamming error-correcting code (ECC) in memristor crossbar array (CBA). Based on unique I-V characteristic of complementary resistive switching (CRS) memristor, this work discovers that a combination of three memristors behaves as a stateful exclusive-OR (XOR) logic device. In addition, a two-step (build-up and fire) current-mode CBA driving scheme is proposed to realize a linear increment of the build-up voltage that is proportional to the number of low-resistance state (LRS) memristors in the array. Combining the proposed XOR logic device and the driving scheme, we realize a complete stateful XOR logic, which enables a fully functional in-memory Hamming ECC, including parity bit generation and storage followed by syndrome vector calculation/readout. The proposed technique is verified by simulation program with integrated circuit emphasis (SPICE) simulations, with a Verilog-A CRS memristor model and a commercial 45-nm CMOS process design kit (PDK). The verification results prove that the proposed in-memory ECC perfectly detects error regardless of data patterns and error locations with enough margin.

I. INTRODUCTION
emristor crossbar is expected to make sign ificant contributions to future computing system in various ways, such as high-density and low-latency non-volatile memories [1]- [6], neural networks [7]- [14], and process-in-memory (PIM) [15]-- [20]. However, device reliability issue is one of the main limiters for their realization into a practical computing system [12]. For example, there is an inherent trade-off between performance and reliability of a memristor device that sacrificing reliability is inevitable to operate the memory at speed. Such issue aggravates further when they come to logic operations in neural network and PIM applications where the access rate increases by orders of magnitude.
Error-correctin g codes (ECC) is an essential technique in memory technology that signif icantly enhances data reliability. Hamming ECC, first proposed in 1950 [21], is one of the most popular ECC and is still widely used to address the reliability issue of the various legacy memories [22]- [26]. Parity bits are stored in memory array along with data bits during the data programming, and are used to tell whether there have been any bit flips, or error, occurred durin g a data read out. In general, a n ECC engine located outside of the memory array calculates parity at programming and decodes the ECC at read out to detect and correct the error so that a clean data is sent to user. Fig. 1 gives a brief explanation on how a Hamming ECC for 4-bit data (d1-d4) works, so-called Hamming (7,4) ECC. Here, the entire data consists of 7 bits including 3-b it parity and 4-bit data. Hamming ECC necessitates even/odd parity generation during p rogramming, where the count of '1' in the data+parity codes always becomes either even or odd. Therefore, it starts by creating three parity bits (p1-p3) from three different combination of datasets based on exclusive-OR (XOR) lo gic operation, which is grouped into different circles of Fig. 1 (a). In this case, the XOR lo gic outputs '1' if the number of '1' is odd in any of the three datasets. Therefore, the total number of '1' in a circle of Fig. 1(a), which is comprised of 3-bit data and the corresponding 1-b it parity (e.g., d1, d2, d4, and p1 in the uppermost circle ) should always be even. In other words, if we take an XOR of the 4-bit data in a circle during a read out, it should be zero if there has been no bit corruption. Based on the observation, the Hamming ECC takes those XORs as a syndrome vector to exactly point where the error has occurred among the 4-data bits, as shown in Fig. 1(b). Fig. 1(c) and (d) show an example. With d1through d4 being 1001, p1through p3 becomes 001. With no error, the syndrome vector calculation must give (0, 0, 0 ). However, when an error occurs at d1, the XORs of p1 and p2 circles give 1 instead of 0, and lead to the syndrome vector of (1, 1, 0), which informs that the ECC should correct d1 before the data is sent to user. Similarly, the syndrome vector of (1, 0, 1), (0, 1, 1), and (1, 1, 1 ) point to an error occurred at d2, d3, and d4, respectively. This work proposes a novel in-memory Hamming (7, 4) ECC in a memristor crossbar. Simulation program with integrated circuit emphasis (SPICE) verif ications with a Verilo g-A memristor model and commercial CMOS transistor are provided to fully validate the complete steps of Hamming encoding and decoding, including the parity bits and the syndrome vector generation. To the best of the authors' knowledge, this work is the first ever reported in-memory ECC scheme that implements an entire step within a memristor crossbar. [17] and [27] have proposed an in-memory parity generation and an in-memory parity check schemes, respectively, wh ich in fact have inherent limitations to be extended to cover the entire steps.
The remainder of this paper is organized as follows. Sect ion II introduces a complementary resistive switching (CRS) memristor which is adopted in this work [28]. A novel stateful XOR logic implementation based on the CRS memristor is also proposed in Section II. Section III details the circuit techniques for the robust stateful XOR logic, which are followed by SPICE validation results. Section IV presents the verification results from SPICE simulation with the device model and a complete array. Finally, conclusions are provided in Section V.

II. COMPLEMENTARY RESISTIVE SWITCHING (CRS)
MEMRISTOR AND XOR LOGIC WITH CRS [17] proposes and verifies that a current-mode driving scheme enables a linear increment of word-line voltage with respect to the number of low-resistance state (LRS) memristors in the dataset array. Combined with a unique I-V characteristic of a CRS memristor and gray code, such linear word -line voltage is used to program parity bits to a CRS memristor, which is a base work of this work. Fig. 2 shows an example of I-V curve of the CRS memristor obtained from SPICE simulation with a Verilog-A model, which is used throughout this work. The device parameters are tunable in the model, but most of the parameters are chosen to mim ic the physical device characteristic in [28] such as Roff/Ron. The unique characteristic of the CRS device originates from switching between the opposite defect configurations which can be read as a high resistance. Assuming that the CRS memristor is in itialized to a high-resistance state (HRS), the resistance state goes to the LRS as a conducting path is formed when the applied voltage across the top and bottom electrodes increases to positive direction. However, if the voltage increases further, the device goes to another HRS state (HRS') due to defect migration to the opposite side [17]. On the other hand, once the voltage stops increasing, the device stores the resistance state where the increment has stopped until a higher voltage is applied, or it is initialized to the HRS by reset operation. The reset happens when a strong negative voltage is applied. As discussed in Section I, the binary XOR lo gic operation is the key for realizing the parity generation and the Hamming ECC. Although it is proven that the gray code and the multi-bit operation by utilizin g the intermediate states between HRS/LRS and LRS/HRS' enables parity bit generation without using XOR logic in [17], however a further in-memory processing is not possible due to the complexity of decoding the gray -coded resistance. Therefore, to realize a complete in-memory ECC, this work does not rely on the multi-bit operation but utilizes the stateful in-memory XOR logic. Fig. 3(a) shows a conceptual explanation of the XOR logic with the CRS memristors. Given that a linear incremental voltage with respect to the number of '1's (=LR S memristors) in the dataset is available from the current-mode scheme, an I-V curve of 'M' shape or an R-V curve of 'W' shape can serve as an XOR logic. In addition, of course the curve should keep the resistance state to store the parities and the syndromes, in other words it should be a stateful logic. In Fig. 3(a), the blue line shows a simplif ied I-V curve of a single CRS device. If two CRS devices are stacked in series, its I-V curve must be a stretched version of a single CRS's in V-axis, as the red line in Fig. 3(a). If the stacked devices are connected in parallel with a single device, the total current is the sum of each current, where we obtain the M-shape I-V curve. For the rest of the paper, this cell is referred to an XOR memristor. The concept is fully verified with the SPICE simulation with the CRS device model, which is shown in Fig.  3(b). Neglecting all intermediate states for simplicity, which are indeed included in the SPICE model, the XOR memristor has five resistance states: three HRS states (HRS1, HRS2, HRS3) and two LRS states (LRS1, LRS2). Fig. 3(c) shows a state table of each resistance states for better understanding. Once initialized by a strong negative reset voltage, all three memristors are in the HRS state. If a positive voltage of 1 V is applied, R1 is flipped to LRS, but R2 and R3 stay at HRS because the voltage stresses are only 0.5 V due to the stacking. If the voltage increases to ~1.5 V, then it flips R1 to HRS' but it is still not enough to flip R2 and R3, leading to the overall resistance to HRS2. Once the voltage reaches 2 V, then R2 and R3 become LRS so the overall resistance goes to LRS2. If the voltage keeps increasing, R2 and R3 also reach HRS'.

III. VALIDATION OF IN-MEMORY XOR WITH CURRENT-MODE DRIVING SCHEME
In the previous Section, we discussed how a 'M' -shaped I-V curve is obtained and it can be used as an XOR logic. To fully implement XOR, the I-V curve should be translated to current versus digital input, which means the XOR memristor should be preceded by a digital-to-analog converter (DAC ) wh ich translate a digital data (i.e. how many 'l's in the array) to an analog voltage. Fig. 4(a) and (b) shows a circuit diagram of CRS array which explains how the proposed current-mode drivin g DAC works. Four data memristors (RD1-RD4) and an XOR memristor share a word-line. When the array is configured to the XOR operation, the data memristors are driven by PM OS transistors whose gates are biased to an analog voltage. When a data memristor is HRS, the transistor operates in triode region due to the lo w drain-source voltage (VDS) by the large IR drop by the HRS memristor. In other wo rds, the current of the path is constrained by the HRS memristor. On the other hand, when a data memristor is LRS, the biased transistor behaves as a current source because the IR drop across the memristor is low. As a result, the current flowing through the memristor is constrained by the saturation current of the transistor, wh ich means the word-line voltage has little impact on the current. It allows a linear increase of the total current from the memristor array with respect to the number of LRS cells in the array. Due to the nature of parallel resistance, a typical voltage-mode scheme like [29] cannot provide such linearity. The linear current is translated to word-line voltage by a parallel build-up resistor RBU. Note that during the build-up phase, the XOR path is disconnected so the current solely flows to the build-up resistor as shown in Fig. 4(a). The bleeder current (IBLEED,P) is used to shift the current curve (y-intercept) by introducing a DC offset current to the build-up resistor. Once the word -line is fully charged, the b it-line of the XOR memristor is connected to VSS as shown in Fig. 4(b), and simultaneously the build-up voltage programs the XOR. The program voltage versus the number of LRS cells in the dataset array is shown in Fig. 4(c) and (d), where the 3-input case (parity bit program) and the 4-input case (syndrome vector program) are given, respectively. Differential non-linearity (DNL) p lots normalized to least-significant bit (LSB) are also depicted in Fig. 4(c) and (d), which prove the linearity (max DNL < 0.3 LSB) of the proposed current-mode DAC. The two-step (build-up and fire) scheme is required to reso lve the voltage saturation issue of the one-step scheme in [17], which is explained in Fig. 4(e). If the current d irectly drives the XOR without the build-up step and the resistor, the current-to-voltage transformation cannot be linear. For example, the voltage saturates at ~V2 even though RD1-RD4 are all LRS, because once the resistance enters the LRS region the IR drop cannot be higher.

A. Experimental testbench design
The proposed ECC is verified with SPICE simulation with a commercial 45-nm CMOS process design kit (PDK) and the Verilo g-A memristor model. Note that a specific device shown in Fig. 2 is used in this experiment, however the scheme is not sensitive to the device parameters as long as the on/off ratio is comparable or higher, because the transistor design engineering provides enough flexibility. Fig. 5 shows the circuit d iagram of the crossbar array and the experiment steps. The array consists of four data memristors and four XOR memristors, and they all share the same word-line. Three XOR memristors are to generate and to store the 3-bit parity for the Hamming ECC, but the last XOR memristor is to calculate the syndrome vector. In the first phase of the operation, 4-bit data is sequentially written to RD1-RD4. The word-line is driven by VSS (= 0V) during this phase, and the bit-line of the selected cell is driven by VWRITE, whose value depends on the binary datum to be written to the cell (-1.0 V to program '1' or LRS, +2.5 V to program '0' or HRS).
After programming the data memristors with the desired pattern, the parity generation phase starts. During this phase, the word-line switch to VSS stays disconnected whereas the current-mode scheme drives the bit-line of the selected dataset while the unselected one stays unconnected. This step is repeated by three times to program 3 parity bits to RP1 , RP2 , and RP3, while the selection is based on the parity generation matrix from Fig. 1. For example, for p1 generation, the vector is (1, 1, 0, 1) which means RD1, RD2, and RD4 are driven by the transistors but the bottom electrode of RD3 is floated, so it is the 3-bit XOR logic discussed in Section III. As described in the Section III, during the parity generation step, the bleeder current is also activated to bring the transfer curve to the optimum for the given XOR memristor.
In the next step, the syndrome vector is sequentially programmed to the last XOR memristor (RSYN), and RSYN readout happens for checking error. In fact, the programming-syndrome phase is divided into two sub-phases. Because the syndrome vector does not have to be stored since it will not be used in further in-memory operations unlike the parity bits, it is wasteful to have dedicated memristors for each syndrome elements. Therefore, in this experiment, a sin gle shared XOR memristor is used so each syndrome elements are sequentially processed. As a resu lt, the first sub-phase should be resetting the RSYN to erase the previously stored bit. After the reset, the 4 -bit stateful XOR lo gic happens, which is shown in Fig. 6(c). Durin g this sub-phase, the dataset is also selected from the Hamming ECC criteria shown in Fig. 1. For example, for s1 calculation, RD1, RD2, RD4, and RP1 are selected while the others are floated. Because a different offset should be applied to this 4-bit XOR over the 3-bit XOR , an additional bleeder path is activated to provide an adjusted shift to the curve.
After programming RSYN from the 4-bit stateful XOR operation, the read voltage (VREAD=0.3V) is applied to the bit-line of RSYN and the word-line is connected back to VSS, as shown in Fig. 6(d). The read current is compared to the reference current (ITH) to determine the syndrome is 0 or 1. The syndrome step is repeated by 3 times to achieve the complete syndrome vector (s1, s2, s3).
To fully validate the ECC functionality, an additional step is included in the testbench. As long as the stateful XOR lo gic functions correctly, there is no error in the data and the parity bits (d1-d4, p1-p3). If so, the syndrome vector is always (0, 0, 0). So, an erro r injection step is added after the syndrome phase, and then another syndrome phase is executed to see if the syndrome vector points to the correct error location. Fig. 6 shows the test resu lt from a case with the 4-bit data of  > REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 6 0011 and the intentional erro r injection to the d1. The first and second plots show the bit-line-voltage waveforms of RD1-RD4, RP1-RP3 and RSYN, respectively. The third plot shows the shared word-line-voltage waveform. The last waveform shows the current flow of the RSYN, from which the syndrome vector is obtained during the syndrome readout phase. In the first and the third plots, durin g the first phase (0~4ns), we can see RD1-RD4 are written sequentially while the word-line is t ied to VSS (0 V) and the bit-lines are driven by -1.0 V or 2.5 V depending on the data polarity. In the second phase (4~7ns), the word-line voltage is established based on RD1-RD4 and fired to program correct parity bits to RP1-RP3. In this particular case, the build-up voltages are 1.63V, 1.08V, and 1.08V, respectively, which are mapped to HRS2, LRS1, and LRS1 of Fig. 3(b), respectively. From 7ns to 16ns, the syndrome reset, progra m, and readout are repeated by three times, and each operation are relevant to the syndrome vector elements s1, s2, and s3. During the programming steps, HRS2-corresponding voltages (1.55~1.67V) are applied to RSYN. As a resu lt, because RSYN is HRS2 for all three sub-phases, the read current is always low (~80A), leading to the syndrome vector of (0, 0, 0). After that, the d1 error injection phase starts at 16ns, where the BL1 is driven by +2.5V to bring the RD1 back to the HRS state. From 17ns to 26ns, the array repeats the syndrome phase with the injected error. Because the d1 is flipped, the syndrome vector should be (1, 1, 0) by the definition of the Hamming ECC, as shown in Fig. 1. In the test, the syndrome programming voltages are 1.13V (LRS1), 0.94V (LRS1), and 1.55V (HRS2), respectively. As a result, the correct syndrome vector of (1, 1, 0) is ev idently obtained from the measured current of (495 A, 406A, 74A).

B. Experimental simulation result
Whereas Fig. 6 details the multiple waveforms when the error is injected to d1, Fig. 7 focuses on validating the syndrome vector when the error is injected to d2, d3, and d4, respectively, by observ ing only the syndrome current. For all cases, the syndrome vector is always (0, 0, 0) before the error injection wh ich verifies the correct parity generation by the in-memory stateful XOR logic. On the other hand, the vector becomes (1, 0, 1), (0, 1, 1), and (1, 1, 1) after the error is injected to d2, d3, and d4, respectively. It proves the correct Hamming ECC function as well as the stateful XOR logic. Fig. 8 shows the syndrome current over all 4-b it input data cases, with the error injected to d1. No matter which data is written init ially, the correct syndrome vector for d1 error of (1, 1, 0) is obtained after the error injection. Also, the vector is always (0, 0, 0) before the error injection. In summary, the results given in Fig. 6 and Fig. 7 Fig. 7. Syndrome current measurement when the error is injected to d2, d3, and d4, respectively. points to where the error happens, and the result in Fig. 8 verifies that it functions robustly with enough margin regardless of the stored data.

V. CONCLUSION
This paper presents a novel in-memory Hamming ECC technique in memristor crossbar. A stateful XOR lo gic based on CRS memristors is proposed to enable the in-memory parity bit generation and syndrome vector calculation. In addition, a two-step (build-up and fire), current-mode bit-line driv ing scheme allows a linear digital-to-analog conversion within a crossbar, which helps retaining enough voltage margin for robust operation. Complete array-level SPICE simulations prove that the proposed ECC technique functions perfectly with enough margin, across all possible data input and error injection combinations.