Predicting Academic Achievement with Machine Learning Algorithms

Education systems produce a large number of valuable data for all stakeholders. The processing of these educational data and making studies on the future of education based on the data reveal highly meaningful results. In this study, an insight was tried to be developed on the educational data collected from ninth-grade students by using data mining methods. The data contains demographic information about students and their families, studying routines, behaviours of attending learning activities, and their epistemological beliefs about science. Thus, this research aimed to solve a classification problem, two-class (successful or unsuccessful according to the exam result) was tried to be estimated from the collected data. In the study, the prediction accuracy of the supervised classification algorithms were compared and it was defined which variables were effective in the formation of classes. When the prediction accuracy of machine learning algorithms was compared, the findings indicated that the Neural Network algorithm (98.6%) had the highest score. The accuracy rate of the other algorithms are kNN (86.2%), Logistic Regression (78.4%), SVM (90.3%), Decision Tree (91.9%), Random Forest (90.0%), and Naive Bayes (81.7%). The information gain coefficient of the variables was examined to determine the factors affecting the prediction accuracy. It was revealed that demographic variables of the family, scientific epistemological beliefs of the student, study routines and attitudes towards some courses affected the classification. It can be concluded that there was a relationship between these variables and academic success. Studies on these variables will support students' academic success.


373
such as business, insurance, marketing, medicine, biology, telecommunications, education, etc. (İnan,2003;. The implementation of data mining in education is called educational data mining and it is an interdisciplinary field (Romero and Ventura, 2007). Educational data mining is based on general data mining which is used to examine and the analysis of data obtained from educational environments. The initial aim is to improve the educational outcomes of the students, to understand the educational environments they learn, to understand better how they learn, how they define learning, and how they learn and explain educational phenomena (Romero & Ventura, 2007Baradwaj & Pal, 2012, Osmanoğlu et all., 2020. The data obtained with educational data mining from any learning environment, which is conducive and promotes learning, can be analyzed (Baker, 2010).
Educational data mining methods have different sources such as data mining, machine learning, psychometrics and computational modeling, statistics and information visualization (Algarni, 2016). Ventura (2013, 2020) state that educational data mining studies can be used for: • estimating student performance, • testing or developing technology-supported learning theories, • to give teachers feedback on how to improve the educational outcomes of students, • personalization of the service provided to students, • to give students suggestions about learning activities, tasks, problems to be solved or lessons to be completed, • to inform the stakeholders of education about the undesired behaviours of the students during the learning process, • developing and organizing cognitive models for students to present their knowledge and skills, • grouping or defining students profile , • to help teachers to develop and conduct the content of a lesson and planning lessons for the future.
Baker (2010) classified educational data mining studies into five main categories; prediction, clustering, relationship mining, distilation of data for human judgement, and discovery with models. These categories and some studies are exemplified here.
(1) Prediction models: In prediction, the aim is to develop a model which can infer a single aspect of the data (predicted variable) from some combination of other aspects of the data (predictor variables). Prediction models can be used for identify students at risk or predicting students' achievements. Cha, Kim, Park, Yoon, Jung, and Lee (2006) preferred decision trees and hidden Markov models from classification algorithms to classify students' learning styles and preferences. The data, collected through a smart learning system involving 600 higher education students, were analyzed. Dekker, Pechenizkiy and Vleeshouwers (2009) used decision trees to predict whether students studying in the electrical and electronic engineering department would drop out of school at the end of the first year. Also, they determined the success criteria for the electrical and electronic engineering department. Cano and Leonard (2019) utilized genetic programming classification algorithms in their studies and they developed an early warning system to predict the academic performance of university students.
They stated that the results obtained from the study would help teachers and policymakers to improve education.
(2) Clustering models are used for clustering the data which have same attribute in the data set.
Determining students' behaviours and learning styles can be given as an example. Vale, Madeira and Antunes (2014) conducted a block cluster analysis using the decision trees algorithms to determine the study areas of 443 graduate students who got a degree between 1997 and 2012. D'Mello (2017) used a learning analytics tool to collect interaction patterns and body language data and analyzed this data for the effects of emotions on learning. Şahin, Keskin, Özgür and Yurdugül (2017) collected data about students' interactions and self-regulation skills from an online course named "computer networks and communication". The data were analyzed with grey interaction analysis and optimal scaling analysis and they clustered students depending on their interactions. Milliecamp, Broos, De Laet, and Verbert (2019) collected data online and used visualization tools to identify users' behaviour and improve their learning.
(3) Relationship mining is used for revealing the relationship between two or more variables.
One of the areas of relationship mining is revealing the relationship between students' academic achievement and parents' attitudes. Another area is determining the effect of a learning method on academic success. The four sub-categories are (a) association rule mining, (b) correlation mining, (c) sequential pattern mining, (d) causal data mining.. Rashid, Asif, Butt, and Ashraf (2013) analyzed collaboration and sequential pattern mining by processing feedback on students' own performance in the student information system. They stated that GPS (generalized sequential pattern mining) gave more successful results than Apriori algorithm in terms of finding the relationship between data in text format. Dalkılıç and Aydın (2017) set out the association rules to reveal the causes of university students' absenteeism by using the apriori algorithm.
(4) Distillation of data for human judgement is the fourth type of categories. This model aims to find new ways to define or classify the characteristics of the data and can be used to identify patterns in students' behaviour, learning styles, collaborative work. In their research, Scheuer and McLaren (2012) analyzed students' scientific research performances, research design skills and experimental design skills using data mining methods on the data they obtained from the online research teaching system. Hernández-García, Acquila-Natale, Chaparro-Peláez, and Conde (2018) analyzed data in a learning analytics system to predict students' performance in their collaborative work on a project-based learning task. what kind of data will be collected and the data collection approach. The second one helps in explaining students behaviors' to achieve success. Bravo-Agapito, Frances, and Seaone (2019) analyzed studies that using educational data mining methods to improve foreign language learning. They concluded that studies were usually done to estimate student performance, control students' motivation and provide feedback to teachers. Botelho, Baker and Heffernan (2019) aimed to create a new model that perceives students' behaviour and emotions by using machine learning techniques. They stated that the use of the feature selection method is a more consistent way to develop high-performance models It has been observed that educational data mining studies have increased in recent years (Romero and Ventura, 2020). Based on the results of these studies, we can concluded that educational data mining can be used in different fields to solve problems, and make predictions.
In this study, a model was tried to be developed on the educational data collected from high school students by using data mining methods. Demographic information about themselves and their families, HSE (High School Exam) study routines, class engagement behaviours and scientific epistemological beliefs were collected from 9th-grade students. The research considered as a classification problem, and two classes (being successful in the exam or not) tried to be estimated. Prediction accuracy of the supervised classification algorithms was compared and at the same time, the variables that affect the formation of classes have determined. Based on the studies in the literature, it was understood that the participants of the studies conducted in the field of educational data mining were generally university students and the data collected from the learning analytics or student information systems of universities.
Many phenomena such as academic success, learning styles, and behavioural styles of students were analyzed in the studies. In this current research, a model was created by analyzing various data collected from high school students with classification algorithms. With the model created, it is aimed to determine the academic success of students who are preparing for HSE and to reveal the inputs that will affect their academic success.

METHODOLOGY
Data mining methods are used in this research to create foresight from data and develop strategies accordingly. In this study, the classification method, which takes place under data mining techniques, used for collecting educational data, determining prediction models, classifying and testing the classification accuracy. The CRISP-DM (Cross Industry Standard Process Model -Data Mining) methodology applied in the research process for machine learning. The CRISP methodology proposes step-by-step procedures to make the process reliable and standard. This methodology; follows a cyclical process that includes understanding the problem, understanding the data, data preparation, modelling, implementation, evaluation, and reporting (Chapman, Clinton, Kerber, Khabaza, Reinartz, Shearer, and Wirth, 2000;Almahadeen, Alkaya and Sarı, 2017;Demirkol, Kartal, Şeneler, and Gülseçen, 2019). The path followed in the research process has shown in Figure 1. (iii)Source of Information: Information obtained from books and teachers is always correct.
Items in the sub-dimension state that searching the source of scientific knowledge in books/teachers indicates students' undeveloped/immature beliefs.
(iv)Reasoning: The scientist is curious and creates scientific information based on his initial knowledge, observations and logic. Items in the subdimension state that the scientist uses reasoning and logic in the process of creating scientific information.
(v)Variability Laboratory using Python programming language (Demsar et al, 2013). In the analysis process, 7 different classification algorithms were used. These are kNN (k = 3), Logistic Regression, SVM, Neural Networks, Naive Bayes, Decision Tree and Random Forest algorithms. k-fold cross-validation method used for all algorithms. In k-fold cross-validation, the original sample is randomly partitioned into k equal sized subsamples. Of the k subsamples, a single subsample is retained as the validation data for testing the model, and the remaining k−1 subsamples are used as training data. The cross-validation process is then repeated k times, with each of the k subsamples used exactly once as the validation data. In this study, the value of k was taken as 10. This method aims to determine the best classifier for this dataset. In the analyze process accuracy rates of classification algorithms were calculated, confusion matrices were prepared and ROC analysis was performed. At the same time, in order to increase the success of classification algorithms, the information gain value of the features in the data set was calculated. Histogram graphiscs of features with high values were drawn. The system architecture is given in Figure 2.

FINDINGS
After the pre-processing of the data set, 7 different classification algorithms were used to determine the success of the students. For classification kNN (k = 3), Logistic Regression, SVM, Neural Network, Decision Tree, Random Forest, and Naive Bayes algorithms were used then the results were examined. To increase the success of classification algorithms, information gain values of the independent variables in the data set were calculated and the results are given in Figure 3.

Figure 3. Information Gain values of features
When the results were analyzed, the order of 10 independent variables, in which the greatest value is determined, is (1) education level of the mother, (2) variability of ınformation, (3) education level of the father, (4) authority and integrity, (5) reasoning, (6) study duration per day, (7) history of the Turkish revolution, (8) information generation process, (9) family income, (10) solved problem per day. The size effect of the information gain coefficient is in these variables indicates that it plays an important role in the formation of classes. The histogram plots of these variables and the average values of the distributions are given in Figure   4. The red columns represent high school students who were successful in the HSE (1), and the blue columns represent high school students who were not successful in the HSE (0).

DISCUSSION AND CONCLUSION
In this study, a data set created with students' scientific epistemological beliefs, study routines, behaviours in a particular course, demographic information about families, and attitudes to courses. The data set used for the solution of a classification problem (being successful in HSE).
a model was developed with classification algorithms and these algorithms accuracy rates were calculated. Also, the variables that play a role in the formation of classes was examined.
According to the findings, the Neural Network (98.6%) algorithm has the highest classification accuracy rate. The education levels of mother and father, the income level of the family, the students' scientific epistemological beliefs, study routines and attitudes towards the lessons are effective on HSE success. Numerous studies using classification algorithms for the prediction of academic success have been conducted with different data sets. Some of these studies conducted with high school students were given in this section. Márquez-Vera, Cano, Romero and Ventura (2013) Roy and Garg (2017) concluded that the J48 algorithm (73.92%) was more successful, they concluded that the students' health status, education of their families, alcohol use and friend relationship were among the factors affecting their success. Al-Saleem, Al-Kathiry, Al-Osimi, and Badr (2015) used the J48 and ID3 algorithms to create a model for predicting students' achievements and tried to determine their success in various elective courses with these algorithms. As a result, they concluded that the J48 (83.75%) algorithm gives more successful results than ID3 (69.27%). Strecht, Cruz, Soares, and Mendes-Moreira (2015) compared the success of classification and regression algorithms to predict academic success. They concluded that between these two approaches, classification algorithms are more successful. In the study, they used the university' academic database and they reveal that the SVM algorithm has the highest prediction success. Guo, Zhang, Xu, Shi, and Yang (2015) created a deep learning architecture and used 4 different algorithms to compare students' final grades and their achievements. They stated that the most successful algorithm for that problem was the neural network algorithm, which they named SPPN (Student Performance Prediction Network) with 77.2% success rate.
In the other study conducted by Affendey, Paris, Mustapha, Sulaiman, and Muda (2010), the effects of the courses that taken by students are examined and the predictive success of the algorithms are compared, and the Naive Bayes algorithm has a 95.29% success rate and 5 courses (Computer Programing 2, Multimedia Technology, Computer Organization, Assembly Language and Programming Language) have shown that have a great effect on success.

SUGGESTIONS AND LIMITATIONS
When the research findings and the studies in the field are evaluated together, it was concluded that different algorithms were successful on different datasets, in other words, different solutions were depending on the nature of the problem to be solved. It is understood that the success of the algorithms used in the researches is directly correlated with the collected data which affective on the problem result. In this study, it was revealed that the education level of the family, the thoughts of the students about the nature of scientific knowledge, students' study routines, and the income level of their family play an important role on the success of the algorithm which was used for predicting academic achievement.
Also, there were some limitations to this research. A two-class educational data mining problem was discussed in this article. The data collected were collected from 421 9th-grade students at 14 different schools in the city centre. 29 features used for predicting exam success. It is thought that level of algorithms prediction success is about these limitations. A similar problem can be applied to different settlement levels and locations or larger participant groups. Standard models can be developed by including different variables that affect academic success. Another important issue that stands out here is, the intervention areas can be determined by examining this research and other researches to increase the academic success of students.