A Comparative Study on Quality Characteristics in Designing Educational Applications

Traditional learning methods have significantly changed with the adaptation of modern technologies. Teachers and students have become more interested to build their knowledge when exploring learning materials using various devices and applications. ISO/IEC 25010:2011 and ISO 9241-11:2008 are referred to regarding five quality characteristics in designing educational applications. This paper reports the comparative study that investigates the quality characteristics in existing educational applications by applying the Kirkpatrick Model, which comprises four logical levels in the educational process. The investigation of the quality characteristics involved four types of online educational applications. The analysis shows that more than half (56.66%) of the compared characteristics were not found in the selected educational applications. Thus, the study concludes that the compared educational applications remain to have issues if software developers or software engineers do not consider the five quality characteristics, which include user interface aesthetics, appropriateness recognisability, understandability, effectiveness and satisfaction from the users’ perspectives.


I. INTRODUCTION
Traditional learning methods have significantly changed with the adaptation of modern technology, especially in educational applications. Teachers and students have become more interested in building their knowledge by exploring the learning materials [1]. This encourages more collaboration into flexible zones and interaction between teachers and students in a more effective teaching style [2], [3].
Students are encouraged to create dynamic, cooperative and collaborative learning activities while applying the constructivist approach [4]. This tends to put more effort to reach their learning goals. Besides, teachers can assist their students to overcome the fear of studies and lack of interest [5]. To achieve this goal, developers should clearly understand and be concerned with users' needs, which formally align with the functional and non-functional Manuscript  requirements. The same understanding among the developers will contribute to effective and efficient applications that are normally difficult to achieve [6].
Developers who are involved in the process of producing attractive and good quality educational applications must have enough information, skills and knowledge with complete guidance [6]. A survey has been done on educational applications with the implementation of required quality criteria and identified five quality factors concerned: effectiveness, satisfaction, efficiency, learnability and understandability that are required when designing and developing educational applications. Furthermore, the findings deduce that the five quality factors can improve students' understandability and increase students' motivation in their learning process while being facilitated by their teachers [7]. Quality characteristics in such applications are vital for overall organisational achievement and success in the education domain [8]. ISO/IEC 25010:2011 [9] and ISO 9241-11: 2008 [10] were referred to determine the five-quality characteristics that should be applied when developing educational applications. The list of characteristic, definition and code are shown in Table I. Ua Learnability -Degree to which a product or system can be used by specified users to achieve specified goals of learning to use the product or system with effectiveness, efficiency, freedom from risk and satisfaction in a specified context of use‖ [9] Lb Appropriateness recognisability -Degree to which users can recognize whether a product or system is appropriate for their needs‖ [9] Ar Satisfaction -Degree to which users' needs are satisfied when a product or system is used in a specified context of use‖ [10] Sf Effectiveness -Accuracy and completeness with which users achieve specified goals‖ [10] Av This study aims to perform a comparative study to investigate the quality characteristics of existing educational applications. To determine the comparative items, this study applied the Kirkpatrick Model, which comprises four logical levels in the educational process, namely, reaction, learning, behaviour and results [11]. Subsequently, for the benefit of education in Malaysia, this study has added one more level, which is discipline, to align with the Malaysia Educational Blueprint 2013-2025 (MEB) [12] and the 21 st Century Learning Skill (PAK21) [13].
To define and assign the logical level, the Kirkpatrick model complexity of the evaluation hierarchy was referred to, as depicted in Fig. 1. To align with the research aim, this study has added the discipline level to investigate if the applications contain the criteria as aforementioned MEB [12] and PAK21 [13].  The following sections include the related work in Section II. The comparative study is presented in Section III. The conclusion and future work are stated in Section IV.

II. RELATED WORK
The related work provides the details on the usability characteristics. Seralidou et al. [15] mention that the usability issue needs to be solved such as information arrangement and the content layout, to reduce the complexity of the application and assist in ease of use. In addition, usability can be described from a variety of factors, including the interface design, education field, learnability and understandability, which are all major factors that indicate whether the application is well delivered [16]. Subsequently, heuristic evaluation is the most appropriate to test usability to ensure students can explore and use the learning materials without failure to support their learning process [17].
As mentioned in Table I, five usability criteria were investigated. The user interface aesthetic is a principle in designing educational applications to drive students to learn effectively [18]. Hence, design aesthetics is one of the greater degrees of satisfaction that leads to a greater degree of loyalty, and has the potential to improve user satisfaction, understandability, learnability and ease of use [19]. Besides, it is also effective in enhancing students' overall understanding of the content that is taught in the lesson [20]. Previous work aims to investigate how knowledge interacts with self-esteem and with the self-actualization of the students' understandability that motivates students in their learning process [21].
Another work proposes question and answer on a scale in an educational application, and it was found that the application can be easily understood and identified [22]. Satisfaction is a subjective quality referring to the graphic design, overall visual appeal, colour scheme, and stylistic consistency, which leads to user satisfaction and support on users' learnability [23]. Furthermore, users' aspirations normally lead to users' satisfaction from the beginning of the development [24]. User satisfaction and effectiveness of the application is achieved when the application successfully helps the user in doing tasks [25]. Finally, the effectiveness of the application allows users to easily understand each process during completing a given task in a short period of time with quick learning [26].
The defined characteristic is planned to be implemented in an educational application. Educational applications need to be structured and created simply for youngsters to utilize and learn, which includes the assessment rubric for assessing the convenience of instructive application viewpoints, and the client involvement to check if the necessities are fulfilled [27]. Besides, the application is additionally equipped with material presentation, virtual practicum, exercise things, and media with fascinating and interactive looks [28]. The Kirkpatrick Model [11] is used as a guide to investigate the educational level. This is a model where learning becomes more effective for students once utilized well, and boosts capability and knowledge [29]. Students enjoyed the motivation to engage in a learning task with positive outcomes in a time-efficient manner, for both educators and students [30].
The comparative study is presented in the next section to investigate the findings on defined usability characteristics. A comparative study is a core qualitative analysis approach that is conducted to understand the impact of multiple research works [31]. In addition, a comparative study is a process to analyse the difference among applications in many compromising categories, for the benefits of users in education and learning [32].

III. COMPARATIVE STUDY
The strategy used in the comparative study is illustrated in detail in Fig. 3. This helps to offer an overall view of the conducted activities in a structured and easier manner.
International Journal of Information and Education Technology, Vol. 10, No. 8, August 2020 Six main activities were performed in a sequential manner. This study has adapted the Calero Model [33], [34] shown in Table II. Originally, the table lists three columns, namely, classification, sub-classification and characteristics. Thus, to enhance the initial study on the survey that has been done earlier, this study added the include characteristic column to be used in verifying the sub-classification that should be applied in selecting the educational applications. In Fig. 3, activity A involves the selection of applications for comparison.  [42] were excluded due to operation issues, whereby the applications can only be run on Android and iPhone devices, while others can be run using any devices.
The following Activity B was conducted to select a suitable educational model for guidance in focusing on the educational logic level. This activity has adapted the Bloom's taxonomy hierarchy [43], which is listed in Table IV. Kickpatrik model [11] meets all levels in Bloom's hierarchy and has been selected. Besides, this model also includes the Kickpatrik evaluation hierarchy, which helps The investigation of the quality characteristic involves four online educational applications. This study focuses on online applications only as they have been commonly used nowadays covering general learning materials sharing platform, conversation, language and mathematics subject. Each application was compared and mapped to all the defined codes to investigate the criteria (exist or not) as categorised in Table V. If it exists, there will be another four different categories to be investigated. Totally not exist NE Activity C involved the process to determine the comparative items. The content of the applications is divided into four components, as listed in Table VI. Another code was given to each component (Pn), where P represents a component and n represents each of the components n = [1,2,3,4]. The components to be specific include learning materials, learning exercises, assessment activities and support activities [46]. This study added one more component, which is an aesthetic user interface design where n = [5].
Then, activity D was executed, and all the codes are assigned to the compared educational applications, as depicted in Table VII. The mapping consists of ‗CI', which represents the comparative item, while ‗KM' represents the Kirkpatrick model, ‗QT' represents the quality characteristic. Activity E involves the process to record the results. Each application is explored to identify whether the elements being investigated exist or not based on the five component  In this study, four different applications were selected. First, Schoology, the social educational networking service and virtual learning that allows users to create, manage, use and share learning materials [36]. Second, Conversation starters, an application which provides a sheet of questions and answers in different arrangement [37]. Third, Duolingo is an application to engage users in learning various words and meanings in different language configs [40] and the fourth is Quick math where the application provides automated math solutions [41].
Based on the comparative study, activity F deduced that the selected applications still have limitations in certain criteria and can be improved in the future. The data were analysed to obtain the information on the highest and lowest percentage for the comparative item studied on the quality characteristic. The data (n) which are the categories found in the comparative study were calculated. Only four categories are found, which are ‗FC', ‗PNC', ‗FNC' and ‗NE' as illustrated in Fig. 4. Next, the total data (N = 60) are used to calculate and produce the percentage. The circle graph shows that for ‗FC' which is n = 34 (56.67%) of the comparative items consistently implement the quality characteristic in the application. While for ‗PNC' which is n = 4 (6.67%) of the comparative items partially and not consistent in the implementation of the quality characteristic. Another value is ‗FNC' which is n = 9 (15.00%) of the comparative items are implemented fully and not consistent in term of the quality characteristic. Finally, the value for ‗NE' which is n = 13 (21.66%) of the comparative items do not implement any of the quality characteristics totally.
In addition, three of the applications that are Schoology [36], Conversation starters [37] and Duolingo [40] have the highest total of ‗FC' which are 10 out of 15 compared elements.
Besides, they have a similar result (‗FC') in five out of the 10 items which are item R3 for search function, R5 for lesson function, R8 for feedback function, R14 for audio function and R15 for video function. These features can guide and assist users to complete given tasks effectively. The study also shows that Quick math [41] has the highest ‗NE' that is almost half of the investigated elements (7 out of 15) do not exist. Hence, the study deduces the need to guide educational application developers to ensure most elements that meet the International Journal of Information and Education Technology, Vol. 10, No. 8, August 2020 required quality factors are considered when designing and implementing such applications.

IV. CONCLUSION AND FUTURE WORK
This study concludes that the existing educational applications still have limitations that need to be improved to support the quality and the concerned characteristics that are user interface aesthetics, appropriateness recognisability, understandability and satisfaction. The comparative study can help to recognise the importance of the criteria in educational applications. Early findings show that a good user interface aesthetic design will lead to appropriateness recognisability and understandability. Consequently, the effectiveness in using an educational application leads to the completion of the given task or accessing the educational application successfully.
Future work will be to obtain detailed information on the component elements and comparative items that need to be deployed as the guidance when developing educational applications. Besides, future studies will also gather information on the strengths and weaknesses of each comparative item.

CONFLICT OF INTEREST
The authors declare no conflict of interest.

AUTHOR CONTRIBUTIONS
The first author has conducted the research and wrote the first draft of the paper. Subsequently, the second author has verified the study and revised the overall content of the paper. The third author has revisited on the attractive learning in educational applications. Moreover, the fourth author has revised the scope of the research, which is the educational applications. All authors have agreed and approved the final version.
ACKNOWLEDGEMENT N. Ngadiman acknowledges South East Johor Development Authority (KEJORA) for the scholarship that supports this research for her PhD work and Cost Centre 4B349, Centre for Advancement in Rural Education Informatics (iCARE) community project that provides the platform for the community-based participatory research.