A New Approach for Multi-Level Evaluation of Strategic Educational Goals

Educational organizations with multiple level of management promotes for their strategic educational goals as a correlated and clustered data. The typical assessment and feedback approaches are paper-based where word documents and flowcharts are used to evaluate strategic educational goals augmented with quantitative indicators. Unfortunately, the paper-based approach often neglects the relationship and dependencies between the educational goals defined at different levels. This may lead to complications in the analysis, lack of clarity, and subject to different interpretations by the multiple management. We propose a multi-level model-driven approach that improves the assessment of strategic educational goals, handles the clustered data efficiently and allows the individual and group level assessment to take effect simultaneously. The approach also allows decision makers in academic institution to extract valuable information from goal models at different academic levels and measure the fulfilment of the educational goals with respect to the target performance in a formal way. Keywords—Evaluation process; goal model; multi-level modelling; goal requirement language; program educational goals


I. INTRODUCTION
Academic institutions are using different learning assessment approaches and reviews to evaluate student's learning progress. The evaluation process of students' performance starts by the time they are admitted to the academic institution and continues until four or five years from the graduation when they are engaged with the market.
Strategic educational goals are those objectives and targets that support performance roadmap to measure the institution state and progress. Academic institutions promote for strategic educational goals at six academic levels as in Fig. 1. The top level is the institution where the vision statement is used to describe the future accomplishments and objectives of the institution; also, the mission statement to describe the action needed to be done to meet the vision statement. At the faculty and department levels, mission statements stem their targets from the institution mission. At the program level, the program educational objectives (PEOs) are tailored to serve and promote the mission statement to describe the professional and career accomplishments of graduates during the four to five years from graduation [1]. Finally, the student outcomes (SOs) are defined at the curriculum and course levels, to describe what students are expected to know and practice by the time of graduation from the program.
Several constituencies are involved in the development and evaluation process of strategic educational goals. The major constituencies are program academic members, industrial advisory board (IAB), program alumni, undergraduate students of the program, and employers of the program graduates. Other constituencies may be involved are students' parents, program administrative staff and administration of the academic institution. IAB consists of professionals, experts and/or managers employed at major industries related to the academic program.
Student's performance during the academic semester is used as an indicator of how much the SOs have been met. There are two types of assessment tools are used in measuring student performance: a) direct tools where student assessments are measured though direct examination or various of submitted work, such as assignments, quizzes and exams; b) indirect tools where student achievement requires that academic consistencies infer actual student abilities, knowledge, and values rather than observe direct evidence of achievement, example of indirect tools: surveys and interviews.
The typical paper-based learning assessments and reviews approaches, nowadays, comes in a form of word documents, spreadsheets and flowcharts. They are used in the process of evaluating strategic educational goals with respect to the target goals and objectives of the institution. The paper-based approach often neglects the relationship and dependencies between the educational goals defined at different levels. This might cause confusion in analyzing the learning assessments, lack of clarity, and subject to different interpretations by constituencies. It is desirable to use a model-driven approach with multi-level modeling to improve the learning assessment, evaluate the learning goals in a formal way and extract information at different academic levels.
The proposed paper extends additional details on earlier research results presented at the conference in [1]. In this paper, we proposes a model-driven approach with a multi-level modelling where the Goal-oriented Requirement Language (GRL) is used for assessing the learning goals and objectives. Multi-level Goal modeling provides performance indicators for the quantitative measures of strategic educational goals during the continuous evaluation process. It handles the collective data efficiently and allows the individual and group level assessment to take effect simultaneously. The paper is organized as follows: Section 2 presents the background and related work; Section 3 describes an overview of our goaloriented approach; Section 4 demonstrates the GRL goal modeling; Section 5 shows a case study of evaluation modeling and analysis; Section 6 provides conclusions and future work.

II. BACKGROUND AND RELATED WORK
Goal-oriented modeling languages are used in recent research projects to capture business goals and associate them with performance measures on different quality aspects. The aim of modeling is to improve the decision-making process, provide a structure formality, reduce the lack of clarity in user requirements and detect early of any deficiencies in meeting business goals. The Goal-oriented Requirement Language (GRL) is a standard notation for goal modeling. It is part of the User Requirements Notation (URN) [2] that describes business goals and facilitates the modelers to describe intention elements (e.g., goals, tasks, indicators) their decomposed structure (e.g., sub goals, stubs), connecters (e.g., dependencies, contribution) and their corresponding partners (e.g., actors, agents, teams).
In many recent researches, multi-level goal modeling is used when business goals are organized at more than one level. Multi-level goal modeling is hierarchical structured business goal that allows researchers to investigate the effect of group attributes on individual business goal while accounting for non-independence of observations. The analysis of multi-level goal modeling at the lower level is performed to individual business goals and nested within the accumulated business goals at the higher level [3]. Multi-level modeling is used when the analyzed data have a clustered structure and there is a substantive interest in the individual effects, group effects and the mutual effects. Multi-level modeling was adapted in different researches, such as public health [4] and validating education indicators [5].
As part of the proposed approach, a common open source graphical editor called jUCMNav [6] is used for goal modeling. jUCMNav is an eclipse-based URN tool that supports modeling goals and business processes with GRL and Use Case Map (UCM). It enables generating and managing complex GRL models based on multi-level modeling. It also provides features to utilize strategies using different analysis algorithms, to support execute and visualize analysis results, and to generate reports.
Research projects are using multi-level approaches for different evaluations. Hoe et al. in [7] propose an evaluation framework for usability of a mobile phone using a multi-level hierarchical model of usability factors. Sanders in [8] reviews the development progress of performance/dependability evaluation tools, and the importance of creating modeling frameworks that support multi-level modeling and multiple solution methods as an integrated framework. While Comuzzi et al. in [9] present the fundamental elements and interfaces of the technical architecture for a multi-level SLA management framework. Also, Yang and Sen in [10] develop a general multilevel evaluation process that deals with multiple attribute decision making problem with both quantitative and qualitative attributes.
Several research projects also have been using GRL modeling language for business goal compliance. Tawhid et al. in [11] propose a novel approach that models regulations with the GRL enhanced with qualitative indicators to generate questions for inspection operations and facilitate compliance analysis. A framework in [12] uses metrics defined in goal and scenario models to validate quality assurance of online business processes. Ghanavati et al. in [13] propose a framework that models legal documents with goals and maps such model to the goal model of the organization. To analyze the degree of legal compliance/non-compliance of organizational goals, traceability links are used between these two models and the GRL quantitative and qualitative algorithms.
Also, several learning assessment and feedback approaches and methodologies have been followed recently. The target is evaluating the learning outcomes and objectives of academic institution. Suskie in [14] introduces and analyzes various methods and approaches of assessing student learning outcomes. Gastli et al. [15] propose an innovative tool and process that allows accurate direct and indirect outcomes assessment of courses and programs while facilitating the tasks for the instructors and in their evaluation process. DeLyser and Hamstad in [16] discuss the visit made by the ABET team to review the outcomes assessment process at University of Denver and what changes were and are continually being made. Yue in [17] proposes a course-based approach that associates learning outcome objectives with accreditation standards and courses; a suitable assessment tool can then be used to assess the course. Besterfield-Sacre et al. [18] develop a framework that specifies the learning outcomes of engineering faculty by expanding them into a set of attributes.
In summary, it is obvious that the works above have addressed some features that are similar to our work. The major advantage of the proposed approach compared to the others is that it integrates the features of using model-driven engineering with the multi-level architecture to build an approach for evaluating the objectives and goals of an academic institution. The proposed approach is used in to improve the assessment of strategic educational goals, handles the clustered data efficiently and allows decision makers in academic institution to extract valuable information from goal models at different academic levels and measure the fulfilment of the educational goals with respect to the target performance in a formal way.

III. AN OVERVIEW OF THE PROPOSED APPROACH
In this paper, we propose a multi-level evaluation approach that supports a substantive interest in the individual effects, group effects and the mutual effects of educational. The approach helps avoiding any unintentional complexities with multi-level goal modelling. This allows decision makers in academic institution to extract information from goal models at different academic levels, discovers patterns in large volume of details and investigate the effect of group attributes on individual business goal while accounting for nonindependence of observations. The evaluation approach may occur at six nested academic levels depending on the structure of the academic institution and granular details that are desired, as in Fig. 1. It starts at the course and/or curriculum at the bottom-level, the program and department at the middle-level and ends with the faculty and institution at the top-level when accumulated reviews are desired.
The general view of the multi-level evaluation approach of strategic educational goals is described in Fig. 2. It consists of five steps used at each level: 1) The Definition step is used to identify preliminary details, such as the strategic educational goals under evaluation, the target constituencies responsible for evaluation and collecting data, the direct/indirect assessment tools that will be used and the collection frequency or period of the evaluation.
2) The Assessment step provides two types of assessment techniques to measure the performance metrics of the strategic education goals: a) the paper-based, where documents and spreadsheets are used as a scoring guide, e.g., Rubric [19] and CAP [20]; and b) model-based where software modelling is used for goal assessment, e.g., GRL and i* [2].
3) The Evaluation step, where the output of the assessment techniques is accumulated to the next level of assessment and also is used to develop a set of recommendations for improvement.
4) The Adoption step, where the constituencies deal with two types of recommendations, the short-term recommendation which may be refined in order to meet the regulations and bylaws and adopted during the semester. Also, there is the long-term recommendation which may take a further discussion and approvals.
5) The Implementation step, where the adopted recommendations are implemented, and constituencies are informed of significant improvements during the public forums. The processes return to the assessment and repeat for another cycle. Fig. 3 describe the evaluation process of the proposed approach. The course evaluation process is a bottom-level where assessment seeks input from individual faculty members for each course taught. A course has specific student learning outcomes (SOs) designed to achieve a number of selected learning attributes such as knowledge solving problems, communication skills and leadership. Generally, the direct assessment results of the exams, quizzes, assignments, etc. are reflected to the grades. Based on the performance results of PEOs and SOs, a detailed summary of the improvement introduced in the course report; and a set of recommendations to be approved by a focus group consisting of faculty members who are considered in the knowledge field of that course.
At the curriculum evaluation process, teams of focus groups accumulate the individual contribution from all courses to the curriculum PEOs and SOs in order to assess the contribution of the entire curricula. This helps to minimize inconsistencies in teaching the courses by different faculty members and have a coherent structure of courses in the area.
At the program evaluation processes, the department council, all instructors and part of students are involved. Inputs are obtained through direct interaction with individual students and academic advisors are discussed in the department council and changes, corrective/preventive actions are proposed. The Proposed changes are discussed at the program level. Changes that do not conflict with the institution regulations and bylaws are implemented. While other actions that may conflict are submitted in a form of a proposal to the department council who may take action or may forward them to the Faculty Deans' Council for further discussion.
At the top-level management evaluation process, different councils accumulate the results of every program in order to assess the contribution of the entire programs, and the performance results of the PEOs that are reflected on the department mission.  Another step at the top-level management process is the faculty and institute evaluation process, where the accumulated performance results of the department missions are used as an assessment measures of the faculty mission. Finally, the accumulated performance results of the faculties' missions are used for assessment measures of the institute mission and vision. A set of recommendations is made by the faculty council or institute council for further improvements that comply with the institute mission and vision.

IV. USING MULTI-LEVEL GOAL MODELLING TECHNIQUE
In the proposed approach, a GRL modelling language was used to assess the strategic educational goals using goal models. The GRL goal model is described as part of URN language and is supported by jUCMNav [6]. Each element in the proposed data model is mapped into model element in the multi-level goal models. The high-level goals Vision and Mission are mapped into GRL Softgoal ; while the PEOs and SOs are mapped into the sub-goals . Model elements are linked together respectively and each one of them may contribute fully or partially at different assessment levels. The contribution relationship describes how an element participate to the other elements in GRL model; the contribution value ranges from -100% (negative), 0% (neutral) to +100% (positive).
At the course and curriculum assessment levels, courses that are selected for assessment are mapped into resources . Assessment tools are also mapped into the goal model as key performance indicators (KPIs) . KPI contains the constituency's achievement provided by the direct/indirect assessment tools and contributes to different strategic educational goals in the model. A KPI has an evaluation value that measures the current situation. It ranges between the -100% (negative), +0% (neutral) or +100% (positives) values. Constituencies are represented as actors to define their ownership and responsibility in the goal model. Fig. 4 defines an arbitrary multi-level goal model at two assessment levels. At the top-level goal model, two program educational objective PEO1 and PEO2 contribute to a Mission by 50% and 30% respectively; and at the higher level the Mission contributes to the institute Vision by 100%. There are also two indirect assessment tools (KPIs): Indirect Assessment2 and Indirect Assessment3 contribute both to the Mission by 10% and 20%, respectively. At the bottom-level goal model, a sample Course of the curriculum, represented as a resource contributes to two program educational objectives PEO1 and PEO2 by 25% and 50%, respectively.
To maintain the correlation between the multi-level goal model, traceability links are used between the PEO1 and PEO2 at the top-level and bottom-level GRL models; such that changes in their evaluation values at any GRL model will be reflected to the other models. Two student outcomes SO1 and SO2 contribute both to the Course by 75% and 60% respectively. There are also three assessment tools (KPIs): Direct Assessment1, Direct Assessment2 and Indirect Assessment1 are used for evaluating the student outcomes SO1 and SO2. They contribute by 10%, 25%, 15% and 20% respectively. www.ijacsa.thesai.org

V. A CASE STUDY OF THE PROPOSED APPROACH
We introduce a case study developed at Al-Ahliyya Amman University (AAU) [21], where 10 faculties were involved with a total of 38 departments and 50 programs. Three curriculum used for evaluations: fall, winter and summer with an average of 55 courses used for assessment for each program.
At the Definition step, the participation of the strategic educational goals, constituencies, assessment tools at different levels and an approximate evaluation period is described in Table I. The elements of strategic educational goals are involved such that the vision and mission reflect the objectives at the high levels and PEOs and SOs are used to measure the performance metrics at the low levels. Ideally, all constituencies need to participate actively, however, this ideal approach may not be achieved with ease and become consistent all the times.
The indirect assessment tools are used to evaluate the satisfaction of the constituencies that is reflected on the performance results. Different kind of surveys are used for that purpose with a motive of assessing the institution's regulations, polices and activities with respect to its vision, mission, PEOs and SOs. The common surveys used in academy are trend surveys, panel surveys, cohort surveys [22]. The direct assessment tools are used only at the course level using tools such as assignments, quizzes and exams. Periodic assessment of strategic educational goals varies between levels. It is defined by the participated consistencies during the evaluation process as in Table I. At the Assessment step, a sample of the case study models developed at the Civil Engineering Department in the Faculty of Engineering as in Fig. 5-10. The goal model, in Fig. 5, describes the strategic educational goals at institution level. The Deans Council is the major actor that contains a sample of four Faculties' Missions, each one contributes to the AAU Mission by 25%; while AAU Mission contributes to AAU Vision by 100%. There are also three indirect assessment tools (surveys), each contributes to AAU Mission by 20%. These tools measure the satisfactions of random constituencies with respect to the provided services.
The goal model below, in Fig. 6, describes the strategic educational goals at faculty level. The Faculty Council is the major actor that contains the mission of five engineering departments: computer engineering, civil engineering, communications and electronics engineering, electrical engineering and medical engineering, each one contributes to the Faculty of Engineering Mission by 20%. A traceability link between the Faculty of Engineering Mission in Fig. 6 and its equivalent in upper goal model as in Fig. 5. Two indirect assessment tools (surveys) are also used to measure the satisfaction of provided services at the faculty level. Each survey contributes to the Faculty Mission by 10%.

Course
PEOs, SOs Instructor, Enrolled Students   Every semester  At the department level, in Fig. 7, the goal model describes the strategic educational goals of the civil engineering department. The Department Council is the major actor that contains two programs: bachelor's degree in civil engineering and master's degree in intelligent transportation systems. Each of the programs contributes to Department of Civil Engineering Mission by 70% and 30%, respectively. A traceability link is defined between the Department of Civil Engineering Mission in Fig. 7 and with its equivalent in the upper goal model as in Fig. 6. Each program defines its own PEOs. Four PEOs contributes to bachelor's degree in civil engineering program by 25%; while three PEOs contributes to master's degree in intelligent transportation systems program by 25%, 50% and 25%, respectively. Two surveys are also used to measure the satisfaction of provided services at the department level. Each survey contributes to only the master's degree program by 10%.
At the program level below, in Fig. 8, describes the strategic educational goals of the bachelor's degree in civil engineering program. The Department Council is the major actor that defines three semester curricula of the program: first, second and summer. Each curriculum contributes to bachelor's degree program by 25%. There are also four PEOs contribute to semesters' curricula defined as follows:  PEO1: Succeed and excel in developing sound solutions to civil engineering problems.
 PEO2: Communicate and work competently in one or more core of civil engineering areas of practice or through graduate studies.
 PEO3: Work effectively and conduct themselves ethically in their professional environment and grow in their careers working on projects designed for the wellbeing of their society.
 PEO4: Be aware of contemporary changes and engage in life-long learning in their profession and acquire professional engineering registration.
Traceability links are defined between the bachelor's degree in civil engineering program and PEOs in Fig. 8 with their equivalents in the upper goal model in Fig. 7. Two surveys are also used to measure the satisfaction of provided services at the program level. Each survey contributes to first and second semester curriculum by 25%, respectively.
At the curriculum level below, in Fig. 9, describes the strategic educational goals of the Summer Semester Curriculum of the bachelor's degree in civil engineering program. A Focus Group in the civil engineering program is the major actor that defines two PEOs, PEO1 and PEO2, which contribute to the Summer Semester Curriculum by 30%. Traceability link is defined between the Summer Semester Curriculum in Fig. 9 with its equivalent in the upper goal model in Fig. 8. A sample of four courses are modelled: Transportation Engineering, Statics, Engineering Geology and Graduation Project (2). These courses contribute to PEO1 and PEO2 by 55%, 45% and 25%. There are also four student outcomes SOs [23] contribute to the four courses defined as follows:  SO1: An ability to apply knowledge of mathematics, science, and engineering.
 SO2: An ability to identify, formulate, and solve engineering problems.
 SO3: Recognition of the need for, and an ability to engage in life-long learning.
 SO4: Knowledge of contemporary issues.
At the course level below, in Fig. 10, describes the strategic educational goals of the 0863300 Transportation Engineering course of the Summer Semester Curriculum. The Instructor of the course is the major actor that defines three SOs that contribute to the 0863300 Transportation Engineering course by 30%, 40% and 25%, respectively. The 0863300 Transportation Engineering course contributes to two PEOs, PEO1 and PEO2, by 45% and 55%. Traceability links are defined between the 0863300 Transportation Engineering course and the PEOs in Fig. 10 with their equivalent in the upper goal model in Fig. 9. There are also four types of direct assessment tools: Quizzes, Assignments, Midterm Exam and Final Exam, that contributes to the three SOs.    The proposed multi-level GRL modeling approach provides a model-driven technique in defining the strategic educational goals augmented with metric indicators. It is a rationale tool for modeling, analyzing, validating and documenting the learning assessment process that would be useful in detecting and anticipating any deficiencies in meeting learning goals and objectives where corrective actions can be made. Table II describe a sample of the output of the multi-level goal models. These results are the initial measures which will be used for future continues improvements.
Several challenges were addressed during the case study practice due to large number of participated constituencies and lack of quality former performance measures. In such modeling approach, teams from different disciplines are required to meet periodically to discuss the modeling structure, define the modeling elements and relationship between them and assign the measure values of the model elements, such as contribution values. This may increase the chance of error prone and increase the period of becoming familiar with modeling approach.
Several group decision approaches, and techniques have been discussed in [24] that can be used to ease this challenge. We used a Round- Table Discussion and Consensus (RTD&C) approach where teams assembled in a dialog setting. Groupings of related choices, contained in models, are put up on a screen, and the team members are asked to discuss the model elements and assign their relative weights to each choice in each grouping.
The other challenges were related to how much the measure of the model elements are accurate. It is obvious that the validity of multi-level GRL models depends on the accuracy of the model element measures. Though, we found based on our practice that the accuracy of GRL modeling results deviate towards the improvement as the time proceeds and the participated constituencies are familiar with approach.
During the practice in the case study, we found that the proposed approach can be used as goal modeling data mining, where patterns can be discovered in a large and complex related data. Through the technique of digging and aggregating in the multi-level GRL goals, we were able to detect any deficiencies at the low assessment levels, such as the course or curriculum levels that would affect the top assessment level. VI. CONCLUSIONS AND FUTURE WORK Strategic educational goals in educational organizations with multiple level of management is promoted as a sort of correlated and clustered data. The paper-based approaches often neglect the relationship and dependencies between the educational goals during the evaluation of strategic educational goals augmented with quantitative indicators. This would complicate their analysis, produces ambiguity of the results, and causes different interpretations by the management. We propose a multi-level goal modelling approach that provides a model-driven method to improve the assessment of strategic educational goals. The approach handles the group of data efficiently and allows the individual and group level assessment to take effect simultaneously. As a future work, we plan to include the service operation procedures (SOPs) in the goal modelling to study the effects of services such as admission, registration, withdraw, etc. on the strategic educational goals.
ACKNOWLEDGMENT A special thanks for the ABET committee at the Faculty of Engineering of AAU.