Indirect Evaluation of Program Educational Objectives and Student Outcomes for Engineering Programs: A Case Study

Outcome and objective assessment is a method for determining whether students and graduates have learned, have retained, and can apply what they have been taught. Assessment plans have to include a statement of educational objectives, measurements of attainment of the objectives, and use of the assessment results for continuous improvement. In this paper, the experience of the Electrical Engineering program at Al Imam Mohammad Ibn Saud Islamic University is used as an example of how an indirect assessment method is established and implemented. Many surveys have been used and contributed to the numerical analysis for the indirect assessment of the program outcomes and achievements. The indirect assessment results indicate that the target levels are achieved for all the seven program outcomes and the three objectives that have been considered. Keywords-engineering; education assessment; learning outcomes; educational objectives; accreditation; assessment loop; ABET


INTRODUCTION
The last few years have witnessed a growing recognition of the need for assessment and evaluation of the learning outcome [1][2]. These are directed not only at large-scale, standardized tests but also at classroom assessment practices. At least two factors have contributed to the demands for assessment and evaluation: the changing nature of educational objectives and the relationship between assessment, teaching, and learning. Assessment is defined as one or more processes that identify, collect, and prepare the data necessary for evaluation. Evaluation is defined as one or more processes for interpreting the data acquired through the assessment processes to determine how well the program objectives are attained.
The Electrical Engineering (EE) Department at Al Imam Mohammad Ibn Saud Islamic University (IMSIU) has been implementing different assessment and evaluation approaches [3][4][5] for learning outcomes and Program Educational Objectives (PEOs), the concept of which is adopted to meet the international and national accreditation requirements. This paper describes an indirect method used to assess and evaluate the achievement of program outcomes and objectives. Student Outcomes (SOs) or program outcomes describe what students are expected to know and be able to do by the time of graduation while PEOs describe what graduates are expected to know and be able to do after graduation. The EE program is very keen on developing and implementing high quality education. For this reason, it requests periodic input from its constituents [6]. To improve the curriculum and the overall improvement of instructional delivery, review and evaluation of the program outcomes and objectives are conducted in each academic cycle. This principal feedback loop affects curriculum organization and content. Also, faculty members provide a collection of feedback forms that are specific to a course or relating to the entire program. The EE department receives feedback through these forms to encourage an unencumbered exchange of ideas and responses to the material presented in class and to the overall curriculum design. This is particularly helpful for students to be able to receive and benefit from current and most up to date content and knowledge. The EE program periodically consults the industry, to ensure that the department's efforts align well with its needs [7]. The EE program educational objectives serve its constituencies. The program constituencies are any entity or organizations that benefit from the outcomes of the program. The current EE educational objectives require periodic revisions to reflect the needs of the program constituencies in greater depth. The EE department keeps in contact with current and prospective employers of its graduates. Such constituents are strongly encouraged to join the industrial affiliates program which provides valuable feedback to the department faculty members for student preparation. When necessary, appropriate actions take place through the advising unit and/or the Quality Committee. Exit survey is very important in program outcomes evaluation. Alumni and employer surveys also provide important feedback regarding employment opportunities for students, and the opportunity to assess and evaluate the program educational objectives.
Although many studies have been published on the direct assessment methods of different engineering programs [8][9][10], the indirect assessment approach does not receive much attention. In this study, the measurement of SOs and PEOs achievements provides considerable information on the effectiveness of an EE program seeking to meet the accreditation criterion [11].

A. Program Outecomes or Student Outcomes
The EE program has adopted the new SOs for the engineering program as prescribed in Accreditation Board for Engineering and Technology (ABET) Criterion 3 [12]. The abilities that students must demonstrate at the time of graduation are attained through various courses taken by all students during the program, which are: 1. Identify, formulate, and solve complex engineering problems by applying principles of engineering, science, and mathematics. 2. Apply engineering design to produce solutions that meet specified needs with consideration of public health, safety, and welfare, as well as global, cultural, social, environmental, and economic factors. 3. Communicate effectively with a range of audiences. 4. Recognize ethical and professional responsibilities in engineering situations and make informed judgments, which must consider the impact of engineering solutions in global, economic, environmental, and societal contexts. 5. Function effectively on a team whose members together provide leadership, create a collaborative and inclusive environment, establish goals, plan tasks, and meet objectives 6. Develop and conduct appropriate experimentation, analyze and interpret data, and use engineering judgment to draw conclusions. 7. Acquire and apply new knowledge as needed, using appropriate learning strategies.

B. Program Educational Objectives
Graduates of the EE Program should attain at least one of the following educational objectives after graduation: • PEO1: Serve competently the needs of industry and academia by demonstrating high-quality knowledge, research, and skills in the area of EE.
• PEO2: Pursue professional development through professional study and self-learning with full gratitude of the importance of professional and ethical responsibility.
• PEO3: Contribute to the welfare of society through the responsible practice of engineering, leadership, and teamwork.

C. Consistency of the PEOs with the Mission of the Institution
The following statements summarize the University, College, and Department missions: • Mission1: High-quality Education (University/College/ Department).
• Mission 2: Conduct innovative research/leadership in scientific research/ (University/College/Department) • Mission 3: Local and global community service/offer professional services (University/College/Department).
• Mission 4: Equip students with the skills to be life-long contributors to their profession (Department). Table I shows the mapping between the Institution's missions and the PEOs indicating the consistency of the PEOs with the mission of the institution.

D. Relationship of SOs to PEOs
SOs support the PEOs. The first objective (PEO1) is supported by most of the SOs (1, 2, 5, and 6). The second objective (PEO2) is supported by 4, 5, and 7 SOs. The third objective (PEO3) is supported by SOs 4, 5, and 7. Table II summarizes the mapping between the SOs and the PEOs.
III. INDIRECT ASSESMENT Indirect assessment and evaluation at the EE program were conducted the academic years 2017-18 and 2018-19 (two cycles). The frequency of data collection and review process for the indirect assessment approach of SOs and PEOs are summarized in Table III.  The EE program has approved a benchmark level of outcome and objective achievement (two thresholds) for the indirect assessment. These thresholds are an average score of 20% (Unsatisfactory) and 70% (Satisfactory). For an outcome or objective to be considered achieved, the indirect assessment should reach an average score of at least 70% [13]. This should be considered as evidence illustrating that the level of the student outcome and program objective achievement is satisfactory.

www.etasr.com
Ghaly: Indirect Evaluation of Program Educational Objectives and Student Outcomes for Engineering …

IV. RESULTS OF THE EVALUATION PROCESS
In this section, the results and evaluation process are presented. At first the process in cycle 2 is discussed and then a comparison is performed between the results from cycles 1 and 2. To evaluate the results obtained from the surveys, the performance targets were defined to assess the level of attainment. Table III summarizes the tools for the indirect assessment and evaluation of program outcomes and objectives. The surveys listed in Table III, are being used in the assessment process at various stages. These surveys provide feedback on the weaknesses and strengths of the SOs and the PEOs. The exit survey contributes to the measurement of the indirect assessment of student outcomes 1-7 and provides the program with useful information to aid the development of an action plan for continuous improvement.

A. SO Assesment through the Exit Survey
At the end of each semester, the EE program carries out a survey of its graduating students. The exit survey contributes to the measurement of the indirect assessment of student outcomes 1-7 and provides useful information that aids the development of an action plan for the continuous improvement of the program. In this survey, it is useful to ask students to rate the program learning outcomes or SOs in order to indicate the extent of their achievement. The total number of respondents for both cycles is 40 students. The survey questions are related to the specific outcomes 1-7 using the same key performance indicators (KPIs) that are used in direct assessment [14][15][16] as seen as example for the outcome 1 in Table IV. The data collected from the students' evaluation are stored and analyzed electronically and the results are considered for further evaluation and taking appropriate actions. Your ability to apply mathematical and scientific principles to formulate models and systems relevant to electrical engineering.

1.2
Your ability to solve electrical engineering problems by using the concepts of integral and differential calculus and/or linear algebra.

1.3
Your ability to execute calculations correctly

1.4
Your ability to translate academic theory into engineering applications.

1.5
Your ability to apply analytical, graphical or numerical methods to solve complex problems

1.6
Your ability to solve practical engineering problems logically using correct theoretical concepts.

1.7
Your ability to formulate alternative solutions to a complex engineering problem.

1.8
Your ability to identify the governing concept of a complex engineering problem.
The results obtained for the outcome 1 are summarized in Figure 1. This Figure is shown as an example of the average obtained for each KPI of the outcome 1 from the survey questions on a scale of 5 is very satisfied down to 1 for very dissatisfied and then changed to percentage and shown graphically. Figure 2 represents the exit survey results of cycle 2 (fall 2018 and spring 2019) based on all program outcome assessment data (1 through 7). KPIs assessment results-outcome 1. The achievement is considered satisfactory if the weighted average is greater than or equal to 70%. As seen, all program outcomes meet the program expectations.

B. Comparison of Assessment Results
The comparison of the direct assessments' results of the program outcomes (SOs 1-7) for both cycles (2017-18 and 2018-19) is depicted numerically in Table V and graphically in Figure 3, where the satisfactory results for cycles 1 and 2 are shown in blue and red respectively. It can be observed that the program outcome attainment for both cycles is well above the satisfaction target. However, some attainments in cycle 1 have degraded to some extent as compared to cycle 2. This issue was investigated in the early part of the 2019-2020 fall and the faculty's opinion was sought to suggest measures for improvement.

www.etasr.com
Ghaly: Indirect Evaluation of Program Educational Objectives and Student Outcomes for Engineering … Fig. 3.
Comparison of the two cycles of indirect assessment results.

C. SO and Objective Assessment from the Alumni Survey
During both cycles, the EE Department conducted a survey of alumni to assess their knowledge, skills, and perceptions of relevant aspects of their college experiences in order to evaluate the program objectives. The alumni were asked many questions, including SOs and program objective achievement. The presented data are based on a random sample of 13 alumni. Again the achievement is considered satisfactory if the weighted average is greater than or equal to 70%. The results are shown in Table VI and it can be observed that program outcomes and objectives indicate satisfaction. If a weighted average was less than 70% for an SO, it would be discussed in the EE Quality Committee to analyze the reason(s) behind it. The matter might also be escalated to the Department Council for remedial measures to be taken.

D. SO and Objective Assessment from the Employer Survey
The employer survey is also assessed at intervals of 3 years. A set of randomly selected employers are requested to fill a questionnaire available online. The employer was asked many questions, including SO and program objective achievement. The presented data are based on a sample of 6 employers. The same satisfaction criterion of 70% is applied. The results are shown in Table VII and it can be observed that all program outcomes indicate satisfaction.

TYPICAL INDIRECT ASSESMENT EMPLOYER FORM
The following section of the questionnaire is intended to seek your feedback on "Demonstrated Abilities of our Graduates" while they are employed at your company:    Also, a total of 23 students (seniors) responded to the contacted survey. These 23 students are all in level 7 and 8 classes. The students were presented with the formulation of PEOs along with a brief explanation for each PEO. The results and of the survey are shown in Table IX  More than 65% of the surveyed EE senior students are either satisfied or very satisfied with the EE's PEOs V. CONCLUSION In this work, various surveys were used and contributed to the indirect assessment of the program outcomes and objectives attainment. The results of the exit survey for the last two cycles show that students' perspective on their attainment of the program outcomes is satisfactory, since all the student outcomes are above the minimum required level of attainment. A meeting with the students is arranged every semester to clarify the objectives and the content of the indirect assessment. Additional surveys, such as the employer survey and the alumni survey are assessed every three years to get feedback from the employers and the alumni in the market about the program objectives and at which level they met their carrier requirements.