Engineering Education for Sustainable Development: Evaluation Criteria for Brazilian Context

: Considering the increasing importance of sustainability in future professionals’ education and the role played by engineers in society, this paper aims to analyze the key criteria that should be considered in models to evaluate the insertion level of sustainability into engineering education, considering the Brazilian context. For this, criteria reported in the literature were collected and evaluated by engineering professors. The respondents were asked to classify the criteria as “essential”, “useful, but not essential”, or “not necessary”. Data collected were analyzed through Lawshe’s method. From 15 criteria collected from the literature, 5 were not considered essential to evaluate engineering education for sustainable development (EESD), according to data analysis: C2 (establishment of global partnerships), C4 (encouraging students to volunteer through extracurricular activities), C9 (use of active learning approaches to problem solving to teach aspects related to sustainability), C10 (use of service-learning towards the local community for educational purposes) and C15 (use of sustainability concept in university installations). It was possible to verify that most of these criteria (C2, C4, C10, and C15) were not directly related to engineering curricula, being parallel activities. Regarding C9, active learning approaches can enhance attributes important for students in the context of sustainable development, but they are not goals of EESD. This research contributes to the development of evaluation models for engineering education in the Brazilian context and its ﬁndings can also be useful for studies in other countries. No similar study was found in the literature.


Introduction
Sustainable development has been debated in the literature for years [1][2][3][4][5]. Proper use of natural resources, the use of renewable sources of energy [6,7], and corporate social responsibility have been increasingly relevant for companies [8][9][10]. In this scenario, the publication of 17 Sustainable Development Goals (SDGs), in 2015, by the United Nations [11] can be considered a landmark, as highlighted by Martins et al. [12]. Among these goals, besides the relevance of organizations to enable sustainable development, educational system contribution for it is also emphasized [13,14]. The United Nations Educational, Scientific and Cultural Organization (UNESCO) [15] highlights that education for sustainable development (ESD)-considered within Quality Education (Goal 4)-has a key role to support the achievement of the other 16 goals. The need to develop knowledge and competences in students to enable them to act towards sustainable development is specified in Target 4.7 (from Goal 4) [13].

Theoretical Background
The literature has been showing several issues to be addressed in engineering education for sustainable development (EESD). One of the most relevant issues cited is the need for a transdisciplinary approach [22,[33][34][35]. As argued by Tejedor et al. [17] and Shields et al. [36], more than addressing interfaces among disciplines, transdisciplinarity overcomes their boundaries and articulates different perspectives through a holistic approach.
The second criterion to be mentioned is the establishment of global partnerships. Lazzarini et al. [37] highlight the need to show students the global dimension of engineering education and the global social and environmental responsibility of their actions. In this sense, the establishment of institutional strategies of global partnerships [33] can be a useful means to reach education for sustainability service [38] and to enhance the search for sustainable development through the integration of countries' competences and knowledge [39]. Despite the benefits, challenges are faced for these types of partnerships, when increasing competition among HEIs is established [37].
Another issue related to institutional strategy is its alignment with sustainability insertion and top management support for this insertion. As argued by Iyer-Raniga and Andamon [40], the sustainability insertion in higher education needs to be performed strategically. In addition, these authors highlight that top management also needs to learn about sustainability for this concept to be effectively reached. Indeed, Holgaard et al. [38] and Rampasso et al. [27] emphasize that the lack of this support is considered a barrier for ESD.
The importance of encouraging students to volunteer through extracurricular activities is also addressed. According to McCormick et al. [41], volunteer actions may enhance students' knowledge and understanding of social issues, which can impact their professional actions. Focusing on students who participate in Enactus-a non-profit international organization that engage students in social actions-Rampasso et al. [14] highlight the increased ability of these students to critically analyze ESD.
Another issue to be evaluated regarding EESD is the balanced focus among environmental, social, and economic aspects of sustainability. Edvardsson Björnberg et al. [42] mentioned the difficulty faced by the KTH Royal Institute of Technology (KTH) to address the social aspect in engineering education whereas environmental concepts were properly integrated into engineering programs. In Akeel et al. [43], the same difficulty is mentioned. Regarding sustainability aspects, the economic category is also mentioned in the literature as less considered than environmental issues in engineering education [42,43]. Raoufi et al. [44] highlight the importance of preparing engineering students to consider social, economic, and environmental aspects in their professional decisions. For this, the authors point out the need for teaching decision tools to support students.
A sixth criterion to be evaluated according to the literature is critical thinking development in students. As argued by Guerra [16], critical, systemic, and holistic thinking are required to enable students to insert sustainability aspects into their professional activities and continuously learn to improve their actions. McWhirter and Shealy [35] emphasize the role of active learning approaches, such as problem-based learning, to develop this critical thinking in students and make them consider sustainable development issues in their decisions. Other authors corroborate with this argument and add that this kind of teaching approach enhances systematic and holistic thinking, as well. Among the reasons for this compatibility is the transdisciplinary character of sustainability [16,17].
These arguments justify the need for active learning approaches to problem solving. Among these approaches, PBL is among the most cited. PBL is centered on students conducting activities within teams and needing to search for solutions to real-life problems. In this approach, students are responsible for their learning process and have to be proactive to acquire knowledge and solve proposed problems [16]. In this sense, it is easier to reach a transdisciplinary solution with this kind of approach [16,17]. Specific approaches resulting from active learning proposals can also be interesting. Ramanujan et al. [45] presented an approach to insert environmental sustainability into mechanical engineering courses through a guided discovery approach.
Another criterion relevant for EESD is the presence of discussions of issues related to values and ethics with students throughout the course. Biswas [46] points out that future engineers will encounter demands for ethical behavior concerned with the social and environmental impacts of their work. For Segalàs et al. [47], values and ethics are core competencies for engineering students to learn about and apply for concepts of sustainable development. Byrne et al. [48] emphasize the need for a curricular change for engineers to prioritize ethical behavior rather than exclusively meeting customers' demands without considering the negative consequences. An interesting manner to teach ethics and values for engineering students can be through service-learning.
Service-learning enables students to learn through actions toward the local community [17]. More than learning manners to create things as engineers, service-learning develops students as citizens, increasing their compassion; social skills, such as working within teams; and emotional intelligence [49].
Focusing on the technical aspects of engineering knowledge, another important issue to be addressed is the application of technical knowledge towards sustainability goals. Raoufi et al. [44] highlight the need to prepare students to perform life cycle assessment. In addition to life cycle assessment, Biswas [46] also presents other industrial ecology issues that can be used in EESD. Among them, eco-efficiency and cleaner production can be highlighted as complementary strategies; while cleaner production focuses on operational process improvements to reduce environmental negative impacts, eco-efficiency focuses on greater efficiency in value creation. Regarding social abilities, communication skills are important to enable students to work within multidisciplinary groups [16,46]. Shields et al. [36], however, highlighted that engineering students fail to present communication abilities. For Guerra [16], PBL is an approach able to develop these abilities in students. This approach is frequently mentioned in the literature.
Adequate and constantly updated teaching material that includes sustainability in the course is another criterion to be analyzed for EESD. In addition to a lack of materials for professors to properly provide transdisciplinary knowledge, the lack of constant updates in their content is a challenge when considering the dynamic character of sustainability-related concepts [50,51].
Despite the lack of proper didactic material, professors need to be prepared to support students learning in a sustainable development context. This is especially required because professors need to change their actions and acquire new abilities to work in this new reality [40]. Mulder [52] presents several comments of engineering professors that clearly show they do not understand the role engineers need to play towards sustainable development, which demonstrates the necessity of the mentioned training.
Finally, regarding university installations, there are several opportunities to use sustainability concepts. Among these opportunities, the efficiency in water use, ecologically correct buildings, energy efficiency, and appropriate waste management can be highlighted [38,53].
From this literature overview, it was possible to establish Table 1.

Methodological Procedures
In order to verify the criteria should be considered in models to evaluate the sustainability insertion level in engineering education, this research was developed in two stages: a search in the literature and survey data analysis. Figure 1 shows the steps followed in these stages.

Methodological Procedures
In order to verify the criteria should be considered in models to evaluate the sustainability insertion level in engineering education, this research was developed in two stages: a search in the literature and survey data analysis. Figure 1 shows the steps followed in these stages. A search in the literature was conducted to collect the criteria to be analyzed. The strings "engineering education" AND "sustainability"; "engineering education" AND "sustainable development" in articles' abstracts were used to search for articles in Emerald Insight, Science Direct, and Taylor and Francis. Articles were analyzed and criteria pointed out as relevant for EESD were collected. Only articles from international journals were considered (proceeding were excluded from the sample). From this analysis, 15 criteria (presented in the theoretical background section) were identified and used to structure a questionnaire for a survey of engineering professors in Brazil experienced with sustainability issues. The survey was performed after approval from the university's ethics committee.
To verify which criteria collected in the literature were considered essential, according to engineering professors in Brazil, experts in the subject, Lawshe's method [64] was used. The aim of this method is to quantify consensus. Of course, when all respondents attribute "essential" for an item or no one does, the consensus is clear. However, for scenarios presenting divergencies, Lawshe's method enables the consensus quantification. There are two assumptions in this method: content validity starts to be perceived when at least 50% of experts attribute "essential" for the item; and the content validity degree increases according to the number of experts classifying the item as "essential" [64].
In this method, content validity ratio ( / / ) was used to identify the essential criteria.
For this calculus, experts classified the criteria as "essential", "useful, but not essential", or "not necessary". The number of "essential" answers ( ) was used in the CVR calculation as well as the number of responses (N). CVR value will be negative when less than half of respondents classify the  A search in the literature was conducted to collect the criteria to be analyzed. The strings "engineering education" and "sustainability"; "engineering education" and "sustainable development" in articles' abstracts were used to search for articles in Emerald Insight, Science Direct, and Taylor and Francis. Articles were analyzed and criteria pointed out as relevant for EESD were collected. Only articles from international journals were considered (proceeding were excluded from the sample). From this analysis, 15 criteria (presented in the theoretical background section) were identified and used to structure a questionnaire for a survey of engineering professors in Brazil experienced with sustainability issues. The survey was performed after approval from the university's ethics committee.
To verify which criteria collected in the literature were considered essential, according to engineering professors in Brazil, experts in the subject, Lawshe's method [64] was used. The aim of this method is to quantify consensus. Of course, when all respondents attribute "essential" for an item or no one does, the consensus is clear. However, for scenarios presenting divergencies, Lawshe's method enables the consensus quantification. There are two assumptions in this method: content validity starts to be perceived when at least 50% of experts attribute "essential" for the item; and the content validity degree increases according to the number of experts classifying the item as "essential" [64].
In this method, content validity ratio (CVR = n e −(N/2) N/2 ) was used to identify the essential criteria. For this calculus, experts classified the criteria as "essential", "useful, but not essential", or "not necessary". The number of "essential" answers (n e ) was used in the CVR calculation as well as the number of responses (N). CVR value will be negative when less than half of respondents classify the item as "essential". Ayre and Scally [65]  ). It is important to emphasize that Ayre and Scally [65] state that their values and Lawshe's values for the critical number of experts are generally equal, validating Lawshe's method. After Lawshe's method analysis, the findings were compared with the literature. In the next section, the results from the analysis are presented.

Results and Discussion
The sample of this study was professors of engineering in Brazilian universities. A total of 35 professors answered the survey. Most of them had Ph.D. degrees. Regarding experience as professors, 51.43% had up to 10 years of experience; 40% had between 11 and 20 years of experience; and 8.57% had more than 20 years of experience. All respondents had experience related to sustainability, through research and/or teaching. The respondents were asked to evaluate criteria from the literature as "essential", "useful, but not essential", or "not necessary". As presented in the methodological procedures section, the number of "essential" answers were used to calculate CVR values. Since 35 experts answered the survey, the CVR critical value was 0.31 according to Lawshe [64] and 0.314, according to Ayre and Scally [65] calculus. This similarity is aligned with Ayre and Scally's [65] statement about the equality between their values and Lawshe's values for the critical number of experts. In another perspective, at least 23 respondents must attribute "essential" to the criterion [65]. The results for this sample are presented in Table 2. It is worth highlighting that Lawshe's [64] equation (CVR = n e −(N/2) N/2 ) was used for the calculus. The criteria considered essential by the experts were: C1 (use of transdisciplinarity in teaching); C3 (alignment between sustainability insertion and institutional strategy, with top management support for needed adjustments); C5 (balanced focus among environmental, social, and economic aspects of sustainability); C6 (development of critical thinking in students throughout the course); C7 (development of holistic and systemic thinking in students throughout the course to enable them to make decisions responsibly); C8 (discussion of issues related to values and ethics with students throughout the course); C11 (constant discussion, throughout the course, of industrial applications of technical knowledge for sustainability (for example, life cycle assessment, cleaner production, ecologically efficient strategies for resources use, etc.)); C12 (development of communication skills in students to enable them to work within multidisciplinary groups); C13 (availability of adequate and constantly updated teaching material to include sustainability in the course); and C14 (proper training of professors to insert sustainability into their disciplines).
Regarding the criteria validated as essential by the experts, it is possible to verify that seven of them are directly related to the goals of EESD: use of transdisciplinarity in teaching (C1); balanced focus among environmental, social, and economic aspects of sustainability (C5); development of critical, holistic, and systemic thinking in students (C6 and C7); discussion of issues related to values and ethics (C8); discussion on industrial applications of technical knowledge for sustainability (C11); and development of communication skills in students to enable them to work within multidisciplinary groups (C12). It is possible to observe that all these criteria focus on enhancing students' knowledge and skills related to sustainable development and they may be integrated into engineering education classes.
The other three criteria (alignment between sustainability insertion and institutional strategy, with top management support for needed adjustments-C3; availability of adequate and constantly updated teaching material to include sustainability in the course-C13; and proper training of professors to insert sustainability into their disciplines-C14) are necessary to make the other seven validated criteria viable.
As it is presented in Table 2, C2 (establishment of global partnerships), C4 (encouraging students to volunteer through extracurricular activities), C9 (use of active learning approaches to problem solving to teach aspects related to sustainability), C10 (use of service-learning towards the local community for educational purposes) and C15 (use of sustainability concept in university installations) did not receive enough "essential" answers to be considered essential criteria.
Analyzing these five criteria, it is possible to verify that C2, C4, C10, and C15 are not directly related to engineering curricula, being parallel activities. Regarding the establishment of global partnerships, despite its importance for students to integrate countries' competences and knowledge [39] and become aware of their role as global citizens, it may be a great challenge when globalization causes increasing competition among universities, distancing academics worldwide [37].
The encouragement of students to volunteer through extracurricular activities was not considered essential, too. Here it is worth emphasizing that this does not mean experts did not consider extracurricular activities important, but they do not consider them essential as a criterion for an evaluation model focused on engineering courses. Although this kind of activity may enhance students' knowledge and perception of social issues [41], experts do not consider this encouragement an essential role of HEIs.
A similar analysis can be addressed for the use of service-learning towards the local community for educational purposes. The benefits from it mentioned in the literature [17,49] are not being judged, but the responsibility of engineering courses to make this bridge can be contested.
As mentioned in the literature [38,53], the use of the sustainability concept in university installations presents several opportunities and can benefit the environment and society. However, it may not have a strong connection between HEIs' good practices and students' learning regarding sustainability. The engagement of students in activities to improve HEIs' practices related to sustainability may present better results than implemented practices themselves.
The exclusion of the use of active learning approaches to problem solving to teach aspects related to sustainability from the essential criteria list can be explained by the fact of these approaches being used as a possible means to achieve several attributes for students, such as critical thinking, systemic thinking, and communication skills, among others [16]. Thus, the use of these approaches can enhance these achievements, but they are not goals of EESD.

Conclusions
This article aimed to analyze the key criteria that should be considered in models to evaluate the insertion level of sustainability into engineering education, considering the Brazilian context. After a literature analysis, experts were asked about the relevance of each criterion identified as potentially essential for EESD programs. Data were analyzed with Lawshe's method. According to the sample, C2 (establishment of global partnerships), C4 (encouraging students to volunteer through extracurricular activities), C9 (use of active learning approaches to problem solving to teach aspects related to sustainability), C10 (use of service-learning towards the local community for educational purposes), and C15 (use of sustainability concept in university installations) were not considered essential criteria. Thus, from the 15 criteria collected, 10 were considered relevant to compose an evaluation model.
Regarding the criteria validated as essential by the experts, it is possible to verify that seven of them (C1-transdisciplinarity; C5-balanced focus of sustainability; C6-critical thinking; C7-holistic and systemic thinking; C8-values and ethics; C11-industrial applications of technical knowledge for sustainability; and C12-communication skills to work within multidisciplinary groups) focus on enhancing students' knowledge and skills related to sustainable development and they may be integrated into engineering education classes. The other three criteria (C3-strategy alignment; C13-teaching material for sustainability teaching; and C14-professors' training) are necessary to make the other seven validated criteria viable. The five excluded criteria were not necessarily pointed out as unimportant, but they were considered not relevant to compose an evaluation model for EESD in the Brazilian context. It is possible to verify that most of these criteria (C2, C4, C10, and C15) are not directly related to engineering curricula, being parallel activities. Regarding C9, active learning approaches can enhance attributes important for students in the context of sustainable development, but they are not goals of EESD. These analyses may explain the reason for excluding the five criteria from evaluation models. In this sense, the main contribution of this research was to collect relevant criteria from the literature to be used in evaluation models to assess EESD levels and identify, through experts' perception, those criteria that were essential to compose these models in the Brazilian reality.
Limitations of this study should be mentioned. This was exploratory research. Thus, findings are limited to the sample studied and the articles evaluated. However, it should be highlighted that criteria to evaluate EESD in the Brazilian reality are scarce in the literature and many of the items analyzed were obtained after a deep analysis of articles.
For future studies, there are several paths researchers can follow. Among them, it can be suggested: the establishment of an evaluation model from the essential criteria found, presenting indicators to measure the criteria; replication of this study in other countries and comparison with Brazilian experts; and case studies verifying the criteria's-considered essential in this study-application in engineering courses with different maturity levels.