Evaluation of an NSF Research Experience for Teachers (RET) Program for STEM Development: Water-Energy Education for the Next Generation (WENG)

In recent decades much attention has been given to the optimization of professional development to support education reform; especially as it relates to science, engineering, technology and mathematics (STEM) fields. Many studies have shown that the most effective STEM professional development programs include active inquiry opportunities that take place over long durations. The Colorado School of Mines (CSM) has developed, and hosts, a professional program that works to address these needs. The Water-Energy Education for the Next Generation (WENG) is a National Science Foundation funded Research Experience for Teachers (RET). The WENG program is a sixweek summer training that immerses K-12 educators in state-of-the-art research at CSM through the highly interdisciplinary lens of the water-energy nexus. The WENG model also includes industry field trips, content and pedagogy workshops and book club focusing on science communication. Program effectiveness is evaluated through surveys given to participants before, during and after the program. Preand post-program surveys indicate that WE2NG supports teacher confidence and growth in teachers of all grade levels and that participant satisfaction outcomes are likely related to the ratio of elementary, middle and high school teacher participants. Surveys given during the program reveal consistently positive feedback and that satisfaction with collaborative program elements is related to program duration. Overall, survey results illustrate that the WENG RET program is an effective professional development model for K-12 STEM educators.


INTRODUCTION
Over thirty years ago, The National Commission on Excellence in Education (NCEE) published A Nation at Risk, urging the US Department of Education to reform the American education system. The report describes an ongoing decline in the United States' global competitiveness in education, and especially in science, technology, engineering and math (STEM) fields. Recognizing our emergence into the 'information age', the NCEE warned that "the educational foundations of our society are presently being eroded by a rising tide of mediocrity that threatens our very future as a Nation and a people". While critics of this report have rejected this tone of "gloom-and-doom", (Association for Supervision and Curriculum Development) the NCEE supports its claims with empirical evidence.
Since then, many reports have shown little-to-no improvement in U.S. educational measures; a significant contributor to America's "narrow lead" in overall prosperity. These more-recent reports have pointed to a lack of teacher enthusiasm and preparedness as a major culprit in educational decline (National Academy of Sciences). While the majority of high school science teachers have completed a science degree, less than half of middle school science teachers and only five percent of elementary science teachers hold science degrees. Thus, it is likely that the majority of K-12 STEM educators have experienced minimal opportunities to engage in scientific investigations that would support their ability to deliver STEM curriculum effectively (Wilson, et al).
In response to these warnings, initiatives to improve the American education system (especially in STEM fields) have arisen. These initiatives have taken many forms, but perhaps most-notably are improvements in curriculum and professional development programs for educators. These, of course, are not mutually exclusive, as advancement of curriculum calls for increased educator understanding of content material. The National Science Education Standards (1996) describes an effective secondary-level science teacher as one that is "familiar enough with a science discipline to take part in research activities with-in that discipline". The Next Generation Science Standards: For States, By 4 SUMMER 2020 VOLUME 8 ISSUE 2

ADVANCES IN ENGINEERING EDUCATION
Evaluation of an NSF Research Experience for Teachers (RET) Program for STEM Development Teachers (RET) (NSF). RET programs are hosted by colleges and universities across the U.S. and provide an internship-type research experience for K-12 educators. As noted above, many K-12 teachers lack exposure to post-secondary and applied STEM. The RET provides an opportunity for interested teachers to be immersed (usually for many consecutive weeks) in scientific research under the guidance of college or university research faculty and graduate students. These programs have been shown to improve teachers' self-confidence in their science knowledge and in relating applied science to their classroom practices through inquiry-based pedagogy (Saka; Klein-Gardner, Johnston, and Benson; Trenor et al.) , and to indirectly improve retention of student motivation (Klein-Gardner and Spolarich).

RET WE 2 NG PROGRAM
The Colorado School of Mines (CSM) hosts a summer RET program for local public K-12 STEM teachers. CSM is one of the world's leading science and engineering schools with a long-standing and successful relationship with industry partners. With a focus on the water-energy nexus, the CSM RET site -Water-Energy Education for the Next Generation (WE 2 NG) -seeks to enhance STEM teachers' content knowledge and to advance public knowledge and dialogue on the water-energy nexus through integration of teachers, and ultimately cutting-edge water-energy research and technology. The U.S. Energy Information Administration and American Wind Energy Association rank Colorado 7 th in the nation for total energy production of non-renewables (crude oil, natural gas and coal), 10 th in the nation for installed solar capacity (U.S. States -Rankings) and 9 th in the nation for installed wind capacity (State Wind Energy Facts). Colorado is also home to the headwaters of the Colorado River -the vitality of which seven western states and parts of Mexico depend on for water supplies, per the Colorado River O3: To impact K-12 STEM curricula via the creation of standards-based active learning lessons infused with current research which will be available through local, regional, national, and global forums.
Since each RET program uses a different approach to support teachers, comparability between programs can be difficult. Consequently, the body of literature that evaluates these programs also employs various approaches, or metrics, to describe program success. The purposes of the current study are to 1) evaluate the effectiveness of the WE 2 NG RET model as a means of professional development, in a way that provides insight into the successful structuring of future RET programs, and 2) to introduce a non-parametric method for evaluating how various program elements change with time (given small participant populations, n<30). To our knowledge, the non-parametric method employed in this study has not been utilized in professional development evaluation but is a valuable tool for evaluating change over time and has been previously and successfully used in natural science studies.
The study described herein aligns most closely with the first program objective (O1), evaluating the impact of the WE 2 NG program on teachers and how that relates to the structure of the program as a professional development model. We assume that by impacting teachers (O1), that their students will be similarly impacted (O2). However, directly addressing O2 and O3 is more difficult due to limitations on obtaining student information. Specifically, the research questions motivating this study include 1) What WE 2 NG RET program elements have the greatest control on teacher experience and is teacher learning affected by those elements? And 2) How does teachers' confidence (in their understanding of STEM curriculum) change due to their participation in the WE 2 NG RET program?

General Program Logistics
The WE 2 NG program supports teacher participants to attend a full-time (40 hours/week) paid six-week summer training at CSM where they engage in research under the direction of faculty and graduate student mentors. Participants are paired with a research group on campus (including a faculty advisor and graduate research assistants) based on their STEM content interests and the scope of work of the research group. All research groups affiliated with the WE 2 NG program have a water and energy focus; with research topics including the remediation and reuse of water affected by energy production. The general weekly structure of the WE 2 NG program includes; curriculum development (Mondays and Wednesday afternoons), research within faculty research groups ( Tuesday, Wednesday mornings and Thursdays), and industry field trips (Fridays).
The current RET WE 2 NG grant has supported a total of 27 Denver-metro-area teachers over the course of the first three summers (nine new teacher participants per summer). Due to proximity,

ADVANCES IN ENGINEERING EDUCATION
Evaluation of an NSF Research Experience for Teachers (RET) Program for STEM Development most WE 2 NG participants teach in the Jefferson County School District (JeffCo) -the second largest K-12 school system in Colorado. The demographics in JeffCo vary extensively from school-to-school, with minority populations exceeding 85% and students on free/reduced rates exceeding 90% in certain areas. Initial participant recruiting efforts (for the first summer) were targeted at the Golden and Jefferson articulation areas within JeffCo. These areas were chosen based on greatest potential impact of STEM professional development, the opportunity for vertical teaming benefits and the relative proximity to CSM to enable sustained long-term interaction between CSM faculty and ists from JeffCo and Denver Public Schools. The inclusion of a district curriculum specialist in the WE 2 NG program was a huge asset to the other teachers participating that summer and played a critical role in vertical and horizontal collaboration efforts.

Summer Program
WE 2 NG summer trainings begin with an orientation session that includes an introduction to campus and the various research projects connected to the WE 2 NG program, as well as a laboratory safety training. In the weeks following orientation, teachers spent two and a half days per week immersed in a research project that best fit the needs of their classroom goals. They also spent one day per week at a teacher-training workshop (various topics), one day per week on an industry field trip related to water and energy, and a half day focusing on science communication; through a book club led by new teachers and through curriculum development led by master teachers. The book club was designed with the intention to increase participant exposure to: 1) regional water and energy issues and 2) to explore methods of science communication used in different types of science literature (Table 1).

ADVANCES IN ENGINEERING EDUCATION
Evaluation of an NSF Research Experience for Teachers (RET) Program

for STEM Development
The time reserved for curriculum development created an opportunity for new teachers to work on lesson planning in a space conducive to alignment of curriculum, both horizontally and vertically. At least one 'working-lunch' per week included a research presentation by graduate students involved in work relating to the water-energy nexus. Additional optional opportunities arose throughout the program, including helping other research groups with data collection (on or off site), field activities and campaigns, or instrumentation assembly. The final week of each program included a 'showcase of lessons' where teacher participants presented a final deliverable, in the form of a lesson or unit plan that met the needs of their classroom, to WE 2 NG faculty, CSM students and local school district personnel. After the showcase, the program closed with an extended overnight field trip to parts of Colorado that face unique water and energy challenges (Colorado's Rio Grande basin in 2017 and Colorado's West Slope in 2018). The extended field trip included a variety of industry tours, researchsite visits and meetings with local water-energy stakeholders. As an example, a detailed outline of the 2017 WE 2 NG summer training calendar is presented in the Supplemental Information.

Data Collection: Surveys
WE 2 NG program evaluation was achieved entirely through surveys (using the Google Forms product) administered to teacher participants during the program. Surveys were given immediately  To satisfy the requirements of the NSF funding supporting the WE 2 NG program, all survey questions were collaboratively designed with, and ultimately approved by, an external program evaluator.
Hence, evaluation of WE 2 NG was confined by external evaluator oversight (on surveys). Additionally, program managers are only allowed to evaluate teacher development without obtaining information on their students. A CSM Internal Review Board (IRB) Exemption for Human Subjects Research was obtained to protect the privacy interests of teacher participants. Approval form the JeffCo Review of External Research (JRER) Committee was also obtained to conduct this research and protect the identities of the students that would be affected during the following school year. To encourage un-biased feedback, survey participants were assigned random numerical-identifiers, which were kept anonymous to the authors of this study. To allow time for reflection, but also encourage timely responses weekly surveys and the post-survey were made available for a short window of time (2-3 days for weekly and approximately 1.5 weeks for post). For purposes of consistency, master teachers did not participate in these surveys.

Data Analysis
Analysis was undertaken for new participants (non-master teachers) who participated in both the 2017 and 2018 WE 2 NG cohorts. Surveys included both written/non-numerical and quantitative ranked/numerical responses. The results presented here primarily consider numerical survey results, however, because all survey questions were qualitative in nature, the presented analysis is considered qualitative, and any potential error associated with the results are not quantified. Due to a low number of survey participants (n = 9 in 2017 and n = 9 in 2018) question responses do not fit a normal distribution and are evaluated using a non-parametric approach.
To determine if participant feedback is related to program duration, Kendall rank correlation coefficients ("Kendall's Tau") were determined for median weekly survey responses (note: the median is used here because the median does not assume a distribution). Since Kendall's Tau (KT) provides information on whether a monotonic relationship exists between two variables (in this case; feedback and program duration), and does not imply correlation type, Kendall-Theil robust lines (KTRL) were also fitted to weekly survey responses to determine how/if feedback truly changes with time.

Kendall-Theil Robust Lines:
Where Y is the dependent variable (survey responses), X is the independent variable (week), b 1 is the slope of the line (found by taking the slope between all possible data pairs; the median slope is b 1 ) and b 0 is the y-intercept, defined as follows; Where Y med and X med are the median Y and X values respectively.

Kendall's Tau (τ):
Where n is the number of data pairs and S is defined as follows; Where P is the number of times Y and X increase together, and M is the number of times Y decreases as X increases.
To our knowledge, Kendall-Theil robust lines and Kendall's Tau correlations have not been previously been applied to any similar educational studies. However, KTRLs and KT correlations are appropriate and well-suited inferential statistical methods for small data sets that do not fit a probability distribution. These methods are applied here because they are not strongly affected by outliers (Helsel and Hirsch) and because the datasets are small. Another important consideration is that the sample of responses under question represent 100% of teacher participants for the 2017 and 2018 WE 2 NG programs but represent only a small convenience-sample of STEM teachers in general.
Thus, results of inferential statistical methods should be treated with caution when considering their applicability to the entire population of STEM teachers.

Pre-and Post-Program Surveys
Prior studies have shown that self-efficacy in teachers is critical to both teacher and student  Similarly, when asked about the "Types of Learning" experienced during the WE 2 NG program, the 2017 cohort agreed more strongly (and in some cases, unanimously) that they increased their knowledge or understanding in various ways during the program (Figure 2). Even with slight differences in feedback from the 2017 and 2018 cohorts, participant feedback with respect to the "Types of Learning" they experienced during WE 2 NG was high overall (lowest median = 8, highest median = 10).
Post-program surveys evaluating the WE 2 NG "Level of Fit" and "Personal Impact" for each participant show relatively low variability in responses (most agreement between participants especially in the 2017 cohort). In both categories, five (out of five) was the lowest median response in 2017 (Figure 3).
The 2018 cohort also responded positively to these program elements, but with more variability in responses. Thus, participants mostly agreed that these areas of the program were successful. Taken together with "Types of Learning" (Figure 2), these results are important in addressing the first program objective (O1) because they expose personal responsiveness of participants to the WE 2 NG program.

Table 2. Range of Median Responses to Categories of Post-Survey Questions. Note:
all median responses shown here are normalized to a scale of 1:10 for consistency (some questions were originally ranked on a scale of 1:5).

Question Category Year Lowest Median Highest Median
General Satisfaction With overall high feedback in post-program surveys, implied (2017) and explicit (2018) gains in teacher's KAUP in STEM content, the authors feel confident that the WE 2 NG program is wellstructured to meet the needs of K-12 teachers with varying levels of STEM knowledge and abilities. These results suggest that the WE 2 NG program is a strong RET model with promise for other institutions who wish to promote increased STEM knowledge, perspective and enthusiasm in K-12 teachers. While the sample of teachers responding to these surveys represent a convenience sample of teachers-in-general, it is likely that other RET host-institutions would recruit similarly-representative samples of teachers, as we assume that teachers willing to dedicate their summer to STEM professional development tend to be highly motivated and opportunistic educators.

Figure 2. Sample Post-Program Survey Results Evaluating Types of Learning during the WE 2 NG Program. For a complete list of questions asked in each category, please see the Supplemental Information section.
Evaluation of an NSF Research Experience for Teachers (RET) Program for STEM Development

Weekly Surveys
Another metric of program evaluation took the form of weekly surveys; used to determine if certain elements of participant satisfaction change during the course of the program. Identifying which feedback elements change the most with time is valuable information to future program structuring, and similarly, knowing which types of feedback stay constant is also useful. At the end of each week of the program participants took relatively short surveys to assess their ongoing levels

ADVANCES IN ENGINEERING EDUCATION
Evaluation of an NSF Research Experience for Teachers (RET) Program for STEM Development of satisfaction with various program aspects, and to vet their responsiveness to program activities specific to that week. In this way, program coordinators were able to determine which activities were best-fit to participants and to see what program elements see the most development over time. Of particular interest here is distinguishing between elements of the program that show growth (or otherwise) with time, and those that are consistent throughout the program.
Eight questions (Q1-Q8) and median ranked responses are presented (Figure 4 and Table 3) in the order that they appeared in weekly surveys. Weekly trend analyses are notably different between with the greatest significance are, more often than not, related to group collaboration. The opposite is true for less significant trends (with the exception of Q6, which asks about mentor relationships),  in their teacher mentees and 2) how to cultivate positive and lasting relationships between the two. In 2014, Faber and others found that too much or too little research mentor involvement could negatively impact RET teacher learning outcomes (Faber et al.). These findings support those of Reimers and others (2015), who stress the importance of "building community" among professional development participants, including both teachers and scientists (Reimers, et. al.).
Tying weekly trends in with the first program objective, Q8 shows encouraging results from both years; evaluating teacher's perceptions of how their understanding of STEM was being strengthened due to their participation in the WE 2 NG program. In 2017, median teacher responses were 10 (out of 18 SUMMER 2020 VOLUME 8 ISSUE 2

ADVANCES IN ENGINEERING EDUCATION
Evaluation of an NSF Research Experience for Teachers (RET) Program for STEM Development participants in an environment where they were exposed to (and participated in) scientific research and other professional development activities, all through the lens of the water-energy nexus. The current study evaluated the effectiveness of the WE 2 NG RET model as a professional development tool through participant surveys taken before, during and after the 2017 and 2018 WE 2 NG programs.
The greatest variability in participant response is seen in pre-program surveys; relating to participant self-perception of STEM knowledge, understanding, ability and proficiency. In post-program surveys, where teachers evaluated various aspects of the WE 2 NG program, feedback was highly positive across-the-board. Thus, the authors believe that the WE 2 NG program is a highly effective STEM professional development model for educators of ranging abilities and K-12 grade levels, with the ability to increase STEM knowledge, perspective and enthusiasm in participants.
Key findings from our analysis include: 1) the WE 2 NG RET program can support growth in teacher confidence in participants of all grade levels, 2) program satisfaction outcomes are likely controlled by the e:m:h ratio of each cohort, 3) learning/growth outcomes apparently are not controlled by participant satisfaction with logistical program elements, and 4) the development of collaborative program elements tend to depend on program duration while non-collaborative program logistics do not. Based on results, we recommend that longer-term STEM professional development programs (such as WE 2 NG) allocate ample time for collaborative efforts among all program participants; especially near the beginning of the program.
Survey results are being utilized to inform planning efforts for future WE 2 NG iterations. Due to positive program feedback, the general structure of the WE 2 NG program will continue, and minor adjustments will be implemented to address program elements that received highest variability in post-program survey responses. Results suggest that future iterations of WE 2 NG should tailor activities more specifically to the e:m:h ratio of each cohort. To support teachers with the greatest potential for growth, future program advertisements will specifically target teachers lacking STEM exposure outside the classroom setting. With the knowledge that the development of collaborative program elements (especially intra-research group dynamics) depends on program duration, the structure of future WE 2 NG programs will continue to cultivate and strengthen collaboration by intentionally allocating time for collaborative meetings among teachers (vertical and horizontal alignment) and research mentors throughout each week of the program. Teachers likely need enough mentorship to begin working independently with periodic guidance, but not so much guidance that they never experience cognitive dissonance.
New questions have also arisen during the evaluation of the 2017 and 2018 WE 2 NG programs and call for modification of future survey questions. Most notably, WE 2 NG coordinators are interested in: 1) vetting program effectiveness for master teacher participants, 2) taking a more in-depth look at how the WE 2 NG program shapes-up as a tool for enhancing science communication skills and While these results are encouraging, the sample of teachers responding to surveys described in this work represent a small sample of teachers-in-general and should be interpreted with caution when considering the entire population of STEM teachers. However, it is assumed that teachers who are willing to spend the majority of their summer in an intensive professional development training (i.e. WE 2 NG participants) are highly motivated educators, and likely represent the population of teachers who seek intensive professional development. Data obtained in this study will inform the structuring of future WE 2 NG programs, and the authors hope that dissemination of these findings will serve other institutions who are currently hosting (or plan to host) an RET or other STEM professional development programs.

Ability and Proficiency (Pre-and Post-Surveys)
Please rate your confidence in the following aspects of your teaching, within the particular subject area(s) that you teach.
• Your ability to access experts in your field of science • Your ability to create presentations at teacher in-services or professional meetings • Your ability to network effectively with teachers and other professionals • Your ability to effectively teach and explain scientific ideas and concepts to your students • Your proficiency in using the Internet to access information useful for teaching

Program Level of Fit to each Participant (Post-Survey Only)
To what extent, if any, was your RET experience successful in each of the following areas: • It was responsive to your professional development needs

Types of Learning as a Result of the WE 2 NG Program (Post-Survey Only)
To what extent, if any, did you feel that you experienced each of the following types of learning as a result of your summer experience?