The Impact of Undergraduate Research and Student Characteristics on Student Success Metrics at an Urban, Minority Serving, Commuter, Public Institution

Challenges to establishing and maintaining undergraduate research programs include how to demonstrate impact as evidence for future funding, establish eligibility criteria when resources are limited, and assess new components. To address these challenges, undergraduate researcher GPA, credit accumulation and time to graduation were evaluated longitudinally, at an urban, public, minority and Hispanic serving, commuter college. Students who participated in undergraduate research and matched peers were also compared. Evaluation revealed that all groups benefited from participation in undergraduate research, whether they had fullor part-time status, were STEM or non-STEM majors, or participated in single or multiple semesters of research. Addition of mandatory workshops after the fourth year of the seven years of students evaluated, correlated with longer participation in voluntary undergraduate research. Longer participation correlated with higher GPAs. Entering freshmen and transfer students, who began research with no College GPA, were more likely to have low GPAs during the semester of participation, suggesting that a successful semester at the college before eligibility may be an evidence-based criteria to implement.


Introduction
A key factor to whether college students persist and thrive is the degree to which they participate in educationally effective activities that contribute to their learning, personal development and success (Kuh, 2001;Kuh, 2003;Lopatto, 2006). Undergraduate research is a recognized high impact pedagogical practice that enhances student development and results in increased retention and degree completion; it has been identified as particularly important to the academic success of underrepresented groups. Undergraduate research experiences have been demonstrated to support STEM-related career aspirations and increase STEM graduation rates (Gregerman et al, 1998;Davis, 2009;Espinosa, 2011;Hu et al, 2008;Johnson, 2011;Schultz et al, 2011;Ishiyana and Hopkins, 2005;Seymour et al, 2004;Laursen et al, 2010). Baron, Brown, Cumming, and Mengeling This paper will explore who chose to participate in undergraduate research through the Emerging Scholars (ES) program at a Hispanic and minority serving, urban, public commuter college (New York City College of Technology or "City Tech,"), their academic outcomes, and evaluate results for program optimization and demonstration of its impact on student success. This work adds to the body of knowledge on the impact of undergraduate research experiences on parttime students, a topic of little research, despite the fact that 37% of undergraduates are enrolled part-time, including 61% of students at community colleges (College Board, 2014). Given that 84% of Hispanic students and 81% of black students, versus 72% of white students, enroll part-time at least one semester, a recent study has shown that if part-time students from underrepresented groups graduated at the same rate as their white counterparts, the achievement gap between black and white students would close by 13 points and for Hispanic students, it would close 7 points (EAB, 2019). These authors concluded that because underrepresented minority and first-generation students are more likely to attend part-time, student success initiatives that only target full-time students will not close the achievement gap as effectively as those that focus on part-time students.
Additional topics reported include the impact of the timing of research in a student's academic career, the duration of the undergraduate research experience and the introduction of professional development workshops for students. This work thus serves both as a model for assessment of the impact of undergraduate research experiences as well as provides guiding evidence demonstrating impact and for developing eligibility criteria when resources are limited.

ES Program Components
City Tech is an open access college, offering associate and bachelor's degrees. We have been sponsoring the Emerging Scholars (ES) undergraduate research program since fall 2006. The goals of the program include: a. enhance the intellectual vitality of the college by providing students with the opportunity to apply what they learned in the classroom to discover new knowledge and solve problems, b. promote student academic success through the opportunity to engage with faculty, and c. provide faculty with an "extra pair of hands" to advance their scholarship, in lieu of graduate students.
The Emerging Scholars program is open to all students in good academic standing (minimum GPA 2.0) and provides full-time students (enrolled for 12 or more credits) with $500 stipends for working with faculty on their scholarly activities, approximately 50 hours each semester and part-time students (enrolled for 1-11 credits) with $250 stipends for working approximately 25 hours each semester. All students are required to complete CITI certification in Responsible Conduct in Research within the first month. While the program was initially envisioned as supporting promising students that faculty selected, a growing number of students now approach faculty to become involved.
To Introductory workshop goals include developing the most common research and communication skills needed in research, which reduces this responsibility for faculty mentors, so that they can focus on the research project and preparing students for external research opportunities. Other investigators have noted that faculty mentors have reported students' lack of academic writing experience as an obstacle to engaging students in undergraduate research (Myers, 2018). Advanced workshop goals are to further develop communication skills, and help students prepare for next steps -internships, employment and graduate school.
In the data used for this paper, 214 students (44.7%) participated in the Emerging Scholars progam prior to the addition of the workshop component and 265 students (55.3%) participated for the first time with the workshop component.
This work was part of a long-term plan to better understand and improve our undergraduate research program. Subsequent to this study, the Undergraduate Research Committee also created a mentoring handbook (Mentoring Handbook, 2018) to support faculty mentors, and a Mentor Brochure (Undergraduate Research Mentor Brochure, 2019), highlighting the research interests of faculty, to help students identify and connect with a mentor.

Methodology
Data for all students participating in the Emerging Scholars program from its introduction in the fall of 2006 through spring of 2013 were used in this study (n=479). Data include background variables (e.g., race/ethnicity, sex/gender, and age), pre-participation academic variables (e.g., major, cumulative GPA, cumulative credits earned), and post participation variables (e.g., enrollment, semester GPA, semester credits, and graduation). Post participation data on enrollment, GPA, credits earned, and graduation were available through spring of 2015.
For analyses involving semester GPA and semester credits, mixed model for repeated measures (MMRM) analysis was used to look for overall differences as well as differential change over time. For graduation and persistence, a combination of statistical methods was used. Rates at particular points in time (e.g., one year after participation) were compared using chi-square tests. Survival analysis, specifically discrete-time hazard modeling (Singer & Willett, 2003) was used to analyze longitudinal data. Implications for possible eligibility criteria were explored using chi-square tests and t-tests.
Each student participating in the ES program was matched with two nonparticipating peers based on similarity in terms of degree-level being pursued (associate or bachelor's), major, admission date, cumulative credits and cumulative GPA at the start of the first ES semester, as well as HS GPA (as measured by the College Admissions Average), age, gender and ethnicity. It was often not possible to to find students who matched on all of the criteria simultaneously. Top priority was given to degree-level being pursued and major. Then the closest two matches were chosen based on a weighted average of the other criterion variables. By design, the two groups were identical on enrollment at the college the semester of the ES participant's first research experience (the matching semester), degree being pursued (associate, bachelor's, or nondegree), and academic major. A comparison of the ES participant group and the matched controls found no statistically significant differences on sociodemographic or academic characteristics (Tables 1 and 2).  Baron, Brown, Cumming, and Mengeling A mixed model for repeated measures (MMRM) was used to assess differences in semester GPA over time for ES participants and their matched peers. Variables in the model included 'group', which identified ES participants (Group=1) and matched non-participant peers (Group=0); time measured in semesters with 0 being the semester prior to ES participation (for the participant group as well as the matched non-participant peers); baseline GPA, which is the cumulative GPA prior to first semester of ES participation; and status (full-time or part-time). In addition, the interactions between time and group, and time and status were included in the model. Of specific interest were: 1) the effect for group which would indicate a difference in semester GPA for participants and nonparticipant peers during the ES semester; and 2) the interaction between time and group which would indicate different growth trajectories for participants and non-participant peers after the first ES participation semester. The results indicated that none of the effects relating to status (full-time or part-time) were significant (p-value range = .70-.93), and the model was rerun without status. This finding suggests that benefits associated with participation where similar for both part-time and fulltime participants. Table 3 compares the representation in the ES participant group to the overall enrollment at our College. A chi-square test indicated that Hispanic students were under-represented in the ES program whereas white students were over-represented (p<.0001).

ES Participants
Overall Enrollment + Students may participate in the ES program at any time during their undergraduate studies. The most common year to participate for the first time was as a sophomore. Documentation for the ES progam indicates that preference will be given to full-time students (Undergraduate Research, 2019) and this is reflected in the breakdown of the ES participants with only 19% being part-time students, which is considerably lower than the average percent of students in the college population who were part-time (36%). The breakdown of students according to degree goal was quite similar to the general college population (38% of participants were pursuing a bachelor's degree compared to 36.5% of students ihe overall population). These results are summarized in Table 4. The average cumulative GPA prior to ES particiption was 3.33 (n = 450, SD =.50). Not all 479 ES participants had a GPA prior to participation. Twenty-nine students (6%) participated in ES their first semester at our college and did not have a City Tech GPA involving coursework taken prior to participation in the research experience. Of these, the majority were first time freshmen (n=16; 55%). The rest were transfer students (n=9; 31%), non-degree students (n=3; 10%), and a readmission student (n=1, 3%).
Journal of the Scholarship of Teaching and Learning, Vol. 20, No. 1, April 2020. josotl.indiana.edu

Evaluation for Establishing Eligibility Criteria
Baseline cumulative GPA and baseline cumulative credits earned were examined as predictors of improvement in semester GPA and semester credits earned during and after ES participation. Students with low semester GPAs (Semester GPA < 2.0) or no semester GPA during ES participation (n=28) were more likely to come into the research semester with no cumulative City Tech GPA on record (14.3% of low/no semester GPA students had no cumulative GPA as compared to 5.5% of students with a semester GPA of 2.0 or higher, p=.04). Students with low/no semester GPA during participation who did have a cumulative GPA tended to have a lower cumulative GPA than other ES students prior to participation (2.86 for low/no semester GPA as compared to 3.35 for students with a semester GPA of 2.0 or higher, p<.0001). In addition, these students were younger on average than other students (21.1 years old for low/no semester GPA during participation, as compared to 23.9 years old for students who earned a semester GPA of 2.0 or higher, p=.03) during participation.
In constrast to the low/no semester GPA results, students who earned a lower number of semester credits ( ≤ 6) during ES participation tended to be older than other ES participants (26.6 years old compared to 23.4 years old, respectively, p=.009), have similar cumulative GPAs (3.29 as compared to 3.33, respectively, p=.63), and have higher cumulative credits earned (62.8 credits compared to 49.7 credits, respectively, p=.03), and be part-time.
To summarize, subpar semester GPA during ES participation was associated with younger students and lower or no City Tech cumulative GPA prior to participation. Thus if a goal is to develop additional eligibility requirements to promote positive outcomes for ES participants, requiring at least one semester at the college prior to participation in addition to the current minimum GPA of 2.0 is a possibility. The students with few semester credits earned during participation have similar GPAs and tend to be older. Thus taking fewer credits may just be a sign of stage of life rather than a negative outcome.

Evaluation of the Impact of the Introduction of Student Professional Development Workshops
Semester GPA and semester credits earned were compared for students first involved in the ES program prior to the workshop introduction (semesters prior to Fall of 2010) and after the addition of the workshops (semesters starting in Fall of 2010). Semester GPAs were very similar for students prior to and after the addition of the workshop component (p=.72; see Figure 1). However, as can be seen in Figure 2, semester credits earned differed for the two groups (p=.009). Although the graph hints at an interaction where non-workshop students had a steeper fall off in credits passed after ES participation, the change was not large enough to be statistically significant (p=.37). Thus it appears there may be additional factors not captured by the current data resulting in students in the more recent years passing additional credits prior to, during, and after ES participation. This may be related to the workshop component in terms of the students who chose to participate under the altered ES program, but the data does not support an inference that students are differentially affected during and after participation.  participated in additional research semesters). More students participated in multiple semesters when their first experience involved the workshop than when it did not (32.8% and 23.8%, respectively; p=.045). Evaluation of other possible positive outcomes, such as developing a professional identify, were beyond the scope of this work

Evaluation of the Impact of the Length of Participation
As noted earlier, students are not limited to a single semester of partipation in the ES program and although most students participated only a single time (n=365), 24% (n=114) participated multiple semesters. Students who participated in the ES program once and students who participated multiple semesters were compared on semester GPA, semester credits earned, persistence, and graduation.

Semester GPA
The results indicated that none of the effects relating to status (full-time or part-time) were significant (p-value range = .75-.92), and the model was rerun without status. Neither multiple nor the multiple by time interactions were significant although multiple came close to statistical significance (p=.07 for multiple; p=.69 for the interaction). As can be seen in Figure 3 there was a relatively stable advantage on GPA for multiple semester participants. However, the difference did not exceed what could be explained by differences in prior GPA. "

Semester Credits Earned
There were significant differences in the relationship between multiple semester participation and time depending on full-time or part-time status (p<.0001). For this reason the analysis was rerun separately for full-and part-time students. For full-time students, there was a tendency for multiple semester participants to pass more credits per semester than single-semester ES participants (p=.0008). Although the difference between single semester participants and multiple-semester participants widens over time, the difference in the degree of decline for the two groups was not large enough to be statistically significant (p=.23). See Figure 4 for the trend in average semester credits earned for the two groups.

Figure 4. Semester Credits Earned for Full-Time Single-and Multiple-Semester Research Participants
For part-time students, multiple semester participants were not statistically different from single-semester participants in either average credits earned (p=.45) or in change over time in credits earned (p=.56). Thus for part-time students the determinants of semesters passed seem unrelated to whether or not the student participates in ES for a single semester or multiple semesters (see Figure  5). Thus, if resources are limited, limiting part-time students to a single semester of research, optimally promotes student success in terms of credits earned.

Graduation
For this analysis only graduation involving the degree being pursued at the time of first participation in ES (or a higher degree) is counted as graduating. Also, spring 2013 participants were omitted from the analysis since whether or not they participated the following semester is unknown. Survival analysis was used to fit a discrete time hazard model to the data for single and multiple semester ES participants. Survival analysis indicated no significant difference between single semester participants and multiple semester participants in terms of time to graduate (p=.55). However, given that a student who is graduating soon has less opportunity for multiple semesters of participation, this analysis is difficult to interpret. Overall, the graduation rate for the single semester participants (49.3%; n = 144) was significantly lower than the graduation rate for the multiple semester participants (60.5%; n = 69; p=.042) (see Figure 6). When these results were broken out by degree goal, the significance appeared to be driven by students pursuing associate degrees. The overall graduation rate for single semester ES participants pursuing an associate degree was 43.9% whereas the multiple semester students pursuing an associate degree had a graduation rate of 57.5%, p=.047). For bachelor's degree ES participants 60.6% of single semester particpants graduated whereas 65.9% of multiple semester participants graduated (p=.57).

Persistence
For students who do not graduate, it was of interest to look for difference in persistence between single-and multiple-semester participants. Multiple semester participants have higher persistence rates (see Table 5), but interpreting the meaning of the result is complicated. The only way for a student to participate in multiple semesters is if the students is enrolled at our college in multiple semesters, thus the persistence rate for students participating in multiple semesters of research is very high especially at 1 semester after the first ES participation. On the other hand, a single semester participant may not participate in research again because s/he does not wish to, or because they are not enrolled and thus don't have the opportunity. Whether persistence is responsible for multiple ES participation, or multiple ES participation is responsible for persistence, is unclear.

Comparison of Undergraduate Researchers and a Matched Comparison Group
Both group and the group by time interaction were significant (p=.03 and p<.0001, respectively). ES participants have higher GPAs during the ES semester (effect= .18 on the GPA scale), however over time the difference between the semester GPA for participants and non-participants narrows until at 4 semesters after participation the groups are not statistically different (see Figure 7). It is important to note the groups are matched on prior GPA and furthermore prior GPA was included in the model as a control variable. Therefore, participation is associated with higher GPAs in the semester of ES participation and in the several semesters after participation, and this effect is not explained by pre-existing GPA differences. It is also important to note that the population represented in the analysis changes with time. All students in the ES participant group and the matched controls are present at time 0, however as time increases student are lost to the analysis through graduation or non-enrollment. Thus, the farther out on the time axis, the lower the sample sizes and the less the sample represents the original time 0 sample. Thus, the results out past 3 or 4 semesters should be interpreted with caution.

Credits Earned
The same analysis was conducted comparing semester credits earned for ES participants and their matched peers. In this analysis, whether or not a student was full-time had a significant effect on the relationship between time and group (Group x Time Interaction p<.0001). For this reason, the analysis was repeated separating the full-time and part-time students. For full-time students, group was significant (p=.05) but not the group by time interaction. On average, full-time ES participants earned more credits per semester than their matched non-ES peers (effect = 0.44 credits). Examining Figure 8, the difference between groups appears large in the ES semester and the semester after ES participation, however the fluctuation in the difference from one semester to another was not large enough to rule out it being due to random chance (p=.22). Once again, it is important to note the groups were matched on prior semester credits earned, and prior credits earned is included in the model as a control variable. Therefore, for full-time students, participation is associated with higher credits earned that is not due to pre-existing group differences. Again, as time increases the samples for both groups get smaller and we can have less confidence in the results being representative of the original groups.
For part-time students, group was not significant (p=.91) but there was a group by time interaction (p=.023). In the semester before, during, and after ES participation, the ES participants earned more credits than their matched peers (see Figure 9). However, at two, three and five semesters after ES participation there were not significant differences between the groups (p=.92, . 998, and .40, respectively), and at four semesters after participation, the effect was reversed (p=.02). Once again, it is important to note the groups were matched on prior semester credits earned and prior credits earned is included in the model as a control variable. Therefore, for part-time students, participation is associated with higher credits earned in the semesters near ES participation that is not due to preexisting group differences in credit accumulation.

Graduation
Each semester after the first participation in the ES program, students are classified as 1) continuing; 2) graduated, or 3) not enrolled (and not graduated). Each consecutive semester the student continues to take classes is counted and used to measure persistence. The number of semesters until graduation are counted (including any semesters in which the student took no classes) to measure time until graduation. Graduation is measured in two ways. The first method counts only graduation involving the degree being pursued at the time of first participation in ES (or a higher degree). Thus, a student pursuing a bachelor's degree who obtains an associate degree is not counted as graduated unless or until the student earns the bachelor's degree. However, a student pursuing an associate degree who earns a bachelor's degree is considered as graduating since a bachelor's degree is a higher-level degree than an associate degree. The second way of measuring graduation counts any degree as successful graduation. The results of both methods will be summarized.

Method 1 (Graduation with degree being pursued)
Overall, the graduation rate for the ES participants (50.9%; n = 244) was significantly higher than the graduation rate for the matched sample (44.8%; n = 429; p<.0001). When examined separately for associate degree students and bachelor's degree students, both groups were more likely to graduate if they participated in ES research (Odds-Ratio (OR) for associate degree students 1.24, p = .002; OR for bachelor's degree students 1.25, p=.01). See Figure 10.

Method 2 (Graduation with any degree)
Of the ES participants, 22 (4.6%) received an associate degree while working on a bachelor's degree. Of these students, 12 (55%) had not received a bachelor's degree as of spring 2015. Thus, the analysis counting all degrees as successful graduation has a slightly higher graduation rate for ES participants (53.4% (n=256) compared to 50.9%). Similar results were found for the matched control sample -41 students (4.3%) received an associate degree while pursuing a bachelor's degree, and of these, 18 (43.9%) had not received a bachelor's degree as of spring 2015. The overall graduation rate for the matched sample went up to 46.6% (n=446) from 43.1% when all degrees were counted as successful outcomes. In summary, both graduation rates went up slightly when intermediate degrees are counted, but the gap between the groups remained (p=.016).

Figure 10. Graduation rate for associate and bachelor's students -ES participants and matched controls.
Persistence When comparing persistence rates for the ES participants and the matched controls, differences were not statistically significant at any of the time periods examined (1 semester after ES semester (p=.80), 1 year after ES semester (p=.46), and 2 years after ES semester (p=.66)) (See Table 6 for details). Note, persistence rates are calculated without counting graduates as part of the numerator or denominator.
Is there evidence to guide eligibility criteria?
It appears that there may be benefits for students participating in undergraduate research based on our data demonstrating that ES students have higher semester GPA, earn more credits per semester, and graduate at higher rates than their matched non-participant peers. It was of interest to explore whether these potential benefits are larger for some groups of students than for others. This analysis differs from the earlier analysis due to the comparison with the matched sample. Thus, we are able to discern if the outcomes for ES participants are more positive than the outcomes for their matched peers for specific subsets of students like STEM majors and students earlier in their college education. The MMRM analysis conducted above was repeated with STEM classification added to the model. The interaction between STEM classification and group (ES participants or matched peers) was not significant, indicating that there was not a statistically reliable differential benefit for ES participation for STEM students compared to non-STEM students (p=.30 for semester GPA; p=. 68 for semester credits).
Recall, that based on the earlier ES analysis, no City Tech cumulative GPA coming into the ES participation semester (ie freshmen and transfer students) was related to poorer outcomes than ES participants with a City Tech GPA. MMRM was used to test for differences in outcomes for these students (n=29) and their matched peers (n=58). In terms of semester GPA, ES participants had significantly higher GPA during the ES participation semester (p=.015) and in the two semesters following participation (p=.0003; p=.014). In subsequent semesters, the GPAs were not significantly different (p-value range = .07-.98).
The results for semester credits were similar. No prior-GPA ES participants earned significantly more credits during the ES participation semester (p<.0001) and in the two semesters following participation (p<.0001; p=.026) than their matched peers (see Table 7). In subsequent semesters, credits earned were not significantly different (p-value range = .22-.87) for the two groups. Graduation rates for the no prior GPA ES participants and their matched peers in this at-risk subgroup were low (16.7% for ES participants and 13.3% for matched peers). However, these statistics include the 25% of students from this group who were still enrolled in the spring of 2015 and thus we do not know their ultimate graduation status.   Baron, Brown, Cumming, and Mengeling Persistence rates for the two groups were not significantly different although there was a trend for the no prior GPA matched peers to have higher persistence rates at 1 year and 2 years after the ES participation semester (see Table 8).

Table 8. Persistence No Prior GPA Emerging Scholars Participants and their Matched Case Controls -Percent and Frequency
Note: Graduates do not count in the numerator or denominator when calculating persistence rate.

Conclusions
This work presents a model for evaluation of the impact of an undergraduate research program that yields evidence of value and eligibility criteria and the impact of the introduction of a new component, professional development workshops. Evaluation of this data suggested important directions: 1. While African Americans participate at levels close to their representation in the student population (30.1% and 32.0%, respectively), bucking the trend that underrepresented minority students are less likely to participate than other groups (Finley and McNair, 2013), the underrepresentation of Hispanic students suggests that more intentional recruitment is needed.
2. Subpar semester GPA during undergraduate research participation (GPA< 2.00) was associated with younger students and lower prior cumulative GPA's or no City Tech cumulative GPA. However, when compared to their matched peers the students participating in research had significantly higher GPAs during the participation semester and in the two semesters following participation. The results for semester credits earned were similar. No prior-GPA research participants earned significantly more credits during the participation semester and in the two semesters following participation than their matched peers. Thus, if a goal is to conserve resources for those most likely to benefit, requiring at least one semester at the college prior to participation would be an evidence-based criteria. If resources are available even students with low/no GPAs are likely to benefit.
3. The addition of student professional development workshops correlated with multiple semester participation in the ES program, although it did not have a statistically significant impact on GPA or credit accumulation.
4. For full-time students, there was a tendency for multiple semester participants to earn more credits per semester than single-semester researchers. For part-time students, multiple semester participants were not statistically different from single-semester participants in either average credits earned in the research semester or over time. Thus, while there is no supporting evidence that full-time participation should be limited to one semester to conserve resources, no evidence of the benefits of multiple semester benefits for part-time students was found. Thus if resources are limited, limiting part-time students to one semester of participation may be a reasonable approach. 5. For full-time students, researchers earned more credits during and one semester after participation than their matched peers. For part-time students, ES participation was associated with higher credits earned in the semesters near ES participation than their matched peers, which was not due to preexisting group differences in credit accumulation. This points to the value of offering undergraduate research opportunities to part-time students.
6. The graduation rate for undergraduate research participants was significantly higher than that of the matched sample. This is evidence for sustaining an undergraduate research program.
7. The academic outcomes were similar for students majoring in STEM compared to non-STEM majors. This suggests that the benefits of ES participation are not isolated to a specific area of study but rather provide positive experiences for the larger undergraduate community, inclusive of a variety of major areas of study. This finding is in agreement with other studies (Healy and Jenkins, 2009;Ishiyama, 2002). Previous work has also shown that there are fewer opportunities for undergraduate research in the social sciences and humanities than in the natural sciences (Seymour et al, 2004).

Limitations
The primary limitation of this study is the observational nature of the data. Because students choose to be in the ES program and, in fact, put forth effort to get into the program, there are most likely substantial differences in the personal characteristics of ES participants and non-participants. A matched sample was drawn in an attempt to control as many of these personal characteristics as possible, but it is important to keep in mind that this is not a complete solution to the problem. There are likely unmeasured personal factors that are still influencing the results for the two groups and it is important not to over-interpret the results. This is true of any observational study, even though it is sometimes not acknowledged in research reports. Thus, the results of this study are suggestive of factors that contribute to student success, but they do not prove that it is the ES program responsible for higher success rates.