What Factors Influence Pedagogical Methods in Informal Learni ng Spaces?

Public engagement is an important component of the operation of many p hysics organizations, such as physics departments. However, unlike in formal education settings, relatively little r es arch has been done on pedagogical approaches in informal physics education; commensurately little res earch has been done on the methods and means of training the facilitators who engage in that pedagogy. Following up on a previous study that characterized modes of pedagogy in informal physics educators, we presen t here preliminary results from a detailed study on the ways in which volunteers in a physics outreach program at the Univ rsity of Colorado understand and operationalize these pedagogical modes, and the ways in which our v olunteer training methods promote or inhibit volunteers’ efficient uptake and utilization of these teaching tools. We find that volunteers’ conceptions of these pedagogical modes tends not to persist robustly from their pre paration at the start of program participation, and that other factors, such as practiced pedagogical behaviors gleaned from formal learning environments, may play a large role in UEs’ enacted pedagogies in PISEC. From these fin dings, we begin to form the basis for changes to the ways in which we support and prepare our volunteers, w ith implications for informal programs at large.


I. INTRODUCTION
The physics community has increasingly recognized public engagement, commonly called "outreach," as an important means of increasing awareness [1], building interest and identity in physics [2], developing skills relevant to the physics enterprise [3,4], and both improving concrete skills and broadening perspectives of the outreach facilitators themselves [5][6][7][8]. Indeed, the National Science Foundation (NSF) has articulated in its strategic plan the importance of efforts to broaden participation in physics, especially ". . . from underrepresented groups and diverse institutions across all geographical regions in all NSF activities. . . " [9]. Given the value of public engagement, it is important for organizations supporting outreach programs, such as university physics departments, to implement robust preparation of the facilitators who lead or participate in these programs.
However, while ample time and resources have been devoted toward studying and characterizing methods of pedagogy in formal settings as well as toward development of techniques to improve instructors' ability to meet the goals of formal teaching [10,11], comparatively little work has been done to develop a rigorous framework for teaching and teacher training in the informal environments -after-school programs, museums, traveling physics shows, etc. -that comprise the teaching spaces for public engagement. For example, an "Outreach Guide" published by APS provides programmatic insight and logistical information on topics such as working with schoolchildren; however, no information on effective pedagogy techniques is included alongside the logistical guidelines [12].
One step toward such a rigorous framework was taken in 2016 with the publication of a study by Hinko et al. observing multiple "modes of pedagogy" enacted by teachers in the Partnerships for Informal Science Education in the Community (PISEC) program, an informal physics education program sponsored by the University of Colorado Boulder (CU Boulder) [13]. Observing the pedagogy of PISEC volunteers, known as "University Educators" (UEs) over the course of a semester, the study characterized three pedagogical modes -instruction, consultation, and participation -according to their traits, techniques, and apparent objectives for engagement. To briefly describe the modes: instruction mode is characterized by more directive pedagogy, framing the UE as a leader; consultation mode by more facilitative pedagogy, framing the UE as an advisor; and participation mode by more cooperative pedagogy, framing the UE as an equal partner to the students. As a closing exhortation, the authors suggested teaching UEs about the pedagogical modes during PISEC training so as to better inform and equip them for utilizing the modes as teaching tools.
To build upon this prior work, we have performed an indepth study of the ways in which UEs understand and operationalize the pedagogical modes when exposed to them as part of PISEC training. In the present work we will share a few examples and key themes from our preliminary findings and discuss implications for PISEC training, toward improvement of the PISEC training model and, potentially, informal STEM learning programs at large.

II. METHODOLOGY
The study encompassing the present work employed both qualitative and quantitative techniques to investigate UE pedagogy in PISEC. Because UEs participate in PISEC for an entire semester, including training, we were able to engage both in broad, quantitative surveying to determine facets of UEs' stances on pedagogy and in targeted qualitative study over the semester with a small, select number of UEs. Here we detail briefly the facets of PISEC's design and implementation and the study techniques employed to produce the findings discussed in the present work.

A. Study Contenxt: The Partnerships for Informal Science Education in the Community and Program Training
PISEC [14] is jointly supported by CU Boulder and the JILA NSF Physics Frontier Center [15], and PISEC UEs are recruited primarily from among CU's undergraduate, graduate, and postdoctoral STEM population. The program is based upon the Fifth Dimension (5D) model [16] and prioritizes long-term educational partnerships with local community sites, blending cultures and leveraging the diversity of participants and facilitators toward a mutually beneficial end. In contrast to more "traditional" content-focused goals, PISEC works to provide engaging opportunities for students to develop identity and affect in physics alongside their UE mentors at ongoing site visits throughout the semester.
At the start of each semester of PISEC, UEs attend a oneday training session that covers conceptual and theoretical facets of PISEC teaching, such as inquiry-based education and the zone of proximal development (important in the 5D model but originally conceptualized by Vygotsky [17]) as well as concrete components of the UE experience such as curriculum practice and pedagogical techniques. Following the recommendations in Ref. [13], PISEC added a module in Fall 2016 on the modes of pedagogy. UEs are explicitly exposed to the modes through a short presentation, then given the opportunity to practice the modes, taking turns as "mentor" and "student" in roleplaying modes using real PISEC activities.
UEs are surveyed both before and after training on their attitudes and beliefs about public engagement/outreach, their personal teaching experience and philosophy, components of training that they felt useful or not useful, etc. As described in Ref. [18], these surveys currently incorporate questions designed to either infer or explicitly determine UEs' preferences between the pedagogical modes. In that work we reported that PISEC training produces a significant shift in UEs' explicitly-stated pedagogical preferences toward both more participatory pedagogies (greater preference for participation mode) and more nuanced pedagogies (preferences toward multiple modes) [18]. This study extends prior work to connect these professed beliefs to UEs' enacted pedagogies and investigates how UE behaviors align or not with those beliefs and which factors might influence UEs' pedagogical choices during PISEC site visits.

B. Study Methods
In order to obtain a comprehensive picture of the ways in which UEs conceptualize and implement the modes of pedagogy, we employed three main techniques: surveys, in-situ video, and semi-structured interviews. As described in Section II A, we surveyed UEs for their pedagogical preferences both before and after training. We also surveyed UEs at the end of the PISEC semester, again asking questions about their attitudes and beliefs toward pedagogy, including about the modes. This trio of surveys was used to provide a quantitative understanding of both the effect of training on UEs' pedagogical preferences and the persistence of those effects over the course of the PISEC semester.
To address the question of how UEs actually teach in PISEC following training, we also utilized in-situ video of PISEC site visits, recording UE interactions with PISEC students throughout the course of the semester. At each hourlong site visit, rotating sets of groups of UEs and their students were selected and recorded such that we obtained 3-4 hours of video for each UE over the semester. To analyze this video, we scanned weekly recordings for clearly-defined "rich" moments of pedagogy, moments where UEs engaged in pedagogy with students over a sustained period of time. For each of these moments, we followed the methodology of Ref. [13] and analyzed the UEs' pedagogy using culturalhistorical activity theory (CHAT) [19? ], grouping interactions by their CHAT elements to assign them to one or more pedagogical modes. We utilized CHAT in part because of its utility in situations, like those common in PISEC, where an agent's behavior is mediated both through artifacts and cultural contexts (e.g., a UE handling experimental materials alongside students).
Third, to obtain a more nuanced understanding of the factors influencing UEs' pedagogical preferences and teaching methods, we conducted interviews with multiple UEs at the end of the PISEC semester. These semi-structured interviews followed a protocol that included questions about UEs' attitudes toward the PISEC program as a whole, guided reflection on their survey answers and their beliefs about pedagogy in PISEC, and stimulated recall involving the UEs watching clips of their own pedagogy and reflecting on their thoughts and actions during the moment of teaching, following the methdology of Spike et al. [20].
As described above, our study utilized both quantitative methods (surveys) and qualitative (in-situ video, interviews). For the targeted qualitative component, we selected six UEs who had attended training and had demonstrated a shift in modal preference between their pre-and post-training surveys. We recorded early-, mid-, and late-semester video for those six UEs and performed CHAT analysis on their pedagogy. From among those six UEs, we interviewed three, utilizing the protocol described here. Comprehensive analysis of all six UEs will be published in a subsequent work; in the present work we discuss a few salient preliminary findings drawn largely from case analysis of a UE, "Moira," who taught with PISEC in the Fall 2018 semester, occasionally drawing from analysis of other UEs as well. We discuss here these findings and their implications for PISEC training, opening a discussion about how they can be used as a starting point to construct a foundation for a new paradigm in informal teacher training.

A. UEs' Understanding of the Modes is Not Robust
One important finding from Ref. [13] was the lack of observation of "a 'hierarchy' of modes that are either 'good' or 'bad' at achieving [PISEC's goals]" -instead, Hinko et al. reported that each mode had "both affordances and limitations. . . " In agreement with this conclusion, we explicitly encouraged UEs during training to think of the modes as situational tools for teaching with benefits and drawbacks. While UEs appeared to agree with this emphasis during training, the majority of their post-training survey answers expressed a preference for a single mode, either consultation or participation. Their enacted pedagogies as well tended, in line with the observations in Ref. [13], to reflect a large preference for the consultation mode. While we would not necessarily expect our emphasis on modal "equality" to result in strictly equal modal proportions in enacted UE pedagogy, we also observed evidence that, despite our efforts, UEs may put these modalities in a hierarchy, and that current training methods may inadvertently reinforce this hierarchical understanding.
In her interview, Moira described how she operationalized the information provided on PISEC's program goals to inform her hierarchical understanding of the modes. When asked the extent to which training affected Moira's ability to teach in her preferred mode, she said, "I would say it has pretty significant impact . . . And when you presented [the modes], you didn't present them in a hierarchy, but I put them in one in my head [laughs]. 'Participation is ideal, followed by consultation, followed by instruction. '  As described here, Moira's understanding of PISEC's toplevel affect-and identity-related goals appear to have "over-ridden" our explicit message in training that all three modes could be useful situationally for achieving those goals (as was observed both in this study and in Ref. [13]). Interestingly, this answer was given a few minutes after Moira, reflecting on her answer on the post-training pedagogical preference survey, had affirmed the value of all three modes, instruction, consultation, and participation, saying "I found, during the site visits with the kids, I'm kind of swaying between all three . . . I really would swing back and forth because sometimes the students totally take over with some of the activities. It was totally self-directed to just have them mess around with things. So I would say that was participation. . . . Consultation, I would say . . . best when students need only guidance and clarification . . . and then sometimes instruction -I think especially when we were setting up activities that they were unfamiliar with, like the laser game or something like that." As with almost all of the UEs we studied, Moira was observed to teach about two-thirds of the time in consultation mode -in disagreement with her stated preferences. We did, however, observe Moira teaching in both of the other two modes, as she mentions above. The "laser game" referenced in Moira's quote is a board game where players assemble mirror-holding pieces on a game board, then attempt to use the placement of their pieces to cause a laser to strike one of their opponents' pieces. We observed Moira effectively implementing a more authoritative instruction mode with her students, assembling the board and reading the game rules herself, then "translating" her understanding for her students. We also observed this more directive style of pedagogy during this activity with other UEs as well. Multiple groups split the activity over multiple weeks because they ran out of time at a single site visit; this decision to "streamline" the activity via instruction mode may be due to this combination of activity complexity and PISEC's programmatic constraints.
The apparent discrepancy between the mode or modes prioritized in Moira's "hierarchy" (participation), her survey results (equal preferences), and her teaching (consultation), may demonstrate the impact of emergent, real-time constraints in PISEC's programmatic structure (e.g., realizing the site visit is almost out of time) as well as imply that her conception of the modes is not robust across different contexts and throughout the PISEC semester. In fact, Moira hinted at the latter immediately prior to her first response above, saying "I think I broadly remembered what the different [modes] were, but I sort of just fell into a groove and then kind of stuck with it after a while. So I would say, yeah, I was thinking about it maybe 30 percent of the time [laughs]." Other UEs also exhibited an "inaccurate" reckoning of the pedagogical modes and their engagement with them; one UE, "Winston," reflected on his post-semester pedagogical preference survey "I found that participation was the most useful . . . I could . . . nudge them in the right direction without being the forceful authority." However, Winston was observed to teach in participation mode only twice throughout the entire semester -approximately 10% of his observed pedagogy. Additionally, the description Winston gave of his pedagogy more nearly aligns with the consultation mode, not participation. Not only does Winston's apparent understanding of the modes not agree with our definitions in training, but his self-assessment using his recollection of the modes does not match his actual pedagogy. Clearly, current pedagogical modes training in PISEC is insufficient to convey an adequate, accurate, and persistent understanding of the pedagogical modes.

B. UEs' Expectations of PISEC May Shape Their Pedagogy
One of the largest questions stemming from our observations of PISEC UEs' enacted pedagogies is: what factors influence their preferences between the modes reported in Ref. [13]? The authors of that work suggested as one potential factor the stability or volatility of group dynamics, since PISEC UEs tend to work with the same group of students throughout the semester, only switching when logistical factors such an absent UE require it. Our video analysis suggests this and other emergent conditions and situational factors, as noted above, may indeed affect pedagogical preference. Interviews with the UEs, however, indicate that their a priori expectations of PISEC, perhaps shaped by their previous teaching experience, may in turn shape their pedagogical activity in PISEC.
In her interview, Moira described how, in addition to participating in PISEC, she was also serving as a Learning Assistant (LA) [21] and enrolled in a pedagogy course at CU. Throughout her interview, Moira brought up her experiences in the pedagogy course and its effect on her beliefs about PISEC. Particularly demonstrative of this influence is Moira's answer to the pre-training pedagogical preference survey question, "What is your philosophy/style of teaching?" Moira answered, "Encourage interest in the subject, foster a growth mindset, focus on what a student does right rather than wrong even if their overall approach is not correct so that they at least feel they have a foundation for the appropriate methods." When asked to reflect on this answer in her interview, Moira affirmed "Okay. Yeah, this is very clearly from the pedagogy class I was taking at the time." She also described how her experiences in pedagogy course blended with her PISEC training to inform her teaching: Besides Moira, other UEs referenced LA or TA training in their survey responses and/or interviews, either explicitly through mentioning their training, or implicitly, through describing techniques, such as the Socratic method, that are known to be employed at CU in those environments. This in itself is not necessarily surprising, since many CU undergraduates and grads participate in these formal settings and the cross-section of PISEC UEs who also serve as TAs or LAs is high. However, as embodied in Moira's response above, such experiences do not happen in a vacuum; they likely interface with and impact UEs' conceptualization of PISEC teaching structures, including the modes, as well as their expectations for how those structures are implemented in PISEC teaching.
In addition to the survey responses mentioned above, the interview of one UE, "Jesse," may support the idea that UEs' prior expectations of PISEC influence their teaching. Jesse's post-training pedagogical preference survey response explicitly indicated a preference for consultation mode. When asked about his expectations of PISEC, Jesse described how the informal, playful nature of PISEC was at odds with what he anticipated, saying "I don't know why, [ Whether PISEC UEs have explicit LA/TA training or not, the majority of their educational experiences are in these formal settings, and those experiences may lead to expectations that shape UEs' conceptualization and implementation of PISEC teaching methods. Jesse's inclination toward a more facilitative, consultation-like technique (as opposed to a more playful, participation-like technique) and his apparent surprise at the informal nature of PISEC may be reflective of this formal influence.

IV. DISCUSSION AND IMPLICATIONS FOR TRAINING
The above findings represent only a few salient examples from the analysis of UE pedagogy in PISEC but already provide several implications for UE training in PISEC and, potentially, other informal programs. As discussed in Section III A, one start-of-semester session detailing the modes and explicit presentation as equally valuable is not enough to prevent the development of a hierarchy of modes or to develop a metacognitive awareness of the modes as situationallybeneficial pedagogical moves. To address both of these points in training, we will present more explicitly examples of "good" teaching in each mode (such as Moira's use of instruction mode during the setup of the "laser game") -as well as "bad teaching" when applicable. Beyond PISEC, a similarly self-reflective observation technique could be employed to determine the pedagogy methods naturally employed by facilitators in other programs (including, potentially, the modes of pedagogy) and iteratively adjust training in a design-based manner to empower those facilitators to think more explicitly about their pedagogy.
As discussed in Section III B, UEs' prior experiences in formal teaching may shape their understanding of the nature and goal of teaching in PISEC, and those experiences may shape how they choose to engage in pedagogy. We intend to provide more of an opportunity in training for PISEC UEs to share their prior pedagogical experience and conceptions with one another and with the program director, and to create more opportunities for UEs to discuss more deeply similarities and differences between teaching in formal and informal settings.
PISEC training currently spans the course of one half-day on a weekend before site visits start. To complement existing, less-structured efforts to address the two issues mentioned above for UEs (e.g., site visit deconstructions during transit, PISEC socials), more formal elements, such as a PISEC course or seminar, could be implemented to reinforce ideas brought up in training. Indeed, the CU LA program, mentioned by Moira, supports its student teachers through a weekly pedagogy course -while such structure is not appropriate or feasible for every program, especially those with resource or location constraints, thoughtful consideration of ways to provide continued enculturation and support to facilitators is likely to prove fruitful.

V. CONCLUSIONS AND FUTURE OPPORTUNITIES
We have illustrated in this work a few of the salient factors that likely influence PISEC instructors' enacted pedagogies; future work will present a comprehensive look at these factors and more, and will investigate the causes for these influences. For example, as mentioned in Ref. [13], the possibility exists that these modes manifest in formal spaces as well as informal. A study looking at pedagogy, especially by UEs who participate in both PISEC and a program like CU's LA program, in these spaces could help understand the link between experiences in those spaces and pedagogy in PISEC.
Additionally, we anticipate that findings from this work and improvements to the ways in which PISEC supports and prepares its instructors may be applicable to programs beyond PISEC as well. Public engagement programs are often designed toward providing benefit to participants and, potentially, toward meeting goals of community and culture as well. PISEC has historically striven to ensure that instructors are stakeholders in and beneficiaries from program success, rather than simply the means of the program's operation; a program design paradigm oriented toward improving facilitator support based on research, similar to the present work, into the unique experiences, expectations, and needs of those facilitators could allow programs to more efficiently align facilitators' behavior with program goals while simultaneously making participation more rewarding and beneficial for these instructors, who represent critical components of successful physics outreach.
We gratefully acknowledge the participation of the local community education partners in the PISEC program, as well as the university students who volunteered their time and effort as UEs. This work is supported by the JILA NSF Physics Frontier Center (NSF PHY-1734006) and STROBE (NSF DMR-1548924).