a large and growing body of research indicates that undergraduate students learn more in courses that use active - engagement instructional approaches ( prince , 2004 ; knight and wood , 2005 ; michael , 2006 ; blanchard et al . ,
, the importance of teaching science , technology , engineering , and mathematics ( stem ) courses more effectively has been stressed in numerous reports , including the president 's council of advisors on science and technology engage to excel report ( 2012 ) , the national science foundation / american association for the advancement of science vision and change report ( aaas , 2010 ) , and the national research council discipline - based education research report ( singer et al . , 2012 ) .
given these compelling , evidence - based recommendations and the recognized need for measures of teaching effectiveness beyond student evaluations ( association of american universities , 2011 ) , higher education institutions are struggling to determine the extent to which faculty members are teaching in an interactive manner .
this lack of information is a major barrier to transforming instruction and evaluating the success of programs that support such change . to collect information about the nature of stem teaching practices as a means to support institutional change , faculty at both the university of british columbia ( ubc ) and the university of maine ( umaine ) created classroom observation programs .
the results of such observations were needed to : 1 ) characterize the general state of stem classroom teaching at both institutions , 2 ) provide feedback to instructors who desired information about how they and their students were spending time in class , 3 ) identify faculty professional development needs , and 4 ) check the accuracy of the faculty reporting on the teaching practices survey that is now in use at ubc ( cwsei teaching practices survey , 2013 ) . to achieve these goals , the programs needed an observation protocol that could be used by faculty member observers to reliably characterize how students and instructors were spending their time in undergraduate stem classrooms .
a critical requirement of the protocol was that observers who were typical stem faculty members could achieve those results with only 1 or 2 hours of training , as it is unrealistic to expect they would have more time than that available . in the quest for a suitable observation protocol , multiple existing options were considered , and ultimately rejected .
the observation protocols considered were divided into two categories : open - ended or structured . when observers use open - ended protocols , they typically attend class , make notes , and respond to such statements as :  comment on student involvement and interaction with the instructor  ( millis , 1992 ) .
although responses to these types of questions can provide useful feedback to observers and instructors , the data are observer dependent and can not easily be standardized or compared across multiple classrooms ( e.g. , all stem courses at ubc or umaine ) .
alternatively , structured protocols provide a common set of statements or codes to which the observers respond .
often , these protocols ask observers to make judgments about how well the teaching conforms to a specific standard .
examples of such protocols include the inside the classroom : observation and analytic protocol ( weiss et al . , 2003 ) and
these protocols consist of statements that observers typically score on a likert scale from  not at all  to  to a great extent  and contain such statements as :  the teacher had a solid grasp of the subject matter content inherent in the lesson 
for example , it has been used to evaluate university - level courses at several different institutions to measure the effectiveness of faculty professional development workshops ( ebert - may et al . , 2011 ) and to compare physics instructors in a study examining coteaching as a method to help new faculty
, 2011 ) . the rtop is also being used to characterize classroom practices in many institutions and in all levels of geoscience classes ( classroom observation project , 2011 ) .
the rtop was found to be unsuitable for the ubc and umaine programs for two main reasons .
the first is that the protocol involves many observational judgments that can be awkward to share with the instructor and/or the larger university community .
the second is that observers must complete a multiday training program to achieve acceptable interrater reliability ( irr ; sawada et al . , 2002 ) .
more recently , new observation protocols have been developed that describe instructional practices without any judgment as to whether or not the practices are effective or aligned with specific pedagogic strategies .
these observation protocols use a series of codes to characterize instructor and/or student behaviors in the classroom ; observers indicate how often each behavior occurs during a class period ( hora et al . , 2013 ; west et al . , 2013 )
one observation protocol in particular , the teaching dimensions observation protocol ( tdop ) , was expressly developed to observe postsecondary nonlaboratory courses . for this protocol ,
observers document classroom behaviors in 2-min intervals throughout the duration of the class session ( hora et al . , 2013 ) .
the possible classroom behaviors are described in 46 codes in six categories , and observers make a checkmark when any of the behaviors occur .
the tdop instrument avoids the judgment issues associated with the rtop , but it still requires substantial training , as one might expect for a protocol that was designed to be a complex research instrument .
preliminary work suggests that , after a 3-day training session , observers have acceptable irr scores when using the tdop ( hora et al . , 2013 ) .
observers at our institutions tried using this instrument , but without the full training , they found it difficult to use the tdop in a reliable way , due to the complexity of the items being coded and the large number of possible behavior codes .
we also found that the particular research questions it was designed to address did not entirely align with our needs .
for example , it covers some aspects that are not necessary for faculty observation programs , such as whether an instructor uses instructional artifacts ( e.g. , a laser pointer or computer ; hora et al . ,
2013 ) and fails to capture others that are needed , such as whether an instructor encourages peer discussion along with clicker questions ( mazur , 1997 ; smith et al . , 2009 , 2011 ) .
we also wanted to better characterize the student behaviors during the class period than the tdop easily allowed .
out of necessity , we created a new protocol called the classroom observation protocol for undergraduate stem , or copus . like the tdop
, this new protocol documents classroom behaviors in 2-min intervals throughout the duration of the class session , does not require observers to make judgments of teaching quality , and produces clear graphical results .
however , copus is different in that it is limited to 25 codes in only two categories (  what the students are doing  and  what the instructor is doing  ) and can be reliably used by university faculty with only 1.5 hours of training ( figure 1 has a description of the codes ; the supplemental material includes the full protocol and coding sheet ) .
observers who range from stem faculty members without a background in science education research to k12 stem teachers have reliably used this protocol to document instruction in undergraduate science , math , and engineering classrooms . taken together ,
the development of copus was an evolutionary process extending across more than 2 years , involving many iterations and extensive testing .
it began at ubc , where science education specialists ( sess ) who were working with science faculty on improving teaching ( wieman et al . , 2010 ) wanted to characterize what both the students and instructors were doing during class .
the sess began testing various existing protocols , including the tdop , in different classes at ubc in late 2011 and early 2012 .
the original tdop did not meet our needs ( as described above ) , so we iteratively modified the protocol through nine different versions .
these changes resulted in a format , procedure , data structure , and coding strategy that was easy to implement on paper or electronically and convenient for analysis and display .
the overall format of the observation protocol remained largely stable , but the categories and codes continued to evolve . during the fall term of 2012 , 16 sess , who are highly trained and experienced classroom observers , used this evolving protocol to observe a variety of courses in singles , pairs , or trios across most of the departments in the ubc faculty of science ( including the disciplines of biology , computer science , earth sciences , mathematics , physics , and statistics ) .
we analyzed the ses generated observation data to identify coding disagreements and met with the sess to discuss the evolving protocol and coding .
these discussions covered observed behaviors they found difficult to code and/or hard to interpret , and other important elements of instructor or student behavior they felt were not being adequately captured .
the final version had substantially simplified categories and all identified problems with the wording on the codes had been eliminated .
notably , it was quite simple to reliably code classes taught with traditional lectures , as a very small number of behaviors need to be coded . therefore , the majority of the work went into improving the protocol so it could reliably characterize classes that had substantial and varied interactions between instructor and students and multiple student activities .
one substantial change during fall 2012 was eliminating a category for judging the cognitive level of the activities .
observers had been asked to code the level of cognitive sophistication of current classroom activities , based on bloom 's taxonomy of educational objectives ( bloom et al . , 1956 ) . after multiple unsuccessful attempts to find a simple and reliable coding scheme that could capture this aspect of the classroom activities , we dropped this category . our decision to drop this category
is supported by recent work showing that , when faculty members write and evaluate higher - order questions , they use several criteria beyond the bloom 's level , including : question difficulty , time required to answer the questions , whether students are using a new or well - practiced approach , and whether the questions have multiple reasonable solutions ( lemons and lemons , 2012 ) .
the second substantial change during this time was changing another category  coding the level of student engagement  from required to optional . having a measure of student engagement
is useful for providing feedback to the instructor and for judging the overall effectiveness of many instructional activities . with the coding of the levels of engagement simplified to only discriminating between low ( 020% of the students
engaged ) , medium , or high ( 80% of the student engaged ) , some observers , particularly those who had some experience with observing levels of student engagement , could easily code engagement along with the other two categories , and there was reasonable consistency between observers .
however , less - experienced observers found it quite hard to simultaneously code what the students were doing , what the instructor was doing , and the student engagement level .
also , there were difficulties with obtaining consistent coding of student engagement across all observers ; the judgments were often dependent on the levels of engagement common to the specific disciplines and courses with which the observers were familiar .
we recommend observers do not try to code it until after they have become experienced at coding the  what the students are doing  and  what the instructor is doing  categories .
another recurring theme of the discussions with the sess was the extent to which classroom observations could accurately capture the quality of instruction or the efficacy of student work . in the end , after sess observed different classes across many disciplines , there was a consensus that accurately evaluating the quality of instruction and the efficacy of student work was generally not possible .
these highly trained and experienced observers concluded that these evaluations require a high degree of training of the observer in the material and the pedagogic strategies , as well as familiarity with the student population ( prior knowledge , typical classroom behaviors , etc . ) .
we concluded that quality judgments of this type were not realistic goals for limited classroom observations carried out by stem faculty members .
thus , the present version of copus captures the actions of both instructors and students , but does not attempt to judge the quality of those actions for enhancing learning .
after the completion of this development work at ubc , the copus was further tested by 16 k12 teachers participating in a teacher professional development program at umaine .
the teachers used the copus to observe 16 undergraduate stem courses in five different departments ( biology , engineering , math , chemistry , and physics ) .
while the teachers easily interpreted many of the codes , they found a few to be difficult and suggested additional changes .
for example , the student code  listening : paying attention / taking notes , etc . 
the code was clarified , so observers knew they should select this code only when the students were listening to their instructor , not when students were listening to their peers .
also , new codes were added to capture behaviors the teachers thought were missing , such as the instructor code 
anq : listening to and answering student questions with entire class listening .  the coding patterns of the two teacher observers in the same classroom were also compared to determine which specific codes were difficult to use consistently .
figure 2a compares how two observers marked this code in the first iteration of testing , when it was described 
observer 2 marked this code throughout most of the class , and observer 1 marked this code intermittently .
follow - up conversations with observer 2 and other teachers indicated that some observers were marking this code throughout the duration of the class , because they assumed individual students were thinking while they were taking notes , working on questions , and so on , but other observers were not . therefore , we clarified the code to be :  ind : individual thinking / problem solving . only mark when an instructor explicitly asks students to think about a clicker question or another question / problem on their own .
 figure 2b shows a comparison of the same observer pair , with the revised 
a comparison of how two observers coded the student code  ind .  ( a ) when the code was described as 
ind : individual thinking / problem solving in response to assigned task ,  observer 2 marked this code more often than observer 1 did .
in addition , the teacher observation data revealed a more general problem : there was a lower degree of consistency in coding student behaviors than in coding instructor behaviors , and the teachers used a very limited set of codes for the student behaviors .
the earlier coding by the sess had shown similar , but less dramatic , trends .
we realized that this problem was due to a natural tendency of observers to focus on the instructor , combined with the fact the instructor - related codes came first on the survey form .
therefore , the protocol was changed , with the student codes viewed first , and we emphasized coding student behaviors during subsequent training sessions ( see further details below in the training section ) .
this version was tested by having the same 16 k12 teachers use it to observe 23 umaine stem classes , and by having seven stem faculty observers use it to observe eight ubc classrooms in pairs after 1.5 hours of training .
the seven ubc stem faculty member volunteers who used the final protocol had not previously used the protocol and were not involved in the development process .
thus , the irr of the protocol has been tested with a sample of observers with a wide range of backgrounds and perspectives . as discussed in validity and reliability ,
information on the courses observed using the final version of the copus stem courses at the first- and second - year levels .
biology , molecular biology , engineering , chemistry , math , physics , and geology .
a critical design feature of the copus is that college and university faculty who have little or no observation protocol experience and minimal time for training can use it reliably .
we summarize the training steps in the following paragraphs , and we have also included a step - by - step facilitator guide in the supplemental material . the first step in the training process is to have the observers become familiar with the codes . at ubc , facilitators displayed the student and instructor codes ( figure 1 ) and discussed with the observers what each behavior typically looks like in the classroom . at umaine , the teacher observers played charades .
each teacher randomly selected a code description from a hat and silently acted out the behavior .
the remaining observers had the code descriptions in front of them and guessed the code .
the remainder of the training was the same for both groups , with a total training duration of 2 hours for the k12 teachers and 1.5 hours for the ubc faculty members .
second , observers were given paper versions of the coding sheet and practiced coding a 2-min segment of a classroom video .
an excerpt from the coding sheet is shown in figure 3 , and the complete coding sheet is included in the supplemental material .
the first video we used showed an instructor making administrative announcements and lecturing while the class listened .
after 2 min , the video was paused , and the group discussed which codes they selected . because faculty at other institutions may have difficulty capturing videos for training , we have included web urls to various video resources that can be used for training ( table 2 ) .
observers place a single checkmark in the box if a behavior occurs during a 2-min segment .
video resources that may be helpful for copus training the observers were then asked to form pairs and code 8 min of a video from a large - enrollment , lecture - style science class at umaine that primarily shows an instructor lecturing and students listening , with a few questions asked by both the instructor and students . to keep the observers synchronized and ensure they were filling out a new row in the observation protocol at identical 2-min intervals , they used either cell phones set to count time up or a sand timer . at the end of 8 min ,
next , as a large group , observers took turns stating what they coded for the students and the instructor every 2 min for the 8-min video clip . at this point , the observers talked about the relationship between a subset of the student and instructor codes .
for example , if the observers check the student code  cg : discuss clicker question ,  they will also likely check the instructor code  cq : asking a clicker question .  to provide the observers with practice coding a segment that has more complicated student and instructor codes , they next coded a different classroom video segment from the same large - enrollment , lecture - style science class at umaine , but this time the camera was focused on the students .
this video segment included students asking the instructor questions , students answering questions from the instructor , and clicker questions with both individual thought and peer discussion .
then observers in pairs coded for an additional 6 min , again taking care to use synchronized 2-min increments .
the observer pairs first compared their codes with their partners , and then the whole group discussed the student and instructor codes for each of the 2-min segments of the 6-min clip . at this point , the training was complete .
copus is intended to describe the instructor and student actions in the classroom , but it is not intended to be linked to any external criteria .
hence , the primary criterion for validity is that experts and observers with the intended background ( stem faculty and teachers ) see it as describing the full range of normal classroom activities of students and instructors .
that validity was established during the development process by the feedback from the sess , the k12 teachers , and those authors ( m.s .
a major concern has been to ensure that there is a high level of irr when copus is used after the brief period of training described above . to assess the irr
, we examined the agreement between pairs of observers as they used the final version of copus in stem classes at both ubc and umaine .
the two observers sat next to each other in the classroom , so they could keep identical 2-min time increments , but the observers were instructed not to compare codes with each other . to summarize how similarly observer pairs used each code on the final version of the copus , we calculated jaccard similarity scores ( jaccard , 1901 ) for each code and then averaged the scores for both the ubc and umaine observers ( table 3 ) .
for single codes , we calculated jaccard similarity scores instead of irr cohen 's kappa values , because observer pairs occasionally marked the same code for every 2-min increment throughout the duration of the class .
for example , in a class that is lecture - based , observers would likely mark the student code 
, the observer opinion is defined as a constant rather than a variable , which interferes with the irr calculation .
average jaccard similarity scores for copus codes across all pairs observing in all courses for both ubc faculty observers and maine k12 teacher observers ; numbers closer to 1 indicate the greatest similarity between two observers  sp : presentation by students  and  tq : test / quiz  were not selected in any of the observations at ubc or umaine .
this result likely occurred because when we asked ubc and umaine faculty members if we could observe their classes , we also asked them if there was anything unusual going on in their classes that day .
we avoided classes with student presentations and tests / quizzes , because these situations would limit the diversity of codes that could be selected by the observers .
nc ) , where nc = the number of 2-min increments that are marked the same ( either checked or not checked ) for both observers , na = the number of 2-min increments that are marked the same for both observers plus 2-min increments observer 1 marked that observer 2 did not , nb = number of 2-min increments that are marked the same for both observers plus 2-min increments observer 2 marked that observer 1 did not . for example , for the data in figure 2b , the class period is 42 min in length , so there are 21 possible 2-min segments .
the student code  ind : individual thinking  was marked 12 times by observers 1 and 2 , not marked eight times by both observers , and marked by observer 2 one time when observer 1 did not .
numbers closer to 1 indicate greater consistency between how the two observers coded the class .
eighty - nine percent of the similarity scores are greater than 0.90 , and the lowest is 0.80 .
the lowest score for both the ubc and umaine observers was for the instructor code  pq : posing nonclicker questions . 
comments from observers suggest that , when instructors were following up / giving feedback on clicker questions or activities , they often posed questions to the students .
fup : follow - up  to describe this behavior but stated they occasionally forgot to also select the instructor code  pq .  to compare observer reliability across all 25 codes in the copus protocol , we calculated cohen 's kappa irr scores using spss ( ibm , armonk , ny ) . to compute the kappa values for each observer pair , we added up the total number of times : 1 ) both observers put a check in the same box , 2 ) neither observer put a check in the same box , 3 ) observer 1 put a check in a box when observer 2 did not , and 4 ) observer 2 put a check in a box when observer 1 did not .
for example , at ubc , when looking at all 25 codes in the copus , one observer pair had the following results : 1 ) both observers put a check in 83 of the same boxes , 2 ) neither observer put a check in 524 of the boxes , 3 ) observer 1 marked six boxes when observer 2 did not , and 4 ) observer 2 marked 12 boxes that observer 1 did not . using data such as these , we computed the kappa score for each of the eight ubc and 23 umaine pairs and report the average scores in table 4 .
we also repeated this calculation using either the subset of 13 student or 12 instructor codes ( table 4 ) .
average irr kappa scores from the observations at ubc and umaine the average kappa scores ranged from 0.79 to 0.87 ( table 4 ) .
these are considered to be very high values for kappa and thus indicate good irr ( landis and koch , 1977 ) .
notably , the kappa values , as well as the jaccard similarity scores , are comparably high for both ubc faculty and umaine k12 teacher observers , indicating that copus is reliable when used by observers with a range of backgrounds and 2 hours or fewer of training .
to determine the prevalence of different codes in various classrooms , we added up how often each code was marked by both observers and then divided by the total number of codes shared by both observers .
for example , if both observers marked  instructor : lecture  at the same 13 time intervals in a 50-min class period and agreed on marking 25 instructor codes total for the duration of the class , then 13/25 , or 52% of the time , the lecture code occurred for the instructor .
figure 4 shows observation results from two illustrative classes : one that is primarily lecture - based and one in which a combination of active - learning strategies are used .
this example illustrates how , at a glance , this visual representation of the copus results provides a highly informative characterization of the student and instructor activities in a class . a comparison of copus results from two courses that have different instructional approaches . at a department- or institution - wide level ,
 across all courses observed , because this student code is the most indicative of student passive behavior in response to faculty lecturing (  lec  ) with or without real - time writing (  rtw  ) .
however , at umaine , some of the classes have greater than 76% of the student codes devoted to listening .
faculty who teach these classes may benefit from professional development activities about how to design an effective active - learning classroom .
in addition , the data can be analyzed for a subset of faculty members who are using active - learning strategies , such as asking clicker questions .
thirty - eight percent of ubc and 43% of the umaine classes that were observed used clickers .
however , student code prevalence in these classes show that not all faculty members used clicker questions accompanied by recommended strategies , such as peer discussion ( mazur , 1997 ; smith et al . , 2009 , 2011 ; figure 6 ) .
faculty members who are not allowing time for peer discussion may benefit from professional development on how to integrate peer discussion into clicker questions .
prevalence of student codes in four example courses that use clickers . in courses that use clickers with no or minimal peer discussion
copus was developed because university observation programs needed a protocol to : 1 ) characterize the general state of teaching , 2 ) provide feedback to instructors who desired information about how they and their students were spending class time , and 3 ) identify faculty professional development needs .
copus meets all of these goals by allowing observers with little observation protocol training and experience to reliably characterize what both faculty and students are doing in a classroom .
faculty members can receive pie charts with their code prevalence results ( examples in figure 4 ) .
these results provide a nonthreatening way to help faculty members evaluate how they are spending their time .
we discovered that faculty members often did not have a good sense of how much time they spent on different activities during class , and found copus data helpful .
in addition , faculty members can use copus data in their tenure and promotion documents to supplement their normal documentation , which typically includes student evaluation information and a written description of classroom practices .
having observation data gives faculty members substantially more information to report about their use of active - learning strategies than is usually the case .
for example , anonymized , aggregate copus data across all departments have been shared with the umaine center for excellence in teaching and assessment , so workshops and extended mentoring opportunities can better target the needs of the faculty .
one area in particular that will be addressed in an upcoming professional development workshop is using clickers in a way that promotes peer discussion . the idea for this workshop came about as a result of the copus evidence showing the prevalence of umaine stem classes that were using clickers but allowing no or minimal time for recommended student peer discussions ( figure 6 ) .
other planned uses for copus include carrying out systematic observations of all instructors in a department at ubc in order to characterize teaching practices .
the information will be used with other measures to characterize current usage of research - based instructional practices across the department 's courses and curriculum . in the end , the choice of observation protocol and strategy will depend on the needs of each unique situation .
copus is easy to learn , characterizes nonjudgmentally what instructors and students are doing during a class , and provides data that can be useful for a wide range of applications , from improving an individual 's teaching or a course to comparing practices longitudinally or across courses , departments , and institutions .
 to help institutions collect information on undergraduate teaching practices , the authors developed a new classroom observation protocol known as the classroom observation protocol for undergraduate stem ( copus ) . 
 this protocol allows college science , technology , engineering , and mathematics faculty , after a short training period , to reliably characterize how faculty and students are spending their time in class . 
INTRODUCTION
DEVELOPMENT
TRAINING
VALIDITY AND RELIABILITY
ANALYZING COPUS DATA
DISCUSSION AND SUMMARY
Supplementary Material