Using Annotation for Transparent Inquiry (ATI) to Teach Qualitative Research Methods

In political science, qualitative analytic methods are rarely taught using “active learning” strategies. We discuss a novel approach to teaching such methods: having students engage with scholarship that has been annotated using Annotation for Transparent Inquiry (ATI). ATI allows authors to annotate passages in a digital publication to clarify methodology, add detail about evidence or analysis, or link to data sources. Learning methods through engagement with annotated articles allows students to interact with original data and to better understand and evaluate how authors collected, analyzed, and used those data. This leads students to learn research methods in a way that more closely approximates how they will use those methods in their own research. We present a general description of strategies for teaching with ATI. We illustrate the approach using three examples of instructors teaching both undergraduate and graduate students. We conclude with recommendations for effectively using ATI in the classroom.


Introduction
The early 2000s saw the beginning of a renaissance in qualitative research methods in the discipline of political science (e.g., Collier andBrady 2004, George andBennett 2005). This renaissance has included the development of more systematic and analytically explicit approaches to using qualitative evidence for descriptive and causal inference. Unfortunately, however, the teaching of qualitative research methods has not kept pace with their development. In particular, a recent study of the methods curriculum in 25 top political science doctoral programs between 2010 and 2015 found that qualitative methods instruction tends to take a passive rather than an active form: in marked contrast to the teaching of quantitative research methods, students rarely learn qualitative analytic methods by practicing their use on data generated and shared by other researchers (Emmons and Moravcsik 2019, 4). While not necessarily representative of the broader discipline, these findings suggest an important gap in the instruction of qualitative methods at the graduate level.
In this brief piece, we discuss a novel approach to teaching qualitative methods that employs more "active learning": through students engaging with scholarship that has been annotated using Annotation for Transparent Inquiry (ATI). ATI is a new approach to achieving transparency in qualitative and multi-method research that allows authors to annotate specific passages in a digital publication in order to clarify methodological challenges and choices, add detail about evidence or analysis, or link to data sources that underlie claims.
Learning methods through engagement with annotated articles allows students to interact with original data and to better understand, evaluate, and critique how authors collected, analyzed, and used those data to draw inferences and develop arguments. Compared with traditional pedagogical techniques, this approach thus leads students to learn research methods in a way that more closely approximates how they will use those methods in their own research.
Our discussion in this essay is somewhat speculative. To date, few of the teaching strategies that we propose have been used in practice, for reasons we discuss below.
Nonetheless, case studies we present in this piece demonstrate that initial efforts to employ these techniques have yielded promising pedagogical outcomes. These results, in tandem with the broad consensus in the education literature on the value of active-learning techniques, recommend continuing to develop ATI-oriented strategies. We hope this essay catalyzes further exploration and experimentation.
We begin by offering a few observations on qualitative methods instruction in political science, in particular elaborating on the infrequency with which methods for analyzing qualitative data are taught via active learning with data generated and shared by other researchers. We then briefly describe ATI and its potential to facilitate the active learning of qualitative analytic methods, and illustrate that potential using the experiences of three methods instructors. We conclude with a short discussion of some limits of this pedagogical approach.
The Status Quo and the Challenge Active learning -the use of "instructional activities involving students in doing things and thinking about what they are doing" (Bonwell and Eison 1991, 2) -is widely considered to be an effective pedagogical technique that facilitates skills acquisition and generates student interest (National Research Council, 2000;Hmelo-Silver & Rehak, 2017). While active learning techniques are often used to teach techniques for collecting qualitative data (e.g., archival research, interviewing, and ethnography), recent research suggests that it is unusual for students to learn qualitative analytic methods through practicing their use on data that other scholars generated and shared (Emmons and Moravcsik 2019, 4). 1 This marks a notable contrast with the instruction of quantitative methods, in which students often analyze shared datasets to complete exercises and problem sets, or seek to reproduce using computational methods the findings published in research articles with open data and materials (King 2006;Janz 2016).
Two factors likely contribute to this gap in qualitative methods instruction. First, in contrast with the dozens of textbooks treating quantitative analytic methods, the books most 1 Teaching qualitative analytic methods using shared data rather than data students generated themselves will often be preferable to having students analyze data they have generated themselves since the former approach allows the instructor to select data that are optimally suited for, or to tailor the data to, the particular analytic method being taught. Also, teaching with shared data allows students to learn collaboratively and/or compare inferences from the same data (see National Research Council 2000 on the benefits thereof).
often used in the instruction of qualitative methods tend to focus on the epistemological underpinnings and inferential logics of qualitative analytic methods rather than the practical steps involved in using them. Moreover, few such books include problem sets or other practical exercises. 2 Second, practicing analytic methods using data gathered and shared by other scholars is limited by the paucity of shared qualitative data. Relatedly, qualitative data sets specifically created for teaching are practically non-existent.
To date, then, few graduate students have had the opportunity to learn qualitative analytic methods by practicing their use on shared data. Conditions for employing this promising teaching practice are beginning to improve, however, as scholars are developing new strategies to share their qualitative data in ethical, legal, and meaningful ways (Kapiszewski and Karcher 2020). In the remainder of this piece, we suggest a set of pedagogical strategies that capitalize on these developments and are likely to enhance the instruction of qualitative analytic methods.

A Solution: Annotated Scholarship as a Teaching Tool
In order to make the instruction of qualitative analytic methods more active, we propose that instructors have their students engage with scholarship that has been annotated using ATI. 3 ATI, developed by the Qualitative Data Repository (QDR) and the software nonprofit Hypothesis, is an innovative approach to achieving transparency in qualitative and multi-method research. Authors who use ATI digitally annotate empirical claims or conclusions in their manuscript. ATI employs open web annotation, which allows for the generation, sharing, and discovery of digital annotations across the web. Annotations may include full citations to underlying data sources; "analytic notes" clarifying how the author generated or analyzed their data and/or how the data support inferences or interpretations; excerpts from data sources; and links to the data sources themselves when they can be shared ethically and legally. Annotations, in other words, surface the "analytic scaffolding" and 2 Notable exceptions include "SAGE ResearchMethods" (a subscription based online database with video tutorials, short courses, practice datasets, and case examples) and Cambridge University Press's "Method for Social Inquiry" book series; see also Collier (2011). 3 ATI is similar to "Active Citation" (Moravcsik 2010), an earlier approach to achieving transparency in qualitative inquiry; the two differ in that ATI leverages open annotation technology and emphasizes the importance of linking to underlying data. evidentiary underpinnings that are rarely included in published qualitative work due to space constraints. Annotations are displayed on the same web page as the digitally published article or book that they accompany; they and underlying data sources are curated and preserved by a data repository (see Figure 1). 4 Figure 1: ATI at a Glance: How a passage in the text of an article annotated with ATI appears to readers We highlight two overarching strategies that instructors of qualitative analytic methods might use to integrate scholarship that has been annotated with ATI into their teaching. 5 One strategy entails students examining the analytic scaffolding that ATI reveals; a second involves students engaging in re-analysis and analytic extensions. Both approaches 4 See also Karcher and Weber (2019) and the materials on QDR's website, qdr.syr.edu/ati. 5 A list of published ATI projects can be found at https://qdr.syr.edu/ati/ati-models. work best when the annotations in the scholarship used include links to a rich set of underlying qualitative data.
The first strategy can be thought of as standing in for the step-by-step introduction to qualitative analytic techniques that is often missing from textbooks. In this approach, instructors draw on annotations that elaborate on the analysis of qualitative data, in tandem with materials 6 and data shared by the author, to highlight and discuss with their students the logic of inquiry and specifics of analysis that underlie the exemplars. Students thus learn qualitative research methods by retracing and examining an author's analytical steps (and, potentially, mis-steps). For instance, students can see how authors identified and deployed causal process observations (CPOs) to advance narrative process tracing; learn about the mechanics of Bayesian process tracing from annotations that discuss how prior probabilities were assigned and posterior estimates progressively updated; or gain insight into the conceptualization and coding that underlie the process of calibration in Qualitative Comparative Analysis (QCA).
In the second pedagogical approach, instructors devise a set of structured exercises that call on students to actively engage with an author's analysis through the annotations provided in a publication. Such exercises are somewhat akin to the reproducibility exercises commonly used in quantitative methods instruction. For instance, students can reanalyze the supporting data provided in ATI annotations, using the same method the author used, in order to evaluate the strength of particular claims made in the text of a publication, or regenerate and validate its tables and figures.

Examples of Teaching with ATI
One of us (Jacobs) piloted an ATI-supported exercise using both types of strategies in a graduate qualitative methods class in the fall of 2019. Focusing on annotated portions of either Saunders (2011, Chapter 4) or O'Mahoney (2017), students selected individual CPOs for which the annotation provided either excerpts or reproductions of source materials. In the assignment writeup, students first retraced the author's steps, summarizing how the author sought to link the observation to an empirical implication of the article's or book's theoretical argument. Second, students critically evaluated the inferences the author drew from the evidence, as presented in or linked to from the annotation. Here the annotations allowed students to more thoroughly assess the original inferential reasoning, or provide more plausible alternative explanations, than they could have done from the article text alone.
Engaging with the fuller evidentiary record gave students insight into how authors had selected observations from their source materials and helped illuminate how the meaning of a statement or observation might depend on the context in which it appears. Moreover, reanalyzing the data presented in the article and found in the source material helped demonstrate the differing inferences that might be drawn from a given observation, depending in part on assumptions that the analyst makes and readings of the context in which the observation arises.
In a similar exercise used over multiple semesters in an upper-level undergraduate course focused on primary source-based inquiry, Robert Adcock (American University) asked students to assess the annotated evidence in the Saunders chapter just mentioned. Students drew on the shared source materials to assess the accuracy and precision of the summary or use of that source in the chapter's text, and the sufficiency of the evidence as support for the claim. 7 Students were also asked to think about whether, in aggregate (given their number, type, and authoritativeness), the sources could plausibly be considered representative of the broader set of potential sources. One factor students were asked to consider was the growing variety and number of primary sources available online, which had been discussed earlier in the course. The goal of the exercise was for students to acquire, via actively engaging with Saunders' meticulous use of primary sources, a working model of what they should aspire to in using and presenting primary source evidence as they drafted their own research papers in subsequent weeks of the course. 7 Personal correspondence, Robert Adcock to the authors, January 20-25, 2021, see also the description in Karcher (2016). Adcock's written instructions for this exercise are included in this article's supplementary material. Juan Masullo (Oxford University and Leiden University) has integrated ATI into his instruction of process tracing using both types of strategies described above. 8 In one exercise, 9 Masullo leads students to discover process tracing's architecture by asking them to read passages of an article annotated using ATI without the annotations before class, identifying evidentiary and analytic weaknesses, gaps, or silences; he and the students then re-read those passages with the annotations together in class, assessing how well the annotations address the pre-identified issues and contextualize and characterize the value of evidence. In another exercise, Masullo helps students to reveal the analytic architecture of their own empirical work by asking them to annotate it with ATI. In correspondence with us about this exercise, Masullo has emphasized the importance of guiding students to annotate with purpose, for instance, to justify citing evidence or discuss its probative value; to describe analytically relevant aspects of respondents' profile; or to make more explicit the logic of process tracing tests or Bayesian analysis. 10 One could imagine several extensions of these assignments in which students are encouraged to go beyond the analytic methods, data, and/or materials used by the authors of annotated work. For instance, students might be asked to re-analyze data shared in ATI annotations using a different analytic method from the one the author used to see if they arrive at the same results (e.g., they could apply formal Bayesian process tracing to an article that analyzed evidence informally). Even more ambitiously, instructors could ask students to search in the broader corpus of information to which the author linked in the annotations (or provided in addition to them) for evidence that corroborates or calls into question a publication's claims. To go one step further, students might practice reasoning with evidence by seeking to develop and answer research questions different from (but perhaps related to) those asked in an annotated publication using the corpus of information authors provide in their annotations or by triangulating between those and other data.
In sum, the growing availability of ATI-based publications enables more active forms of learning qualitative analytic methods. Active learning of these methods can generate better 8 In preparation, students read three pieces on process tracing tests and three on Bayesian approaches to process tracing. Masullo's description of his exercise is included in this article's supplementary material. 9 Masullo often uses his own work (e.g., Masullo (2020) for this exercise, which facilitates more in-depth discussion with students of the general logic of, and specific justification for, annotation. 10 Personal correspondence, Juan Masullo to the authors, January 21-24, 2021. understanding of how scholars draw descriptive and causal inferences from qualitative evidence, making students more sophisticated consumers of qualitative research. Moreover, active engagement with shared research data helps students learn to use qualitative methods by applying them.

Conclusion
In this article, we have described some general strategies for, and offered some examples of, introducing active learning elements into qualitative methods instruction using ATI. We discussed the experiences of three instructors who used ATI in the classroom; each found that doing so improved learning outcomes and all three continue to employ and refine their approach. As more articles annotated with ATI are published, instructors will be able to use them in teaching a broader range of analytic methods. 11 We note that there are similarities between the logic of teaching qualitative methods with ATI and the logic of teaching quantitative methods via reproduction exercises. The usefulness of reanalysis exercises using ATI, however, does not at all hinge on that parallel.
When authors use ATI, students can assess authors' findings by seeing if those findings can be reproduced from the data made available. Yet because most qualitative analysis involves some degree of subjective interpretation, different qualitative analysts might reasonably draw different inferences even when using the same methods to analyze the same data. In addition, some qualitative methodologies, particularly those operating within an interpretivist paradigm, eschew empiricist notions of researcher-independent objectivity. In these research traditions, reproducibility is itself incoherent as an evaluative standard. Even where analytical imprecision or epistemological premises make reproducing findings implausible or meaningless, however, students are likely to arrive at a better understanding of the logic and practice of different methods if they have the opportunity to apply them to sources or evidence underlying a publication, and compare their reasoning and findings to those presented by the original authors.
Even within a replicationist paradigm, learning qualitative methods with ATI has limits. Even when authors provide or link to sources or data in their annotations, annotated articles rarely offer visibility into how authors selected those sources or data from all the information they encountered or collected. 12 Further, not all work annotated with ATI is equally suitable for teaching. The initial experiences of teaching with ATI described here suggest that annotated publications that provide at least some associated primary material will be most useful pedagogically. Also, annotations that effectively lift the curtain on the research process --e.g., by discussing the evidentiary value of data cited in the text, commenting on how to adjudicate between contradictory evidence, and highlighting choices made during data collection -can be particularly enlightening for students.
Finally and more broadly, although the initial experiences recounted here are encouraging, we have limited evidence to date on the efficacy of using ATI to teach qualitative methods. We hope that this article encourages more instructors of qualitative research methods to bring ATI into their classrooms so that together we can continue to develop, assess, and capitalize on its pedagogical promise.