Exploring Research Topics in the Field of School Librarianship based on Text Mining

Copyright of works published in School Libraries Worldwide is jointly held by the author(s) and by the International Association of School Librarianship. The author(s) retain copyright of their works, but give permission to the International Association of School Librarianship to reprint their works in collections or other such documents published by or on behalf of the International Association of School Librarianship. Author(s) who give permission for their works to be reprinted elsewhere should inform the Editors of School Libraries Worldwide and should ensure that the following appears with the article: Reprinted, with permission, from School Libraries Worldwide, Volume 24, Number 1, January 2018 pages 1-27. doi: 10.14265.23.1.0011 Exploring Research Topics in the Field of School Librarianship based on Text Mining

"Insanity is doing the same thing over and over and expecting a different result." This quote, which is generally attributed to Einstein, rings true for school library research. For decades, school library scholars (Clyde, 2002;Fitzgibbons & Callison, 1991;Grover & Fowler, 1993;Mardis, 2011) have lamented the lack of diversity in research methods used to investigate problems and issues in school librarianship. Recent actions and calls by the American Association of School Librarians (2014) have focused on the importance of and need for causal studies in school library research as one means to stop the madness. Indeed, experimental and quasi-experimental studies are important for any field, but they tend to be particularly challenging to pull off in educational contexts (Villarreal, Gonzalez, McCormick, Simek, and Yoon, 2013). Yet, there are other methods beyond experimental and quasiexperimental that are also germane to, yet untried in, school library research contexts.
This study provides an illustration of text mining as an example of one such method useful to school librarianship. We apply the method to explore topics in the two major journals of school librarianship (Beesoon & Branch-Mueller, 2015;Clyde, 2004): School Library Research (SLR formerly named School Library Media Research) and School Libraries Worldwide (SLW).

Research Questions
In this study, we are interested in what topics are studied in the field of school librarianship. The following research questions guide the investigation: 1. What research topics emerged from school library research journals over the past 10 years? 2. How have trends changed or developed across those 10 years? 3. What are the topical differences between SLW and SLR?
Results of the study can serve as a resource for both practicing school librarians and school library researchers to better understand the topics of recent school library research, but our larger goal is to position the method as one useful to school librarians and worthy of further attention, and to encourage school library researches to engage in interdisciplinary research.

Literature Review
For several decades, researchers have been analyzing published research in the field of school librarianship to determine trends, identify methodologies, and evaluate the extent to which research addresses the needs of practitioners. Investigations of this type are particularly appropriate for school librarianship given the recent guidance issued from the United States Department of Education (2016) about the importance of evidence-based interventions as a means for strengthening educational outcomes. Indeed, taking stock of the existing research in a field supports both the professionals who shape their practice according to study results, as well as the researchers who build off previous studies and also address gaps in the professional knowledge base. Analyses (Eisenberg et al, 1990;Fitzgibbons & Callison, 1991;Grover & Fowler, 1993) in the 1980s and 1990s pointed to trends resulting from the infusion of technology in education, as well as issues that continue to face school libraries today such as: outreach to underserved groups, information literacy promotion, literacy promotion, program evaluation, and intellectual freedom. Additionally, these analyses revealed some alarming tendencies including limited use of data collection techniques, specifically an over-reliance on survey-based studies, a failure on the part of researchers to build on previous studies, and minimal use of experimental design in research, and the need for evaluation studies focused on student learning.
In the early 2000s Clyde (2002Clyde ( , 2003Clyde ( , 2004 brought an international scope to the analysis of school library research. Clyde examined a total of 484 school library research articles published between the years 1991 and 2000 in an effort to better understand topics explored, methods employed, and where, both country and publication venue, studies were conducted. Clyde used a classification scheme of library and information science (LIS) topics developed by Jarvelin and Vakkari (1993) but found that the topics of studies within the domain of school library research differed significantly from those of the larger LIS field; thus, she adapted the tool.
In the last decade other scholars have continue this line of analysis. Wirkus (2006) analyzed articles published in School Library Media Research (the predecessor to SLR) between 1998 and 2004 as well as other articles published between 1992 and 2003 retrieved from the ERIC database using the search, ""school libraries" AND "research" NOT "student research"" (p. 9). In addition to examining methods, the study used the five-category classification scheme developed by Grover and Fowler (1993) to label topics: technology, clientele, information resources, library media specialist, and library media center. Oberg (2006) updated the work of Clyde using the same protocol as the earlier studies to analyze research published between 1995 and 2006. Mardis (2011) also took an approach similar to Clyde, but rather than journal articles, this research examined IASL Research Forum papers from 1998 through 2009, and in addition to using the modified classification scheme of the earlier studies, she also matched the research according to school librarian role (AASL, 2009): teacher, information specialist, instructional partner, and program administrator. Very recently, Beesoon and Branch-Mueller (2015) analyzed research published in SLW and SLR between 2009 and 2013 as well as five other school library research articles published in other peer-reviewed journals and retrieved through database searches. Using similar methods and categorizations developed by Clyde and adapted by Mardis, these researchers concluded that topics of school library research have expanded recently to include the role of the school librarian, programming issues, standards, and student achievement.
Taken together, results of the topic analysis studies conducted in the 21st century corroborate some of the methodological concerns of the earlier analyses (Eisenberg et al, 1990;Fitzgibbons & Callison, 1991;Grover & Fowler, 1993): disproportionate use of surveys and questionnaires to collect data and few experimental studies. While all of these investigations have examined the topics of school library research and have contributed much to the professional literature, they have all used similar methods, and most used the same classification scheme.

Research Domain Analysis Using Text Mining
The benefits of using text mining in research domain analysis lie in the ability to extract key terms and topics objectively from a large set of text data. Traditionally, clustering and principal component analysis (PCA) have been widely used in research domain analysis. For example, Biehl, Kim, and Wade (2006) employed both hierarchical clustering and PCA to examine the knowledge structure of business research. Similarly, Kipp, Joo and Choi (2013) used clustering and PCA at the same time to explore dimensions of consumer health information based on text analysis of social tags. In this way, clustering and dimensional reduction have served as popular tools for analyzing a specific domain for a long time.
Latent Dirichlet allocation (LDA) topic modeling is a relatively recent method which is used to discover hidden topics or themes from large unstructured text collections. It is used to identify prevailing topics in a certain research field or domain. LDA topic modeling is a statistical probabilistic method to analyze text data, which was first described by Blei, Ng and Jordan (2003). The LDA method is based on the analysis of the relationships between observed words in the text document sets to abstract topics underlying the text. This method has been applied in various areas, in particular text mining and information retrieval. For example, Griffiths and Steyvers (2004) analyzed research topics in the Proceedings of the National Academy of Sciences of the United States of America (http://www.pnas.org) based on text mining. Using topic modeling, they identified a range of topics constituting a meaningful knowledge structure of the document collection. Moreover, they analyzed the trend of the topic changes and identified which topics were most discussed in the field. Blei (2012) showed text analysis cases to extract topics in a specific research discipline. For example, Blei analyzed articles of Yale Law Journal to extract topics using topic modeling and then demonstrated that topic modeling can be widely used for topic analysis in various research fields, such as political science and psychology. Recently, Wang, Joo and Lu (2014) employed the LDA topic modeling to identify 25 key subjects in the field of data science.
In addition, topic modeling can be used to trace the topic changes over time in a specific field. Wang and McCallum (2006) proposed a model of topics over time (TOT) based on topic modeling to trace the topic structure changes over time. They applied the TOT method to examine topical changes in multiple document sets, such as personal emails, conference research papers, and presidential state-of-the-union addresses. The method they suggested shows topic modeling can be used to examine time-series patterns of topics in a specific domain. Park and Song (2013) explored topics of information science research in Korea by analyzing 3,834 articles from four journals between 1970 and 2012, and estimated the growing trend in popular research topics over decades. Moreover, topic modeling can be used to identify the relationships between researchers in certain disciplines. For example, Lu and Wolfram (2012) proposed a new method to measure author research relatedness based on topic analysis. They calculated topic similarities between researchers in the field of information science, and drew a multi-dimensional scaling map to show the relationships among researchers in the field. Joo and Lu (2013) analyzed the author vector space derived from LDA to examine a knowledge structure in the information science field.

Data Collection
Previous research (Beesoon & Branch-Mueller, 2015;Clyde, 2004) demonstrates that SLW and SLR account for the lion's share of published research in the field of school librarianship. Thus, we delimited our analysis of school library research to these two journals, SLW and SLR, for this study.
SLR is the research journal of AASL. The purpose of SLR "is to promote and publish high quality original research concerning the management, implementation, and evaluation of school 18 library programs" (AASL, 2013). The articles are compiled into volumes annually with individual articles appearing and accessible on the website as they are accepted and approved for publication.
SLW is the research journal of the International Association of School Librarians (IASL). According to the IASL website, the journal "is issued twice yearly in January and July. Contributors are invited to submit manuscripts for publication on current research on any aspect of school librarianship. Each issue of the journal usually includes a theme section, introduced by a Theme Editor and including three or four articles on the theme. School Libraries Worldwide primarily publishes new scholarly works, such as research reports and literature reviews. On occasion, due to the nature of the theme chosen, the Theme Editor may invite the submission of other types of papers including personal narratives, commentary and opinion" (IASL, n.d).
To capture a full decade of research in the field of school librarianship, we analyzed the most recent 10 years of articles, which include 139 articles from SLW and 86 articles from SLR. We collected full-text data in .pdf format and extracted titles and abstracts. As full-text data tend to contain much noise information, we delimited the analysis to only titles and abstracts, as is standard practice when using text mining in research domain analysis. Titles and abstracts are considered well organized summary information, and are the elements that well represent the key topics of the article content. Therefore, articles without explicit abstract were excluded from the analysis, such as editorials and interview dialogs. In total, titles and articles from 225 articles made up a dataset for this study.

Text analysis
For text analysis, we employed both term frequency analysis and topic modeling. First, term frequency was counted for each term observed from the collected documents. We extracted all terms from the dataset, and created a text corpus for analysis. The text corpus went through the standardized pre-process of text analysis, such as tokenization, stopwords elimination, and stemming. For stopwords, we used the R standard stopwords list suggested by the R tm package (https://cran.r-project.org/package=tm). Also, we added an additional list of stopwords that have little meaning in the analysis, such as "can", "also", "will", and "well". As to stemming, we applied the R SnowballC library, which implements Porter's word stemming algorithm to extract a common stem from the words with the same root (https://cran.r-project.org/package=SnowballC). Table 1 shows some examples of stemming results. After stemming, we tallied frequency for each word to create a term frequency table. Second, topic modeling was employed to extract topics from the text corpus. To be more specific, we used the LDA method which is an algorithm for discovering main topics that pervade a large unstructured collection of text files (Blei, 2012). The assumption of LDA is that a document can exhibit multiple topics and each topic can be represented as a distribution of multiple observed terms. Figure 1 shows the process of LDA topic modeling. Each document reveals a distribution of topic θ and emerges from a latent Dirichlet distribution, and a topic that comprises of terms φk is generated from a latent Dirichlet distribution with a prior of β (Lu & Wolfram, 2012). As shown in Figure 1, the process of LDA starts with sampling a document θd from Dir(α). Then, a word w in a document is allotted to a topic z selected from θd, and the word is selected according to z and φk 19 (where k = number of topics, Nd = number of words in a document). This procedure iterates until the model converges, and as a result, hidden topics will be extracted from the text collection (Blei, 2012).

Frequent Terms
First, we analysed term frequency. Overall, 2,297 unique words and 20,917 tokens are observed from the corpus. On average, 92.96 tokens are observed in each article. The rank-frequency pattern exhibits a typical Zipf's law pattern showing that top frequent terms make up a large portion of the corpus (Figure 2). The top ten terms account for 18.8% of the total tokens while the top 101 terms do for 47.3%.

Figure 2. Term-Rank pattern of the corpus
We counted frequency for each unique stemmed term to explore which terms were frequently adopted in school librarianship research. Table 2 lists 68 most frequent terms that were observed more than fifty times from the collected corpus. 20 The two most frequently observed terms are "school (4.98%)" and "librari (2.94%)", which are obviously the bases of the phrase, "school libraries". Then, "librarian", "student", "inform", "studi", "teacher", and "research" show more than 1% of the total observed tokens respectively. As the selected two journals are research oriented, research related terms occur frequently, such as "studi (1.57%)" and "research (1.33%)". Also, learning and education related terms are among frequent terms, such as "learn (0.99%)", "educ (0.83%)" and "instruct (0.48)". Another frequent word group is program related terms, such as "program (0.67%)", "develop (0.63%)", and "read (0.53%)".

Research Topics in SLW and SLR
LDA topic modeling was conducted to extract 20 topics from the entire corpus, as shown in Figure  3.

Figure 3. Topics Extracted from the LDA model
Topic 11 turned out to be observed most frequently (8.9%), which is about the impact of school library programing. Topic 1, showing 8%, is the second ranked topic, which is related to information literacy. The third most prevailing topic is Topic 18 (7.1%), which addresses the issue of teacher and librarian collaborations. This result implies that school library programing, information literacy and teacher/librarian collaboration are most widely discussed in school library research. Topic 15, which is related to cultural competency and the changing needs of education and youth, and Topic 19, which is focused on the professional roles of the school librarian are tied in ranking, accounting for 6.2% respectively. Topic 3 is focused on reading and reading motivation, Topics 5, 11, and 13 on resources and programming, and Topic 17 on evidence-based practice. Some topics are relevant to research design and method. For example, Topic 9 (5.8%) is closely relevant to "research design" while Topics 6 (3.6%) and 16 (2.2%) describe data collection and methods such as interview studies or online survey results. Topic 7 shows a topic of policy and management distinctively (4.9%), and Topic 10 discusses technology issues (5.8%).
Trends Over Time. Next, we analyzed the trends over time. By estimating key term occurrence probabilities for each topic, we identified three most "hot" topics and "cold" topics from the data. Figure 3 illustrates the results.

Figure 4. Hot topics and cold topics
As shown in Figure 4, Topic 19 (slope=0.32) related to professionalism, Topic 4 (slope=0.32) related to advocacy, and Topic 10 (slope=0.11) related to the Information Specialist role could be considered hot topics while Topic 3 (slope=-0.14) related to reading motivation, Topic 1 (slope=-0.21), information literacy and Topic 12 (-0.36), principal support, are cold topics. R square values turned out over 0.25 in all six cases. The result shows that there has been increased research on advocacy, the school librarian profession, and technology and digital leadership. On the contrary, there was declining attention on principal support, information literacy, and the school librarian's role in reading and reading motivation across the 10-year span.

Comparison between SLW and SLR
Finally, we compared prevalent topics between SLW and SLR. Twenty topics are extracted from each journal respectively. In this analysis, we compare the topics that account for more than 6% of each journal document set. Six topics are identified from SLW and four topics from SLR. Table  4 presents frequent topics, showing 6% or higher, from the two journals. In both the journals, school library programing turned out to be a frequent topic. In SLW, additional popular topics are: the roles and support of school librarians within the teaching profession, technology and digital leadership, learning community, and inquiry learning. In SLR, information literacy and reading are among popular topics.

Discussion
Using term frequency analysis and LDA topic modeling, this study explored the topics and trends of school library research based on content analysis of 225 journal titles and abstracts for the period between 2006 and 2015. In this study, we extracted 20 topics (k=20) from the entire set of 225 documents using LDA topic modeling. In addition, we analyzed the topic trend over the 10 years by estimating the probability of each topic. From the trend analysis, we identified the three hottest topics and three cold topics respectively. Moreover, we compared popular topics between SLW and SLR. We extracted 20 topics from each journal document set respectively, and identified frequent topics that show more than 6% of the documents in each journal set. In this way, we tried to investigate topical differences between the two journals.
What research topics emerged from school library research journals over the past 10 years? Term frequency analysis indicates a substantial recurrence of terms across the text corpus. Not surprisingly, frequently observed words include "school", "librari", "librarian", "student", "inform", and others, as anticipated. Term frequency analysis also points to the importance of learning, education and programing in school library research conducted across the past 10 years. LDA topic modeling returned 20 research topics underlying the 225 documents. The result of the topic modeling gives an overview of research topics that have been conducted in the school library area for the past ten years. Those twenty topics involve school library programing, information literacy, professional roles, digital and technology leadership, research design, policy and management, and others. Topic modeling analysis reaffirms that programing related research has been most widely conduced in the field. Also, it identifies information literacy and teacher and librarian collaboration are topics investigated frequently in school library research. As the selected journals are research oriented, research relevant topics are also observed from the results, including research design, data collection and analysis, and survey.
How have trends changed or developed across those 10 years? Five or 10-year spans are fairly standard for analyzing trends within education and librarianship (e.g. Goodwin & Goodwin, 1985;Luo & McKinney, 2015;Schram, 2014;Wells, Kolek, Williams, & Saunders, 2015). By choosing to use a 10-year span, we were able to observe some areas of growth and decline in research across time. For example, there is increased attention on the topics of professional roles and digital and technology leadership. On the contrary, there is slight decrease in the amount of research in the field focused on information literacy and reading.

24
What are the topical differences between SLW and SLR? Finally, are study identified differences in the topics covered in the two primary research journals in school librarianship. Both SLW and SLR frequently cover library programing related issues. However, there are some discrepancies in popular topics between the two journals. For example, professional role, technology, and inquiry skills are amongst popular topics in SLW, while information literacy, reading, and learning are more frequently researched in SLR. Even though both journals are venues for researchers, popular research topics show different patterns.

Conclusion
The study reported in this paper uses text mining, to investigate the topics addressed in school library research. While existing research points to trends in school library research using traditional content analysis methods, text analysis has not yet been applied to this field. Text analysis has the obvious benefit of enabling the collection and objective analysis of large sets of text documents. This study applies text mining, in particular LDA topic modeling, to understand research topics in the school library research community by analyzing the two leading research journals in the field: School Library Research (SLR) and School Libraries Worldwide (SLW).

Limitations
This study yields a methodological contribution to the field as it is one of the first attempts to apply text mining methods in exploring topics in the school library research field. However, there are limitations in the study. As there are not many research journals in the field, only two journals were selected as analysis objects in the study. Even though SLW and SLR are leading publication channels for school library researchers, 225 articles might not be able to represent the entire research area of the school library field, or the studies published in these two journals may differ from school library research published in other venues. Further, because one of the journals, SLW, is thematic, results are likely skewed. Also, this study is limited to topic issues, but did not investigate the relationships among key researchers in the field nor research methods.

Directions for Further Research
It would be a useful next step to explore topics by researcher to identify who are the key researchers in a particular subfield. More importantly, it is worth investigating trends in school library practitioners' journals, such as Knowledge Quest, School Library Connection, Teacher Librarian, and School Library Journal to determine the extent to which research informs practice, and researchers respond to the needs of practitioners. These limitations illustrate future research that includes both research and practice-oriented journals to compare the gaps between the two sides. Additionally, it is would be useful to analyze topics and authors together to give ideas of who are experts in a particular area in the field.

Text Mining as a Tool for School Library Research.
Text mining can be a useful tool in school library research in various topics. First of all, as shown in this study, topic modeling enables exploration of topics in school library publications, such as academic journals, practitioners' journals, and newsletters. As text mining can automatically produce secondary information from the original resources, it can give additional useful information to the school library community, for example, popular topics, issues, and summarized information in the community. Second, text mining can be used to analyze the content of resources that students use. Using text mining, we can further investigate what kinds of topical content are consumed by students when using libraries. Topic modeling and other text mining techniques allow researchers and practitioners to identify the content of resources students interact with in reading programs or other library activities. Text 25 mining will empower researchers to view different aspects of reading programs, for example, relationships between the reading content and its influences.

Collaboration, Collaboration, Collaboration.
While the results of this study are interesting and useful for researchers, we believe the true value is the methodological contribution to the field. Guidelines for school library programs and school librarians (AASL, 2009;American Library Association & AASL, 2010;National Board for Professional Teaching Standards, 2012;Schultz-Jones & Oberg, 2015) focus much more intently on school librarians' roles in information and traditional literacy instruction and promotion, instructional partnership, technology integration, leadership, and program administration than on research and research methods. Though evidence-based practice has received significant attention in the last decade (e.g. Cahill & Richey, 2012;Gordon, 2009;Mardis, 2011;Oberg, 2006;Richey & Cahill, 2014;Todd, 2008), the focus has been on collecting local evidence through means such as action research (Gordon, 2006) rather than a call for formal training in traditional research design and methods. Thus, most school library scholars have concentrated on fine-tuning their skills at teaching and researching these focal areas of school librarianship rather than developing their methodological expertise in quantitative research methods.
If school library scholars are more expert in qualitative methods as previous research has suggested (Clyde, 2002;Fitzgibbons & Callison, 1991;Grover & Fowler, 1993;Mardis, 2011), how might more quantitative methods be applied in the school library context? Through a process that school librarians have embraced for some time: collaboration! Only this time, the collaboration applies to school library scholars and the LIS and education research communities rather than school librarians and teachers. Forging partnerships with scholars in other disciplines would put school librarians on foot with the broader scientific community. In fact, a recent issue of Nature posited that in order to resolve the major challenges facing humanity, scholars must engage in interdisciplinary research (Ledford, 2015), and for more than a decade in the United States, the National Academy of Sciences in collaboration with the National Academy of Engineering, and the Institute of Medicine (2004) have been pushing researchers across disciplines to work together to stimulate innovation, advance general knowledge and tackle the larger problems of national and international importance that have gone unresolved using traditional discipline-specific methods and practices.
Just as school librarians the world over have touted the contributions of school libraries and school librarianship to student learning and teacher effectiveness, Mardis (2009) urged information and education scholars to recognize the importance of school libraries in LIS research. Perhaps through collaboration, school library scholars might resolve this major issue facing school librarianship.
28 user modeling, and large data analytics. He has also worked on text mining of social media data in public libraries.
Maria Cahill is an associate professor in the School of Information Science and the Department of Educational Leadership Studies at the University of Kentucky. Her research focuses on the services and programming of libraries and the role of children's and school librarians in supporting the literacy development of children and adolescents.


s2fieldsofstudy: Education Sociology

text:
Using Annotation for Transparent Inquiry (ATI) to Teach Qualitative Research Methods

In political science, qualitative analytic methods are rarely taught using “active learning” strategies. We discuss a novel approach to teaching such methods: having students engage with scholarship that has been annotated using Annotation for Transparent Inquiry (ATI). ATI allows authors to annotate passages in a digital publication to clarify methodology, add detail about evidence or analysis, or link to data sources. Learning methods through engagement with annotated articles allows students to interact with original data and to better understand and evaluate how authors collected, analyzed, and used those data. This leads students to learn research methods in a way that more closely approximates how they will use those methods in their own research. We present a general description of strategies for teaching with ATI. We illustrate the approach using three examples of instructors teaching both undergraduate and graduate students. We conclude with recommendations for effectively using ATI in the classroom.

Introduction
The early 2000s saw the beginning of a renaissance in qualitative research methods in the discipline of political science (e.g., Collier andBrady 2004, George andBennett 2005). This renaissance has included the development of more systematic and analytically explicit approaches to using qualitative evidence for descriptive and causal inference. Unfortunately, however, the teaching of qualitative research methods has not kept pace with their development. In particular, a recent study of the methods curriculum in 25 top political science doctoral programs between 2010 and 2015 found that qualitative methods instruction tends to take a passive rather than an active form: in marked contrast to the teaching of quantitative research methods, students rarely learn qualitative analytic methods by practicing their use on data generated and shared by other researchers (Emmons and Moravcsik 2019, 4). While not necessarily representative of the broader discipline, these findings suggest an important gap in the instruction of qualitative methods at the graduate level.
In this brief piece, we discuss a novel approach to teaching qualitative methods that employs more "active learning": through students engaging with scholarship that has been annotated using Annotation for Transparent Inquiry (ATI). ATI is a new approach to achieving transparency in qualitative and multi-method research that allows authors to annotate specific passages in a digital publication in order to clarify methodological challenges and choices, add detail about evidence or analysis, or link to data sources that underlie claims.
Learning methods through engagement with annotated articles allows students to interact with original data and to better understand, evaluate, and critique how authors collected, analyzed, and used those data to draw inferences and develop arguments. Compared with traditional pedagogical techniques, this approach thus leads students to learn research methods in a way that more closely approximates how they will use those methods in their own research.
Our discussion in this essay is somewhat speculative. To date, few of the teaching strategies that we propose have been used in practice, for reasons we discuss below.
Nonetheless, case studies we present in this piece demonstrate that initial efforts to employ these techniques have yielded promising pedagogical outcomes. These results, in tandem with the broad consensus in the education literature on the value of active-learning techniques, recommend continuing to develop ATI-oriented strategies. We hope this essay catalyzes further exploration and experimentation.
We begin by offering a few observations on qualitative methods instruction in political science, in particular elaborating on the infrequency with which methods for analyzing qualitative data are taught via active learning with data generated and shared by other researchers. We then briefly describe ATI and its potential to facilitate the active learning of qualitative analytic methods, and illustrate that potential using the experiences of three methods instructors. We conclude with a short discussion of some limits of this pedagogical approach.
The Status Quo and the Challenge Active learning -the use of "instructional activities involving students in doing things and thinking about what they are doing" (Bonwell and Eison 1991, 2) -is widely considered to be an effective pedagogical technique that facilitates skills acquisition and generates student interest (National Research Council, 2000;Hmelo-Silver & Rehak, 2017). While active learning techniques are often used to teach techniques for collecting qualitative data (e.g., archival research, interviewing, and ethnography), recent research suggests that it is unusual for students to learn qualitative analytic methods through practicing their use on data that other scholars generated and shared (Emmons and Moravcsik 2019, 4). 1 This marks a notable contrast with the instruction of quantitative methods, in which students often analyze shared datasets to complete exercises and problem sets, or seek to reproduce using computational methods the findings published in research articles with open data and materials (King 2006;Janz 2016).
Two factors likely contribute to this gap in qualitative methods instruction. First, in contrast with the dozens of textbooks treating quantitative analytic methods, the books most 1 Teaching qualitative analytic methods using shared data rather than data students generated themselves will often be preferable to having students analyze data they have generated themselves since the former approach allows the instructor to select data that are optimally suited for, or to tailor the data to, the particular analytic method being taught. Also, teaching with shared data allows students to learn collaboratively and/or compare inferences from the same data (see National Research Council 2000 on the benefits thereof).
often used in the instruction of qualitative methods tend to focus on the epistemological underpinnings and inferential logics of qualitative analytic methods rather than the practical steps involved in using them. Moreover, few such books include problem sets or other practical exercises. 2 Second, practicing analytic methods using data gathered and shared by other scholars is limited by the paucity of shared qualitative data. Relatedly, qualitative data sets specifically created for teaching are practically non-existent.
To date, then, few graduate students have had the opportunity to learn qualitative analytic methods by practicing their use on shared data. Conditions for employing this promising teaching practice are beginning to improve, however, as scholars are developing new strategies to share their qualitative data in ethical, legal, and meaningful ways (Kapiszewski and Karcher 2020). In the remainder of this piece, we suggest a set of pedagogical strategies that capitalize on these developments and are likely to enhance the instruction of qualitative analytic methods.

A Solution: Annotated Scholarship as a Teaching Tool
In order to make the instruction of qualitative analytic methods more active, we propose that instructors have their students engage with scholarship that has been annotated using ATI. 3 ATI, developed by the Qualitative Data Repository (QDR) and the software nonprofit Hypothesis, is an innovative approach to achieving transparency in qualitative and multi-method research. Authors who use ATI digitally annotate empirical claims or conclusions in their manuscript. ATI employs open web annotation, which allows for the generation, sharing, and discovery of digital annotations across the web. Annotations may include full citations to underlying data sources; "analytic notes" clarifying how the author generated or analyzed their data and/or how the data support inferences or interpretations; excerpts from data sources; and links to the data sources themselves when they can be shared ethically and legally. Annotations, in other words, surface the "analytic scaffolding" and 2 Notable exceptions include "SAGE ResearchMethods" (a subscription based online database with video tutorials, short courses, practice datasets, and case examples) and Cambridge University Press's "Method for Social Inquiry" book series; see also Collier (2011). 3 ATI is similar to "Active Citation" (Moravcsik 2010), an earlier approach to achieving transparency in qualitative inquiry; the two differ in that ATI leverages open annotation technology and emphasizes the importance of linking to underlying data. evidentiary underpinnings that are rarely included in published qualitative work due to space constraints. Annotations are displayed on the same web page as the digitally published article or book that they accompany; they and underlying data sources are curated and preserved by a data repository (see Figure 1). 4 Figure 1: ATI at a Glance: How a passage in the text of an article annotated with ATI appears to readers We highlight two overarching strategies that instructors of qualitative analytic methods might use to integrate scholarship that has been annotated with ATI into their teaching. 5 One strategy entails students examining the analytic scaffolding that ATI reveals; a second involves students engaging in re-analysis and analytic extensions. Both approaches 4 See also Karcher and Weber (2019) and the materials on QDR's website, qdr.syr.edu/ati. 5 A list of published ATI projects can be found at https://qdr.syr.edu/ati/ati-models. work best when the annotations in the scholarship used include links to a rich set of underlying qualitative data.
The first strategy can be thought of as standing in for the step-by-step introduction to qualitative analytic techniques that is often missing from textbooks. In this approach, instructors draw on annotations that elaborate on the analysis of qualitative data, in tandem with materials 6 and data shared by the author, to highlight and discuss with their students the logic of inquiry and specifics of analysis that underlie the exemplars. Students thus learn qualitative research methods by retracing and examining an author's analytical steps (and, potentially, mis-steps). For instance, students can see how authors identified and deployed causal process observations (CPOs) to advance narrative process tracing; learn about the mechanics of Bayesian process tracing from annotations that discuss how prior probabilities were assigned and posterior estimates progressively updated; or gain insight into the conceptualization and coding that underlie the process of calibration in Qualitative Comparative Analysis (QCA).
In the second pedagogical approach, instructors devise a set of structured exercises that call on students to actively engage with an author's analysis through the annotations provided in a publication. Such exercises are somewhat akin to the reproducibility exercises commonly used in quantitative methods instruction. For instance, students can reanalyze the supporting data provided in ATI annotations, using the same method the author used, in order to evaluate the strength of particular claims made in the text of a publication, or regenerate and validate its tables and figures.

Examples of Teaching with ATI
One of us (Jacobs) piloted an ATI-supported exercise using both types of strategies in a graduate qualitative methods class in the fall of 2019. Focusing on annotated portions of either Saunders (2011, Chapter 4) or O'Mahoney (2017), students selected individual CPOs for which the annotation provided either excerpts or reproductions of source materials. In the assignment writeup, students first retraced the author's steps, summarizing how the author sought to link the observation to an empirical implication of the article's or book's theoretical argument. Second, students critically evaluated the inferences the author drew from the evidence, as presented in or linked to from the annotation. Here the annotations allowed students to more thoroughly assess the original inferential reasoning, or provide more plausible alternative explanations, than they could have done from the article text alone.
Engaging with the fuller evidentiary record gave students insight into how authors had selected observations from their source materials and helped illuminate how the meaning of a statement or observation might depend on the context in which it appears. Moreover, reanalyzing the data presented in the article and found in the source material helped demonstrate the differing inferences that might be drawn from a given observation, depending in part on assumptions that the analyst makes and readings of the context in which the observation arises.
In a similar exercise used over multiple semesters in an upper-level undergraduate course focused on primary source-based inquiry, Robert Adcock (American University) asked students to assess the annotated evidence in the Saunders chapter just mentioned. Students drew on the shared source materials to assess the accuracy and precision of the summary or use of that source in the chapter's text, and the sufficiency of the evidence as support for the claim. 7 Students were also asked to think about whether, in aggregate (given their number, type, and authoritativeness), the sources could plausibly be considered representative of the broader set of potential sources. One factor students were asked to consider was the growing variety and number of primary sources available online, which had been discussed earlier in the course. The goal of the exercise was for students to acquire, via actively engaging with Saunders' meticulous use of primary sources, a working model of what they should aspire to in using and presenting primary source evidence as they drafted their own research papers in subsequent weeks of the course. 7 Personal correspondence, Robert Adcock to the authors, January 20-25, 2021, see also the description in Karcher (2016). Adcock's written instructions for this exercise are included in this article's supplementary material. Juan Masullo (Oxford University and Leiden University) has integrated ATI into his instruction of process tracing using both types of strategies described above. 8 In one exercise, 9 Masullo leads students to discover process tracing's architecture by asking them to read passages of an article annotated using ATI without the annotations before class, identifying evidentiary and analytic weaknesses, gaps, or silences; he and the students then re-read those passages with the annotations together in class, assessing how well the annotations address the pre-identified issues and contextualize and characterize the value of evidence. In another exercise, Masullo helps students to reveal the analytic architecture of their own empirical work by asking them to annotate it with ATI. In correspondence with us about this exercise, Masullo has emphasized the importance of guiding students to annotate with purpose, for instance, to justify citing evidence or discuss its probative value; to describe analytically relevant aspects of respondents' profile; or to make more explicit the logic of process tracing tests or Bayesian analysis. 10 One could imagine several extensions of these assignments in which students are encouraged to go beyond the analytic methods, data, and/or materials used by the authors of annotated work. For instance, students might be asked to re-analyze data shared in ATI annotations using a different analytic method from the one the author used to see if they arrive at the same results (e.g., they could apply formal Bayesian process tracing to an article that analyzed evidence informally). Even more ambitiously, instructors could ask students to search in the broader corpus of information to which the author linked in the annotations (or provided in addition to them) for evidence that corroborates or calls into question a publication's claims. To go one step further, students might practice reasoning with evidence by seeking to develop and answer research questions different from (but perhaps related to) those asked in an annotated publication using the corpus of information authors provide in their annotations or by triangulating between those and other data.
In sum, the growing availability of ATI-based publications enables more active forms of learning qualitative analytic methods. Active learning of these methods can generate better 8 In preparation, students read three pieces on process tracing tests and three on Bayesian approaches to process tracing. Masullo's description of his exercise is included in this article's supplementary material. 9 Masullo often uses his own work (e.g., Masullo (2020) for this exercise, which facilitates more in-depth discussion with students of the general logic of, and specific justification for, annotation. 10 Personal correspondence, Juan Masullo to the authors, January 21-24, 2021. understanding of how scholars draw descriptive and causal inferences from qualitative evidence, making students more sophisticated consumers of qualitative research. Moreover, active engagement with shared research data helps students learn to use qualitative methods by applying them.

Conclusion
In this article, we have described some general strategies for, and offered some examples of, introducing active learning elements into qualitative methods instruction using ATI. We discussed the experiences of three instructors who used ATI in the classroom; each found that doing so improved learning outcomes and all three continue to employ and refine their approach. As more articles annotated with ATI are published, instructors will be able to use them in teaching a broader range of analytic methods. 11 We note that there are similarities between the logic of teaching qualitative methods with ATI and the logic of teaching quantitative methods via reproduction exercises. The usefulness of reanalysis exercises using ATI, however, does not at all hinge on that parallel.
When authors use ATI, students can assess authors' findings by seeing if those findings can be reproduced from the data made available. Yet because most qualitative analysis involves some degree of subjective interpretation, different qualitative analysts might reasonably draw different inferences even when using the same methods to analyze the same data. In addition, some qualitative methodologies, particularly those operating within an interpretivist paradigm, eschew empiricist notions of researcher-independent objectivity. In these research traditions, reproducibility is itself incoherent as an evaluative standard. Even where analytical imprecision or epistemological premises make reproducing findings implausible or meaningless, however, students are likely to arrive at a better understanding of the logic and practice of different methods if they have the opportunity to apply them to sources or evidence underlying a publication, and compare their reasoning and findings to those presented by the original authors.
Even within a replicationist paradigm, learning qualitative methods with ATI has limits. Even when authors provide or link to sources or data in their annotations, annotated articles rarely offer visibility into how authors selected those sources or data from all the information they encountered or collected. 12 Further, not all work annotated with ATI is equally suitable for teaching. The initial experiences of teaching with ATI described here suggest that annotated publications that provide at least some associated primary material will be most useful pedagogically. Also, annotations that effectively lift the curtain on the research process --e.g., by discussing the evidentiary value of data cited in the text, commenting on how to adjudicate between contradictory evidence, and highlighting choices made during data collection -can be particularly enlightening for students.
Finally and more broadly, although the initial experiences recounted here are encouraging, we have limited evidence to date on the efficacy of using ATI to teach qualitative methods. We hope that this article encourages more instructors of qualitative research methods to bring ATI into their classrooms so that together we can continue to develop, assess, and capitalize on its pedagogical promise.


Parental Influence on Students’ Career Choice and its Effect on Their Academic Performance. A Case of Schools in Rulindo District

The goal of this research was to determine the degree to which parental influences impact job choice and performance among advanced level students in Rulindo District, Rwanda. Additionally, the research intended to identify the interaction between children and parents in terms of profession choice and the value of collaboration. The research population consisted of all 2000 pupils enrolled in Rulindo's three elementary schools. The research surveyed a total of 108 pupils, and they all answered. Providing a response rate of 100%. Purposive sampling was employed to sample three schools in Rulindo District, while simple random sampling was utilized to sample the pupils. Questionnaires were employed to gather data. The data were analyzed using the Statistical Package for Social Sciences (SPSS) program. The data were analyzed using descriptive statistics such as frequencies and percentages. Tables and graphs were used to show the findings. The study's findings indicated that parental variables had an effect on students' profession choices in Rulindo. These variables included the parents' greatest level of education, their employment, their beliefs and expectations, and their parentchild connections. The research advised that parents and children discuss higher level courses before to enrolling in them, emphasizing on the learner's strengths and preferences in order to minimize potential difficulties. This enables students to make informed job choices based on their educational attainment and professional goals.

Introduction
Parents are often said to be a child's first and finest instructors. The research corroborates this assumption, demonstrating that parents' effect on their children's education starts as early as preschool. Parental influence extends beyond the parent-child bond to parental qualities, education, and money, as well as parental interactions with their children and their children's schools (Zhao et al., 2017;Lu, 2012). Parental perspective is critical in many aspects of a child's life, including academic, professional, and personal. Parental guidance is critical in a child's life and has a substantial impact on two important life events: marriage and career choice (Hasib et al., 2021). The impact of parents on their children's professional choices is widely recognized, regardless of gender or race.
They forfeit even opportunities to perform what they want, so that the task for which they have been picked provides them no pleasure. However, little studies have been conducted to date on the issue of parental impact on children's choices and educational influences in a variety of nations. While a recent research examined the impact of parental influence on kids' job choices and academic achievement. By highlighting the detrimental consequences on educational growth, particularly in the Rulindo District's schools.

Statement of the Problem
Parents have a significant effect on their children's job development and professional choices. Parents want their children to be happy and successful in life, and one aspect that contributes to happiness and success is profession selection. Additionally, research suggests that when adolescents feel supported and loved by their parents, they develop a greater confidence in their own capacity to do job study and pick an intriguing and exciting vocation. This is important because research indicates that teenagers who feel competent in terms of profession choice make more satisfied job choices later in life Clausen, 1991).
Though parents have a critical influence in their children's job choices, parents might have a detrimental impact on their children's choice of Advanced level courses in Rwanda's present education system. This sometimes happens when a youngster chooses a professional path based on his or her ambitions and abilities, such as Maths-Economics and Geography (MEG), but is persuaded by parents to pursue another path, such as Maths-Chemistry and Biology (MCB) or Physics-Chemistry and Mathematics (PCM). They are often persuaded to choose alternatives or combinations within the Rwandan educational system that are contrary to their aspirations and interests by their parents, who want them to pursue careers as engineers, doctors, or other professionals. Regardless of whether kids are capable of doing better or not.
As a result, some students in Advanced levels, which are equivalent to senior four, five, and six, face numerous challenges throughout their education, including failures, stress, poor performance in certain subjects, a lack of interest and motivation as a result of having their first choice (career) rejected and being forced to pursue other combinations. Parents should be warned against putting their own objectives on their children or seeing their child's success as a mirror of their own (Alderson, 2007). While parents should demonstrate real interest in and support for their children's job choices, they must also enable youngsters to find their true selves on their own.
Clearly, the entire study's objective was to ascertain parental influence on students' career choices and its effect on academic performance, as well as to determine whether parents' influence and guidance in the area of career choice and vocational development provide students with support and encouragement to explore the numerous options available to them in order to find the best career that fits their unique combination of skills and interests.

Research Objectives
This dissertation has several objectives to put across but the main ones are presented as follows: (1) To find out the parental influence on students' career choice and its effect on their academic performance in Rulindo secondary schools; (2) To assess the challenges that students face in their daily learning activities due to the rejection of their wills and preferences; (3) To evaluate how parents can listen to their children in terms of career choices; (4) To investigate how important it is giving children the freedom to choose plays a significant role in their academic success; (5) To formulate suggestions on what should be done to prevent the challenges that students face in career choices.

Theoretical Literature
Parents with varying demographic and psychographic features have varying expectations for their children's schooling, which ultimately influences their children's job choices. The prevailing perception of parents towards their children's education is that it should be used to get more respect and money rather than to effect good change in society.

Parental Influence
Parental influence is described as any view, attitude, or action on the part of the parent (other than direct tutoring) that moulds or molds the child's reading attitudes in some way. Involvement is defined as any direct tutorial assistance a youngster gets in order to improve his or her reading ability.
A career is distinguished from a job in the following ways: A job is something you perform only for the purpose of earning money; a career is a linked sequence of work chances. While a job has little bearing on your future work life, a career gives experience and education that will power your future.

Career Choices and Development Theories
Professional choice is the process of determining an appropriate career path. A professional decision is critical since it impacts your performance in subsequent phases of life (Murphy et al., 2018). Individual growth, organizational growth, economic growth, and waste reduction are all critical components of job choice.

Holland's Theory of Vocational Personalities in Work Environment
Holland's career choice theory was one of the most thoroughly explored hypotheses, arguing that congruence between one's fundamental personality type and chosen job was a critical component in achieving professional success. His hypothesis was often utilized as the foundation for career counseling sessions by experts. According to this theory, the majority of individuals fall into one of six fundamental personality types: Realistic, Investigative, Artistic, Social, Enterprising, or Conventional. These are often referred to as Holland Codes or RIASEC.
Theorists argued that individuals who worked in an environment that matched their personality attribute were more prosperous and satisfied.

Factors Influencing Career Choices
When an individual considers his or her future vocation and job choice, various considerations come into play. The most critical factors are the job description, the necessary skills and education, the compensation, and the career prospect.
However, there are other other significant elements that might impact a person's profession decision. The researcher discusses some of these elements in this study. It is important to consider them before making a choice on one's future job. Everyone has to make the appropriate option, consider the advantages and disadvantages, and this will assist him or her in finding the ideal vocation.

Critical Review and Research Gap Identification
Numerous studies undertaken by various scholars examined the effects of parents on their children's job growth and choice. Keller asserts that parents have a significant effect on their children's professional development and decision-making. Parents want their children to be happy and successful in life, and one aspect that contributes to happiness and success is profession selection.
Additionally, research suggests that when adolescents feel supported and loved by their parents, they develop a greater confidence in their own capacity to do job study and pick an intriguing and exciting vocation. This is important because research indicates that teenagers who feel competent in terms of profession choice make more satisfied job choices later in life ).
Parents are often said to be a child's first and finest instructors. The research corroborates this assumption, demonstrating that parents' effect on their children's education starts as early as preschool. Parental influence extends beyond the parent-child bond to parental qualities, education, and money, as well as parental interactions with their children and their children's schools.
According to academics Kathleen Cotton and Karen Reed Wikelund, "parental participation in children's learning is favorably associated to achievement" for all kinds and ages of students, and the greater the intensity of this participation, the more beneficial the benefits.
Career is defined by Redman and Wilkinson (2001) as the application of a person's cognition and capacities, offering mastery over a profession, timely job knowledge, and a foundation for creating and improving business networks. Individuals undertake career planning in order to accomplish professional goals by staying informed about impending possibilities, their outcomes, and their timely reviews. It is regarded a starting point in the phases of professional selections but is still critical. Individuals want careers that may give a solid foundation for a

Research Design
As stated before, the purpose of this study was to investigate parental impact on students' job choices and its impact on academic achievement in Rulindo District. The researcher accomplished this by collecting both primary and secondary data. Primary data were collected through questionnaires, interviews, and observations completed by respondents from chosen schools in Rulindo District, while secondary data were gathered from printed sources such as books, journals, reports, archives, and also electronic sites linked to the issue.

Target Population And Sampling Procedures
The term "population" refers to a group of individuals or any other collection of objects considered for the purpose of study. According to Grinnell and Williams (1990:136), a population is defined as the whole of individuals or things included in a research. The population consists of Advanced level students (Senior four, Senior five, and Senior six) from chosen schools, as well as some parents. Carl and Roger (1991) said that a population research is concerned with the overall group of individuals from whom the forms are derived.

Sampling Procedures
A sample is a subset of the population that has been purposefully chosen for the aim of examining the target population's features (Kainada, 2008). We worked with a sample size of 108 pupils, including parents, in this example of Rulindo District.

Results and Discussion
This study examined the impact of parental influence on students' profession choices and academic performance in a case study of schools in Rwanda's Rulindo District. Chapter 1 introduced the study's context, which includes global, regional, and local views on parental influence and its impact on students' career choices. The chapter addressed the research issue statement, emphasizing the goal, goals, and research questions that guided the overall body of study. Additionally, the study's scope and importance were determined. Chapter 2 established a correlation between parental influence on a kid's job and the impact on the youngster. The theoretical review, empirical review, and conceptual framework all aided in establishing this connection. The theoretical examination uncovered flaws, gaps, urgent relevance, and contributions. The empirical review included scientific results about the relationship between parents' professional choices and their children's academic success. The conceptual framework highlighted the characteristics of the two variables in the study, and the analysis of their connection contributed in achieving the study's stated research aims and offered evidence that professional barriers may be overcome.
Chapter 3 discussed the study's approach. The methodology included the study's design, population, sampling techniques and equipment, as well as data processing and analysis. This chapter also discussed limitations and ethical issues. The research used the field survey technique to examine data obtained from a 2000-person study population and a 108respondent sample size estimated using the Bouchard formula. As a result, 108 legitimate responders were obtained. As a result, the study's findings were compiled using both respondents' perspectives and data from key informants. This methodology made a significant contribution to the research's validity.
Chapter 4 contains the presentation of the data analysis and interpretation. Secondary schools in Rulindo District served as a baseline for empirical evaluation. Respondents' perspectives were studied, as were the data supplied by significant respondents, and conclusions drawn on parental influence on students' careers and its impact on academic success. In Chapter 4, the respondents' perspectives align with the study questions and aims. While addressing observations, solutions were offered to reduce difficulties. This chapter provided the researcher with further insight into the impact of parental influence on students' careers and academic achievement. As a result, chapter 5 would summarize the study's results, conclusions, and recommendations, as illustrated below.

Conlusion
The initial purpose was to determine the effect of parental influence on profession choices. The researcher posed questions and the following findings were obtained: Respondents' perspectives on how parents affect their children's choice of combination: Not sure replies accounted for 2.8 percent, disputed answers accounted for 4.6 percent, strongly disagreed answers accounted for 4.6 percent, and strongly agree answers accounted for 26.9 percent. Those who agreed with the allegation accounted for 61.1 percent. Only 61% agreed that their parents affect their choices. The second description of the study goal, which was to determine if students suffer difficulties when forced to pick a combination different from their first selection. The responses indicate that both strongly agree and agree cover 70% of the results from the questionnaire given to learners, indicating that once parents influence students to choose a combination other than their first choice, many students perform poorly in those combinations, despite the fact that some students cope and manage to keep up with them and earn good grades. The third purpose was to determine the extent to which allowing children to make their own choices contributes to their academic achievement. Participants were asked a series of questions and agreed at an 87 percent rate that their views must be heard. Finally, several ideas were made on how profession choices may be improved.

Recommendations
It has been established that parents have an influence on the career choice of students' combinations of studies. In light of this, it is recommended to do the following: (1) Parents' roles are important in education but they need to strongly hear the children upon their choices; (2) Parents should not force students to pursue careers similar to their own against their will. This is because majority of the students indicated they are affected by this negatively; (3) The ministry of education should strengthen the career counselling services in schools; (4) Schools should also develop the career counselling programs that enables learners to grow with career decisions before the completion of ordinary level; (5) Parents should realize that their values and expectations influence the career choice to a great extent. In this respect it is recommended that parents should deliberately communicate their expectations to their children without being overly persuasive; (6)Finally students should also play a big role in showing what they are able to do and why they want to pursue this or that combination.