The Effectiveness of Library Instruction for Graduate/Professional Students: A Systematic Review and Meta-Analysis

Objective This study sought to assess the effectiveness of library instruction for increasing information literacy skills and/or knowledge among graduate and professional students. Methods A search was conducted in Library Literature and Information Science Index (H. W. Wilson); Library, Information Science & Technology Abstracts; Medline; CINAHL; ERIC; Library and Information Science Abstracts (LISA); and ProQuest Dissertations and Theses Global. Studies were included if they were published between 2000 and 2019, in English, reported on library instruction for graduate or professional students, and objectively measured change in information literacy knowledge/skills. Evidence Based Library and Information Practice 2020, 15.2 101 Results Sixteen studies were included in the systematic review; 12 of the 16 studies included sufficient information to be included in the meta-analysis. The overall effect of library instruction was significant [SMD = 1.03, SE=0.19, z=5.49, P<.0001, 95% CI=0.66-1.40], meaning that on average, a student scored about one standard deviation higher on an information literacy assessment after library instruction. High heterogeneity indicated a need for subgroup analysis, which showed a significant moderation of effect by discipline of students, but none by format of instruction. However, subgroup analysis must be viewed with caution due to the small number of studies in several of the subgroups. Conclusions This meta-analysis indicates that library instruction for graduate students is effective in increasing information literacy knowledge and/or skills. However, to strengthen the accuracy of results of future meta-analyses, there is a need for more precise descriptions of instructional sessions as well as more complete data reporting by authors of primary studies. There is also a need for the publication of more studies, particularly studies of hybrid and online instruction.


Introduction
Regional accrediting standards for colleges and universities emphasize the need for institutions to engage in effective assessment of desired student learning outcomes to substantiate results (Baker, 2002). One common learning outcome for university students is the ability to locate, evaluate, and manage information (i.e., to be information literate) (Markle, Brenneman, Jackson, Burrus, & Robbins, 2013). Although information literacy (IL) instruction should be interwoven throughout the curriculum, most academic librarians are invested in collaborating with subject faculty to provide library specific instruction to improve the IL skills of students (McGowan, Gonzalez, & Stanny, 2016) and are interested in assessing the value of that instruction. Library instruction to improve IL is often seen as essential only for undergraduates (Blummer, 2009). However, students in graduate/professional studies do not always have the requisite skills needed for graduate level study and research (Conway, 2011), which suggests they may also benefit from library instruction targeted specifically to graduate students. For example, O'Clair (2013) found that graduate students felt more prepared to tackle thesis research after taking a for-credit information literacy course.

Aims
This study includes both a systematic review and meta-analysis. The systematic review examines the current state of library instruction for graduate students and seeks to determine what formats of instruction are used, the content of instructional sessions, and how instruction is assessed. One issue with assessment of library instruction is that small sample sizes may limit the ability to identify actual change (Coe, 2002;Higgins, 2019). Meta-analysis is one way to combine the results of multiple studies to improve the statistical power and lessen the possibility of failing to identify a true difference (Shinogle, 2012;Thornton & Lee, 2000). Although a meta-analysis has been completed on the effectiveness of library instruction for undergraduates (Koufogiannakis & Wiebe, 2006), none was found for graduate/professional students. This study looks at the effectiveness of library instruction for graduate and/or professional students and if that effectiveness varies by discipline, format, or duration of instruction. Specific research questions include:

Systematic Review
• What formats are used to provide library instruction to graduate/professional students? • What content is covered in instruction sessions for graduate/professional students? • How is instruction for graduate/professional students assessed?

Meta-Analysis
• Does library instruction for graduate/professional students result in improved information literacy knowledge and/or skill? • Does effectiveness of library instruction for graduate/professional students vary by format, duration, or discipline?

Methods
This study was conducted using the guidelines established in the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) statement (Moher et al., 2009). Concepts for library instruction, graduate/professional students, and study type along with synonyms and subject headings were searched on 11 March 2019 in Library Literature and Information Science Index (H. W. Wilson); Library, Information Science & Technology Abstracts; Medline; Cumulative Index of Nursing and Allied Health Literature (CINAHL); Education Resources Information Center (ERIC); Library and Information Science Abstracts (LISA); and ProQuest Dissertations and Theses Global (see Appendix A for search strategies). Academic libraries underwent significant changes in the late 1990's with the advent of personal computers and electronic access to journal articles. Since those changes also affected library instruction, searches were limited to a date range of 2000 to 2019. Literature searches were also limited to English language, but no restrictions were placed on type of publication.

Eligibility Criteria
To be eligible for inclusion in this review, studies had to include instruction for graduate or professional students related to information literacy (IL) knowledge and/or skills. The instruction had to be provided wholly or in part by one or more librarians, and studies had to include a measure of change in IL knowledge/skills. Finally, studies had to include either one or more groups with a pre-and postmeasure of IL knowledge/skills or both a treatment and control group with a postassessment of IL knowledge/skills. Graduate students included students studying for a master's or PhD in any subject area (other than library science), while professional students included any health science student working on a clinical doctorate, including medical, dental, pharmacy, veterinary, nursing, and audiology students. Synthesis studies, studies written in a language other than English, and studies involving medical residents or library science students were excluded. Additionally, studies were excluded if the measure of change in IL skills/knowledge was self-reported by students.

Study Selection
The number of studies examined at each stage of the review process are shown in Figure 1. Both authors independently examined each source, first at the title and abstract stage, then later at the full text stage. After each screening level, the authors compared individual decisions for congruence; conflicting decisions were resolved by discussion.

Data Extraction
Each author extracted data from half the studies to an Excel spreadsheet, and then checked data extracted by the other author for accuracy and completeness. Data collected included information about participants (level of study, Figure 1 PRISMA flow diagram (Moher, Liberati, Tetzlaff, & Altman, 2009) discipline, and geographic location), the intervention (description, duration, format, content taught, and content assessed), the assessment/test (timing, validity, and availability), and study statistics (sample size, mean, and standard deviation). Some studies did not include standard deviation but did provide individual scores. In those cases, standard deviations were calculated using Excel. Seven authors were emailed for additional data, and three replied with the requested information.

Quality Assessment
Quality of each included study was assessed using an instrument developed to critically appraise educational interventions (Morrison, Sullivan, Murray, & Jolly, 1999). The checklist includes nine questions addressing content, context, outcomes, study design, and methods. Both authors independently answered the nine questions for each study with 'yes', 'no', or 'can't tell' and then met to compare results. Differences in answers to individual questions were settled by discussion and reference back to the article. The authors then voted to include or exclude the article based on preponderance of 'yes' answers with more weight given to questions 5 and 6. Those two questions addressed whether the study design was able to answer the posed question and whether the methods used were appropriately measuring the phenomena of interest. All articles except for one received 'yes' answers for both question 5 and 6 from both authors. That article received 'no' to both questions from both authors and was discarded due to quality concerns.

Data Synthesis
Analysis was carried out with R [version 3.5.0 (23 April 2018)] (R Core Team, 2018) using the metafor package (Viechtbauer, 2010) (see Appendix B for data). Standardized mean difference (SMD) was the chosen effect size. SMD represents the difference in the pre-and post-intervention means divided by the pooled standard deviation (Borenstein, Hedges, Higgins, & Rothstein, 2009). When a study includes a small number of samples, SMD may be biased; therefore, SMD with a correction factor (Hedge's g) was used (Borenstein et al., 2009). Hedge's g is computed in R using the SMDH (standardized mean difference with heteroscedastic population variances in the two groups) function; SMDH requires input of the sample size, pre-and post-mean (M) scores, and pre-and post-standard deviations (SD) (see Appendix C for sample R code). When SD was not provided and could not be calculated from available data, an estimate based on the average SD of all other studies was used (Furukawa, Barbui, Cipriani, Brambilla, & Watanabe, 2006). All meta-analyses were conducted using a random-effects model, which assumes that there is not one "true" value that all studies are seeking, but instead that values may vary among studies due to differences in how the studies are carried out (Bown & Sutton, 2010). A random-effects model is recommended when there is assumed to be heterogeneity in outcome estimates (Bown & Sutton, 2010), which was the case in this meta-analysis. The I 2 statistic was used to quantify the heterogeneity of effect sizes. I 2 ranges from 0 to 100%; Borenstein et al. (2009) suggest that a small I 2 (close to 0) indicates that only a small part of the observed variance reflects actual differences in effect size. However, larger numbers indicate a larger proportion of the observed variance is real and suggest a need to carry out subgroup analysis or meta-regression in order to explain the heterogeneity (Borenstein et al., 2009).

Publication Bias
Publication bias can arise through several means, for example, authors may decide not to report non-significant findings, or journals may refuse to publish negative studies. Since metaanalysis depends on finding all studies that answer a specific research question, publication bias has the potential to distort findings (Song, Hooper, & Loke, 2013;Thornton & Lee, 2000). This study followed the recommendations of Song et al. (2013), using a comprehensive search that did not limit results to only journal articles. In addition, publication bias was assessed with both a funnel plot and through use of Rosenthal's fail-safe number. The fail-safe number is an estimate of the number of nonsignificant studies required to nullify the results of the meta-analysis; Rosenthal (1979) suggests a fail-safe number greater than 5n + 10 (where n is the number of studies) is sufficient to consider publication bias inconsequential (see Appendix C for sample code to calculate Rosenthal's failsafe number in R).

Description of Studies
The final 16 studies included one dissertation and 15 journal articles, with publication years ranging from 2004 to 2018 (see Appendix D for list of studies and Table 1 for characteristics of studies). The majority of studies took place in the United States (n=12), but there was one study each from Canada, the United Kingdom, Australia, and Tanzania. A variety of disciplines were represented (see Figure 2), with the largest number of studies including medical students (n=4) followed closely by students in education (n=3). Overall, health science students were included more often, with 10 of 16 studies involving students from some area of health sciences. The 16 studies included 12 studies that were pre-and post-assessments of one or more groups (repeated measures); the remaining four were post-assessment of a treatment and control group (independent groups). Sample size of the repeated measures (RM) studies ranged from 10 to 61 students, while sample size of the independent group (IG) studies ranged from 37 to 300 students. The most common format of instruction was face-to-face (F2F). Six studies included only F2F instruction, five included only hybrid instruction (a combination of face-to-face and some sort of online instruction), and two studies examined only online instruction. An additional two studies compared F2F to online, and one study compared all three formats--F2F, hybrid, and online. Duration of instruction was not reported for every study. Durations that were reported varied widely; for example, for library instruction provided within a subject class, time of instruction ranged from one 70minute session to two 3-hour sessions (see Table  2).

Content of instruction
While not every study included a detailed description of instructional content, certain themes emerged in the studies (see Table 1). All classes taught database searching strategies (n=16). The classes for health sciences students (n=10) provided instruction on biomedical databases (PubMed, International Pharmaceutical Abstracts, MEDLINE, CINAHL, and others) while classes for non-health science students (n=6) taught a variety of resources including chemistry and education databases and sources of data from United States government agencies. Search strategies taught included Boolean logic (n=6), limiters (n=5), and MeSH (Medical Subject Headings) vocabulary (n=5). Other topics included critical appraisal skills (n=6), citation styles and citation managers (n=4), ethical use of information (including plagiarism) (n=3), and library-specific resources and services (n=4).

Assessment
Six of the 16 studies employed a validated assessment tool (see Table 3). Of those six, two used an instrument based on the Fresno test (Ramos, Schafer, & Tracz, 2003), one used the RRSA (Research Readiness Self-Assessment) (Ivanitskaya, Laus, & Casey, 2004), and two studies used rubrics validated in-house. • An on-campus class with face-to-face library instruction. • 70-minute demo followed by application activity.
Principles of library and information research, navigation, and search techniques, practical application of search techniques, locating, evaluating, and citing information.
Assessed: Conceptual knowledge (how information is produced and organized), knowledge of database-searching skills (identifying databases and using Boolean logic), knowledge of institution-specific information (accessing databases and awareness of services). • Library instruction during a class session at the beginning of the course. • Class sessions were from 5:30 to 10 pm but amount of time given to library instruction was not specified.
Taught: Search strategies (keywords, subject headings, and Boolean operators), how to find journal articles, identifying and searching for scholarly journals, searching for articles using the appropriate journal database for the topic, refining the search, evaluating the article, and downloading or ordering the fulltext of the article.
Assessed: Ability to find information, ability to evaluate information, and understanding of plagiarism.

Content Taught Content Assessed
Assessed: Defining a problem or research topic; information sources; internet sources; internet search; library and database searching; evaluating information and sources; referencing; synthesizing information; presenting information.

Figure 2
Number of studies by discipline. Most studies (n=9) used objective tests such as multiple choice and true/false questions to measure recall of knowledge, while five studies measured the application of knowledge by evaluating search strategies or scenario responses. Two studies measured the application of knowledge through short answer and multiple-choice questions that required hands-on use of databases.

Meta-Analysis
Meta-analysis often involves examination of experimental studies involving independent groups (IG), for example treatment and control groups; however, meta-analysis is also possible with repeated measures designs (RM). RM studies involve one or more groups; individuals within the groups are assessed both before and after an intervention. These two types of studies differ in the type of research question involved, with IGs interested in group differences while RMs explore change at the individual level (Morris & DeShon, 2002). Morris and DeShon (2002) point out that combining IG and RM studies may be done, but only if effect sizes are transformed to account for differences in how standard deviations are calculated. The IG studies found in this systematic review did not include the information required to transform the effect sizes to equivalent RM effect sizes as recommended (Morris and DeShon, 2002). In addition, the small number of IG studies was considered insufficient to complete a separate meta-analysis, therefore only the RM studies (pre-and post-assessment of one or more groups) were included in the meta-analyses. When an RM study included multiple groups, for example, a comparison of online versus faceto-face instruction, each group was considered separately in the meta-analysis. Therefore, for the 12 RM studies there were 16 associated effect sizes. Nine of the RM effect sizes involved faceto-face instruction (F2F) by a librarian, three were online modules only, and four were hybrid sessions, involving F2F instruction supplemented with online modules.

Effectiveness of library instruction for graduate students
A meta-analysis run on all RM groups (16 effect sizes from 12 studies) produced an overall standardized mean difference (SMD) of 1.03 [SE=0.19, z=5.49, P<.0001, 95% CI=0.66-1.40], (see Figure 3), which is considered a large effect size (Cohen, 1988). Another way to state the result is that graduate students scored slightly more than one standard deviation higher on a measure of IL skills after receiving library instruction. The I 2 statistic for the meta-analysis was 81.47%, indicating a large amount of heterogeneity (Higgins, Thompson, Deeks, & Altman, 2003) and a need for subgroup analysis. One possibility is that results could have been influenced by the estimation of standard deviation (SD) for two studies (3 associated effect sizes) (Appendix B); however, a test of estimation as a moderator revealed no significant difference between studies with estimated variables and those without (QM (df=1) =0.24, P=.63).

Effectiveness by format, duration, or discipline
Sub-group analysis was completed in an attempt to explain the large amount of heterogeneity in the overall meta-analysis. Format of instruction Scores for treatment group were significantly greater than the control group.
Pre-and post-clerkship survey (self-report); postclerkship NNT (Number Needed to Treat) test (self-report); analysis of articles identified as best evidence. Schweikhard, 2018 The impact of library tutorials… a) Final course papers scored by 2 independent reviewers using a rubric. b) End of course. c) Application. a) Yes, two reviewers for each paper. Reviewers practiced scoring papers not selected for the assessment sample to support interrater reliability. b) Scoring rubric created for this study.

Figure 3
Forest plot of meta-analysis of repeated measures studies.
as a moderator was considered first by comparing the three types of format: face-to-face (nine effect sizes), online (three effect sizes), and hybrid (four effect sizes). Results of the analysis were not significant (QM (df=2) =0.77, P=.68) indicating there was only random variation in effect sizes between format types.
Discipline of students was also considered as a potential moderator. Other than medicine and education, no discipline had more than two associated studies, and many had only one (see Figure 2). However, studies were almost equally divided between those involving health science students (seven effect sizes from six studies) and non-health science students (nine effect sizes from six studies), so those two groups were compared. There was a significant difference in effect size based on discipline as a moderator  For example, for sessions provided within a subject class, duration ranged from one 70minute session to two 3-hour sessions. The small number of studies in each duration length precluded completing a subgroup analysis of duration.

Publication bias
The funnel plot for the meta-analysis of all RM studies is shown in Figure 4. Studies seem to be evenly distributed at the top of the funnel but lacking toward the bottom. However, the failsafe number was calculated to be 750, considerably larger than the minimum of 90 suggested by Rosenthal (5n + 10 =5(16) +10).
Since 750 non-significant studies would be required to reduce the overall effect size to zero, publication bias was not considered an issue.

Discussion
There was a positive overall SMD, which suggests that library instruction does increase information literacy knowledge and/or skills in graduate students, and that the average increase in score is about one standard deviation. Although this appears to be the first systematic review and meta-analysis involving library instruction for graduate students, there is a previous meta-analysis of library instruction for undergraduates, which found similar results (Koufogiannakis & Wiebe, 2006). Like this study, Koufogiannakis and Wiebe (2006) found a positive effect when comparing library instruction to no instruction, but the effect was much smaller, about one-third of a standard deviation (SMD=0.36, 95% CI=0.14-0.50). The smaller effect may be explained by the fact that Koufogiannakis and Wiebe (2006) (2006) also compared traditional instruction to computeraided (online) instruction, and like this study, found no difference in the effectiveness of the two formats. However, findings about hybrid instruction from this study differ from those of another meta-analysis of blended (hybrid) learning in health professions (Liu et al., 2016). While Liu et al. (2016) concluded that blended (hybrid) instruction was more effective than non-blended instruction (SMD=0.81, 95% CI= 0.57-1.05), this study found that there was no statistically significant difference between effect sizes for different formats of instruction, including hybrid, face-to-face, and online. One difference between the two studies is sample size; the small number of studies involving hybrid instruction in this meta-analysis limits the robustness of those results.
Small numbers of studies also impacted the ability to look at effect of instruction by discipline of students. Two broad categories (health science students and non-health science students) were examined rather than individual disciplines. Findings indicated a significant difference in effect size between instruction for health science and non-health science students, with library instruction for health science students slightly less effective (average increase of about two-thirds of a standard deviation) than library instruction for non-health science students (average increase of almost 1.5 standard deviations). This result may be explained in part by the likelihood that assessing the ability to apply knowledge results in smaller changes than simply testing students' recall of information. More than 40% of the studies that included health science students assessed application of knowledge. In contrast, the studies involving non-health students all assessed recall of knowledge, although two of them did also include a few questions that required students to apply knowledge in order to answer multiple-choice questions.

Limitations
One limitation for the overall meta-analysis was the lack of required information from studies, resulting in the need to contact authors and if that failed, to estimate standard deviation for some studies. As pointed out by Gerstner et al. (2017), effective meta-analyses rely on complete data reporting in primary studies. To ensure more complete and accurate meta-analysis of results, studies reporting educational interventions with pre-and post-assessments should either include pre-and post-means and standard deviations or provide raw data so that those statistics can be calculated. A second limitation in the subgroup analyses was the small number of studies in some categories. Duration of intervention, which might be expected to affect effectiveness, was not considered for subgroup analysis because of the lack of information in some studies and lack of uniformity of duration in the remaining studies. In addition, when examining format of instruction, there were only three studies involving online instruction and four with hybrid instruction. Borenstein et al. (2009) point out that in a random effects model, small numbers of studies make it more difficult to estimate error and increase the possibility of not only an inaccurate effect, but also an inaccurate range of effect. Therefore, results of any subgroup analysis with a small number of studies must be regarded with caution. Small numbers of studies may have also affected subgroup analysis of instruction by discipline since no discipline had more than four studies, and studies had to be combined into much broader categories of health science and nonhealth science students.

Implications for Practice
• Library instruction for graduate students seems to be effective in increasing students' knowledge and skills. • There was no significant difference in effectiveness of face-to-face, online, or hybrid formats of instruction. • Content varied but information about searching effectively was present in all studies. • Evaluating students' ability to apply what they learned rather than testing recall of facts may be a more accurate evaluation of instructional impact. • Most researchers created their own evaluation instrument. Using existing validated instruments would allow more robust comparisons. • There is a need for more published studies (particularly for non-health science disciplines) and for more complete reporting of study design including information about timing, duration, and content.

Conclusion
In the current climate of accountability in higher education, it is important to know whether the time and effort spent on providing library instruction for graduate students is effective in producing an increase in information literacy knowledge and skills. However, studies involving library instruction often lack power due to small sample sizes; combining studies in a meta-analysis to determine an overall effect size can overcome that problem. This review found 12 repeated measures studies and four independent group studies that tested the impact of library instruction. Meta-analysis of the 12 repeated measures studies indicate that library instruction for graduate students was effective in increasing information literacy knowledge and/or skills on average by about one standard deviation. Subgroup analysis found a significant moderation of effect between two broad categories of health science and nonhealth science students. Studies involving health science students resulted in a smaller increase of almost two-thirds of a standard deviation, while studies of non-health science students had an increase of almost 1.5 standard deviations. The difference in the two groups may be the result of a difference in assessment, with health science studies more likely to assess application of knowledge rather than recall of information.
Results of subgroup analyses must be viewed with caution due to small numbers of studies in most subgroups. To strengthen the accuracy of future meta-analyses, there is a need for larger numbers of studies that measure the impact of library instruction, particularly instruction provided in an online or hybrid format. There is also a need for precise description of instructional sessions and more robust data reporting by authors of primary studies.