Teaching and User Satisfaction in an Academic Chat Reference Consortium

This study investigated 299 chat reference interactions from an academic library consortium for instances of teaching and compared these against other characteristics of the chat, such as question content, staff type, user status, user satisfaction, institutional affiliation, length, and shift busyness. Statistical analysis revealed that teaching was more likely to occur when the chat was a researchrelated question or when the operator was a graduate student worker employed by the consortium but less likely when the operator was a paraprofessional. Chats with teaching were longer but were also associated with higher user satisfaction scores. Teaching rates did not differ by institutional affiliation, shift busyness, or user type. These results indicate that busy consortial services can offer comparable teaching service to single-institution services. The researchers recommend updating operator training to better emphasize teaching to increase user satisfaction.

Teaching and User Satisfaction in an Academic Chat Reference Consortium Library staff must balance two competing priorities with each new reference interaction: 1) delivering resources to users and answering questions as expediently as possible and 2) emphasizing instruction and taking the time to teach research skills. Academic librarians are increasingly giving precedence to instruction, recognizing their responsibility to participate in the teaching missions of their institutions (Elmborg, 2002). In response, academic librarians are expanding their teaching beyond the classroom and incorporating instruction into their reference work to help patrons become self-sufficient researchers.
As teaching at the reference desk has become a widely accepted practice, attention has shifted to teaching in the online environment. Chat reference providers are well-positioned to teach: patrons initiate chats at their point of need, a time when they are most receptive to learning research techniques (Beck & Turner, 2001;Ellis, 2004). Chat reference staff can take advantage of teachable moments within the interaction to build the user's information literacy skills and offer hands-on, authentic learning opportunities (VanScoy & Oakleaf, 2007).
Teaching within the chat reference environment presents a number of unique challenges, including the lack of personal connection between the participants, difficulties with turn-taking, juggling multiple patrons, time constraints, and technical issues (Steiner, 2011;Woodard, 2005). In these conditions, providing step-by-step instructions can be slow, causing the chat provider to bypass teaching opportunities. Teaching is further complicated in consortial chat arrangements where staff may be unfamiliar with other libraries and the courses and assignments students are completing (Singh & Furuta, 2008). This lack of local knowledge may make teaching more time consuming or, in some cases, impossible.
Currently, little is known about teaching practices and their relationship to user satisfaction within consortial chat services. This is an important area of study as approximately a quarter of college and university libraries in North America offer their chat services consortially (Yang & Dalal, 2015). The current study aims to fill this gap through an evaluation of Ask a Librarian, a consortial chat reference service in Ontario, Canada. The study sought to answer the following questions: [ RESEARCH ARTICLE ] Barrett et al. Teaching and User Satisfaction in an Academic Chat Reference Consortium 183 COMMUNICATIONS IN INFORMATION LITERACY | VOL. 14, NO. 2, 2020 • To what extent are operators teaching through chat?
• Are operators more likely to teach if they are from the same institution as the user?
• Do teaching rates vary based on the user type, operator type, or question type?
• Are operators less likely to teach during busy shifts?
• Does teaching affect user satisfaction?

Teaching in Reference
Historically, the librarian's role in reference services has not been clear: do librarians have a responsibility to teach research skills, or should they provide answers to patrons (Hunter et al., 2019)? These two functions have received different emphasis over time. Schiller (1965) stressed information over instruction, arguing that librarians should exercise their professional skill to find and provide information, to save the user's time, and to prevent information overload. However, the popularization of student-centered and active learning and the push for information competency have increased demand for librarians to become learning facilitators (Avery, 2008;Beck & Turner, 2001;Ellis, 2004).
Teaching behaviors are now established in professional standards. RUSA's Guidelines for Behavioral Performance of Reference and Information Service Providers state that librarians should explain the search strategy and how to use sources and offer pointers, detailed search paths, and names of resources "so that the patron can learn to answer similar questions on his/her own" (American Library Association, 2008, para. 4.1.8). These points emphasize teaching as an important aspect of the reference transaction in all settings.

Teaching Practices in Chat Reference
Many studies have estimated the amount of teaching that occurs specifically during chat reference transactions. Hervieux and Tummon (2018) and Taddeo and Hackenberg (2006) found low levels of teaching on their chat services, with 23% and 26% of chats showing evidence of information literacy instruction, respectively. However, most single-institution chat services have found that from 60 to 95% of transactions demonstrate instances of teaching (Dempsey, 2017;Desai & Graves, 2006;Graves & Desai, 2006;Johnston, 2003;Moyo, 2006;Oakleaf & VanScoy, 2010;Ward, 2004;Zhuo et al., 2006). A study of NCKnows, North Carolina's statewide chat reference collaborative, also found Barrett et al. Teaching and User Satisfaction  a high level of teaching, with 97% of transcripts exhibiting instances of information literacy instruction (Daly, 2007), indicating that teaching also takes place in collaborative or consortial services.
Chat operators typically teach information literacy skills related to developing research topics and finding appropriate resources. These skills correspond to standards one and two of ACRL's Information Literacy Competency Standards for Higher Education (Avery & Ward, 2010;Devlin et al., 2007;Ellis, 2004;Smyth, 2003) and the concepts of "Searching as Strategic Exploration" and "Research as Inquiry" from ACRL's Framework for Information Literacy for Higher Education (Hunter et al., 2019). Reflecting this emphasis, the most common teaching techniques on chat are resource or keyword suggestion and modelling or leading the patron through the search process (Graves & Desai, 2006;Hervieux & Tummon, 2018;Moyo, 2006).
The likelihood of teaching on chat is affected by several factors. Dempsey (2017) found that librarians incorporated fewer teaching behaviors when the service's reference policy stated that chat was for brief factual questions. The user's initial question also affects operators' teaching behavior: librarians are more likely to teach when patrons begin the conversation with relational cues (Dempsey, 2016) and when users ask for instruction (Desai & Graves, 2006). Teaching practices are also influenced by librarians' attitudes toward virtual reference (Gronemyer & Deitering, 2009). Staff type may also affect teaching on chat. Desai and Graves (2008) found that librarians taught more than paraprofessionals, but Keyes and Dworak (2017) found no significant difference among student employees, paraprofessionals, and librarians.

Patron Preferences and Satisfaction
Compared to the many studies discussing the library staff perspective, only a few studies have investigated user preferences regarding teaching over chat. Desai and Graves (2006) administered a survey to users of their instant messaging service to determine if they wanted answers to their questions or if they preferred to learn research skills. They found that 62% of patrons wanted to learn, 30% were apathetic, and 8% did not want to be taught. When the researchers repeated their study after their library transitioned to a commercial chat platform, the proportion of patrons who wanted to learn increased to 82%, and only 3% were resistant to instruction (Graves & Desai, 2006). A study by Jacoby et al. (2016) also found that students were receptive to teaching over chat, although they balanced the value of instruction against their desire for speed and convenience. Daly (2007) investigated satisfaction with teaching on NCKnows and found that over 80% of users reported complete satisfaction with the service provided in the exit survey. A statistically significant [ RESEARCH ARTICLE ] Barrett et al. Teaching and User Satisfaction in an Academic Chat Reference Consortium correlation was observed between patron satisfaction and teaching related to library policies, the technical aspects of using or locating information, and the exact source used to satisfy an information need. The specific teaching techniques related to satisfaction, such as providing a direct URL to a resource, helped the patron to fulfill an immediate need.

Background and Setting
Ontario Council of University Libraries (OCUL) is an academic library consortium of the 21 universities in the province of Ontario, Canada. Through the central Scholars Portal team, OCUL provides a number of services to its members, including Ask a Librarian, a shared bilingual chat reference service. Ask a Librarian is offered on an opt-in basis; it is currently running at 15 institutions with a total student FTE of approximately 403,000 (Ontario Council of University Libraries, 2019). Chat operators consist of library staff at the participating universities as well as graduate students or recent graduates who are hired by OCUL for evening and weekend shifts.
Although the service receives mainly positive feedback from users via the exit survey, the research team wanted to evaluate chat transcripts to determine if specific operator-and service-related variables were correlated with satisfaction or dissatisfaction. Previous transcript analysis studies by the research team have focused on the effects of operator behaviors (Logan et al, 2019), communication style (Logan & Barrett, 2019), and internal factors such as scheduling, staffing practices, and policies (Barrett & Pagotto, 2019) on user dissatisfaction. Next, as teaching is such a key component of reference transactions, the researchers wanted to see how Ask a Librarian transcripts compared to other findings in the literature.

Data Collection and Sampling
This study made use of chat data and transcripts from June to December 2016, a period during which 9,424 chats were submitted to the service. Before each chat, users completed a mandatory pre-chat survey that asked their name, institutional affiliation, status (e.g., undergraduate, faculty member), and information need. This information was logged automatically by the chat software and tied to the transcript of the interaction, along with metadata such as the operator's name and affiliation; the date, time, and length of the interaction; and the language of the interaction.
Users could also complete an optional exit survey which asked four questions relevant to the user's satisfaction with their interaction (see Table 1).  Of the 9,424 interactions that occurred during the survey period, 1,395 (14%) included completed exit surveys. The authors randomly selected 299 interactions with completed exit surveys (21%) to achieve a confidence level of 95%.

Data Preparation
The research team obtained approval from the University of Toronto's Research Ethics Board and from the consortium's Ask a Librarian Data Working Group to proceed. The team then removed all instances of the user's name, the operator's name, and the institutions that they belonged to from the metadata about the chat, the pre-chat survey, the chat transcript, and the exit survey.

Variables
The authors required the following variables to answer the research questions:

Teaching
This variable identified whether the operator engaged in teaching behaviors. The authors defined the presence of teaching as the operator providing detailed explanations or instructions for the user

Barrett et al. Teaching and User Satisfaction in an Academic Chat Reference Consortium
to follow. This could include describing where they found information, sending a link to instructions or explanations (such as an FAQ or LibGuide), or walking the user through a process.
The authors coded each of the 299 sampled transcripts for the presence of teaching. Being unable to initially obtain acceptable intercoder reliability, the authors subsequently adopted a multiple coding strategy following Barbour's (2001) model: First, Coder A was assigned a set of transcripts to review and identified all the instances of teaching. Coder B then coded the same set of transcripts without knowledge of Coder A's decisions. Coder C resolved any disagreements between Coder A and Coder B.

Operator Type
Before the researchers removed the operator's name from the chat metadata, they noted operator staff type: • Librarians

• Paraprofessionals
• Student employees (employed by the participating institutions) • Part-time virtual reference operators (VROs) (Master's-level library and information studies students or recent graduates employed by the consortium)

User Type
In the pre-chat survey, users self-identified as either an undergraduate student, graduate student, faculty member, staff member, alumni, or none of the above.

Question Type
Two members of the research team coded the user's initial question in the mandatory pre-chat survey using a coding schema created at their institution (Maidenberg et al., 2012). The researchers had previously coded two test sets of 42 and 44 responses respectively, and established a nearperfect intercoder reliability score of K = 0.876. The coding schema contains ten categories (a more fulsome description of each category is available in Appendix A):

Affiliation Match
Before anonymizing each interaction, the researchers created a variable to note whether both the operator and user came from the same institution (an affiliation match) or if they came from different institutions (an affiliation mismatch).

Busy Shift
The chat metadata included a date and time stamp recorded by the software. The authors consulted Ask a Librarian's chat statistics to determine how many chats were submitted during that shift (one hour of service). Busyness was determined by comparing the number of chats submitted during the hour to the number of operators scheduled to be online. A shift was considered busy if more than three chats were submitted for every available operator. For example, if six operators were scheduled to be online, the shift was considered busy if more than 18 chats were submitted during the hour.

Information Overload
In the exit survey, users were asked if they felt the librarian provided just enough, too much, or too little information. This was an optional question, so not all users who completed the exit survey responded.

Length of Chat
The chat software automatically logged the number of seconds an interaction lasted.

Composite Satisfaction Score
The authors created a composite variable following Kwon's (2006) approach to quantify the exit survey's satisfaction questions. Each response was given a numerical value as displayed in Table 2. The response values were then tallied and divided by the total number of eligible points to get a composite score between 0 and 1. Since all questions were optional, not all users completed every question. The eligible points reflected the maximum number the user could have given based on the number of questions they answered. For example, if they answered one question, they had five eligible points. If they answered all four satisfaction questions, they had twenty eligible points.

Data Analysis
The variables in this study are either categorical or integer types. For the categorical variables, which are not hierarchical, the authors identified the chi-square test of independence as an appropriate statistical test. Length of chat and composite satisfaction score are integer variables. Therefore, independent samples t-test was the most appropriate test to determine if teaching in a chat affected its length and the user's satisfaction because it compares the means between two groups of a population with different characteristics (i.e., those chats where teaching was observed and those where it was not). The authors collated all of the variable data in Excel and performed statistical testing in SPSS.

Data Characteristics
Librarians (n = 106, 35.5%) and part-time VROs (n = 73, 24.4%) served the majority of chats in the sample. Users were mainly undergraduate students (n = 160, 53.5%) and graduate students (n = 78, that the operator provided just the right amount of information. See Table B1 in Appendix B for complete accounting of the data's characteristics. The mean duration of the sampled chats was 1,270 seconds (21 minutes) with a standard deviation of 1,057 seconds (18 minutes), indicating a high level of variability in the interactions' lengths (see Table 3). The mean of the sample's composite satisfaction score was .92 (see Table 4).

Statistical Analyses
The authors compared the observed and expected counts of teaching in the chat transcripts across six variables-operator type, user type, question type, affiliation match, busy shift, and information overload-using Pearson's chi-square test of independence. A statistically significant difference was found between the expected and observed cases of teaching for two variables: operator type, χ 2 (4, N = 299) = 12.26, p = .015, and question type, χ 2 (9, N = 299) = 27.14, p = .001. For operator type, paraprofessionals taught in 9.1 fewer chats than expected while part-time VROs taught in 6.6 chats more than expected. Regarding the question type, teaching occurred in 15.8 more research-related chats than expected, in 6 fewer policy questions than expected, and in 4.4 fewer accounts questions than expected. Table B2 in Appendix B shows a summary of these findings. The results of the independent samples t-test indicate a significant difference between the means of the composite satisfaction scores when teaching was present ( Table 5 for a summary.

Discussion
The existing literature on teaching in chat reference deals only minimally with user satisfaction and even less with collaborative chat services, despite the prevalence of such services. Thus, the findings of this study both enrich the literature and inform the practices of Ask a Librarian and consortial academic chat services.
Two research questions (R2 and R4) addressed operational issues in consortial chat services not studied previously in the literature: R2: Are operators more likely to teach if they are from the same institution as the user?
The study found no significant relationship between teaching and institutional match; operators in a collaborative chat service are just as likely to teach users from other institutions as their own. This affirms the collaborative service model. Users are getting the same level of teaching when they use the service regardless of whether an operator from their institution is online.

R4: Are operators less likely to teach during busy shifts?
The study found that chats that included teaching were longer than chats that did not but chats involving teaching were no less common during busy shifts. One explanation is that this kind of chat, although lengthy, involves more downtime for the operator while the user follows along with their instructions, allowing the operator to turn their attention to another chat. Teaching occurred in roughly one third (33.4%) of chats sampled and in just under half of chats where the initial question was research-related (70 of 162 chats, or 43.2%, see Table B2). This is on the lower end of the findings in the literature although lower rates have been reported (Hervieux & Tummon, 2018;Taddeo & Hackenberg, 2006). This rate is significantly lower than the findings of another collaborative service, NCKnows (Daly, 2007).
The researchers did not assess whether the chats contained "missed opportunities" when teaching did not occur (i.e., instances where teaching would have been helpful and appropriate). Ask a Librarian is presented as a first point of call for library help on many of the partner institutions' websites, which may affect the types of questions students ask (see R3) or how they ask them. As Dempsey (2016) notes, the type of questions asked can affect librarians' behavior.
Additionally, although Ask a Librarian training encourages operators to seek the right balance between answering questions and teaching based on the user's responses and potential time sensitivity, the role of teaching as a whole is not strongly emphasized in training. Service coordinators assume that operators (other than part-time VROs) are already performing reference at their libraries and focus training on chat-specific logistical and communication tips. The level of teaching may therefore vary based on local practices, operators' attitudes towards chat, and/or operators' perceptions of the service, as Gronemyer and Deitering (2009) and Dempsey (2017) found.
R3: Do teaching rates vary based on the user type, operator type, or question type?
Operator type was significantly related to teaching in chat. Paraprofessionals taught in fewer chats than expected, but part-time VROs taught in more chats than expected. This is a departure from findings in the literature that observed no significant differences between student employees, paraprofessionals, and librarians (Keyes & Dworak, 2017).
For part-time VROs, the higher-than-expected teaching rates may in part be related to training. Since part-time VROs are LIS graduate students or recent graduates working directly for the consortium, they receive more intensive training than other operators, including a focus on reference practices. They may also be more conscious of following correct process since, for many, this is only their first or second work experience providing reference services, and they are aware that their transcripts may be evaluated by the Ask a Librarian coordinator.
While librarians taught at about the rate expected, paraprofessionals taught in fewer chats than expected. The same phenomenon was found by Desai and Graves (2008). The difference may simply be practical: librarians are more likely to have teaching duties, especially information literacy instruction, as part of their daily responsibilities, and they may spot teaching opportunities that paraprofessionals miss. Ideologically, librarians are considered faculty members at most of the Ask a Librarian partner institutions and, therefore, may view themselves as part of the teaching mission in their institution in a way that paraprofessionals may not (Elmborg, 2002). More qualitative research would be needed to tease out these relationships and shed light on these hypotheses.
Question type also had a significant relationship to teaching rates. Teaching was more likely to be present in research-related questions and less likely to be observed in policy questions or accounts questions. This was unsurprising as research topics are some of the most likely to benefit from teaching while teaching is not always possible or appropriate with policy or account questions.
No relationship was found between user type and instruction. This seems to indicate that Ask a Librarian chat operators recognize that all user types may benefit from teaching.

R5: Does teaching affect user satisfaction?
The vast majority of users indicated that the operator provided just the right amount of information, and only 1% of users felt they received too much information. This suggests that, at the very least, teaching is not alienating chat users with information overload. On the other hand, users who felt overwhelmed by the amount of information they received may have ended the chat prematurely and did not fill out the exit survey. Thus, these users would not be represented in this study.
A small but significant difference in satisfaction scores between users who were taught during the chat and those who were not indicates that teaching increases user satisfaction. This aligns with previous work in the literature indicating that users want to learn and are more satisfied with services that teach them how to find answers than those that rely on referrals (Graves & Desai, 2006;Daly, 2007). Since the teaching rate in Ask a Librarian was lower than in many other chat services, training staff to teach during chat may help to increase that rate and, by extension, user satisfaction.

Conclusion
While many academic libraries provide chat reference consortially, the literature on the relationship between user satisfaction and teaching in consortial chat services is small. This study sought to examine that relationship in a regional university library consortium.
In this study, the relationship between teaching in chat and higher satisfaction scores was statistically significant. However, the generalizability of this finding is subject to several limitations. The researchers simply identified the presence of teaching and did not investigate "missed opportunities," or instances when teaching did not occur but would have been appropriate. User satisfaction was self-reported in an optional survey, creating the potential for nonresponse bias and self-report bias.
Rates of teaching were lower than those reported in other studies but consistent regardless of whether operators were helping users from the same institution or another institution. Chats that occurred during busy shifts were just as likely to include instances of teaching as chats during regular shifts.
The most significant contributing factors to teaching rates were operator type and question type. The difference based on operator type may be influenced by training and the varying relationships that librarians, paraprofessionals, and student employees each have to the university's broader teaching mission. The perceptions of operators towards the service, and how this impacts their behavior, is a potentially fruitful area for future research. Teaching was more likely to be present in response to research questions. This may be due to the fact that research questions lend themselves to more teachable moments.
Overall, consortial chats that included teaching led to higher user satisfaction. New training programs to help operators identify and capitalize on teachable moments can support the chat reference service as it grows from a provider of just-in-time answers and increasingly takes on a teaching mandate.