Towards Understanding Online Question & Answer Interactions and their effects on student performance in large-scale STEM classes

Online question & answer (Q & A) is a distinctive type of online interaction that is impactful on student learning. Prior studies on online interaction in large-scale classes mainly focused on online discussion and were conducted mainly in non-STEM fields. This research aims to quantify the effects of online Q & A interactions on student performance in the context of STEM education. 218 computer science students from a large university in the southeastern United States participated in this research. Data of four online Q & A activities was mined from the online Q & A forum for the course, including three student activities (asking questions, answering questions and viewing questions/answers) and one instructor activity (answering questions/providing clarifications). These activities were found to have different effects on student performance. Viewing questions/answers was found to have the greatest effect, while interaction with instructors showed minimum effects. This research fills the gap of lacking research in online Q & A, and the results of this research can inform the effective usage of online Q & A in large-scale STEM courses.


Introduction
Higher education instructors, regardless of teaching modality and fields, have looked to online communication tools as a means of interacting with and assisting their students. These online communication tools support the primary form of student engagement in online classes, and typically fulfill a secondary or supportive role in classes with face-to-face meetings. These tools may be used to support teacher-directed activities (e.g., required topical discussion of course content), or student-initiated ones. In particular, students may find online communication tools useful when they are confused or need support to complete their course assignments. These student-driven help and information-seeking activities, in which questions and answers are exchanged between human partners, may rely on peer or instructor responses to student queries (Puustinen & Rouet, 2009). Although students might also seek help and ask questions during a face-to-face class session or instructor office hours, or alternatively by email, asynchronous online communication tools offer some distinct advantages in this scenario (Hwang, Singh, & Argote, 2015;Ryan, Pintrich, & Midgley, 2001). They can increase student access to help in situations where class sizes are large and opportunities for face-to-face interaction are limited. They allow students to ask questions when questions arise and receive crowdsourced answers from their peers and instructors (Reznitskaya & Wilkinson, 2017;Xia, Wang, & Hu, 2009). The generated questions and responses on asynchronous online communication tools can be archived, indexed and viewed as often as is needed, and all students in the class can benefit from the answers to a question asked by a single student.
Questions can be asked and answers can be provided in any online discussion forums, but special question and answer (Q & A) platforms have been developed to facilitate this specific form of interaction. These Q & A forums are designed to focus on help-seeking interactions. Rather than writing more general posts, as is done on discussion forums, users on Q & A forums are prompted directly to ask or answer questions. This approach is more effective than a general discussion forum for Q & A interactions (Hao, Barnes, Branch, & Wright, 2017a). Each question begins its own thread, and the appropriate form of response is an answer. Q & A forums vary in their sophistication; for instance, some platforms allow users to upvote popular responses, allow users to prompt question-askers to rephrase asked questions, or associate user reputational information with questions and answers. On the Internet at large, social question and answer (SQA) platforms like Stack Exchange (www.stackexchange.com), Yahoo! Answers (answers.yahoo.com) and Quora (www.quora.com) allow individuals to seek answers to their questions from the general population. SQA platforms crowdsource answers, in contrast to Virtual Reference (VR) services, where experts are contacted to provide answers (Shah & Kitzie, 2012). In a class setting, peer responses are most similar to crowdsourced responses on SQA platforms, varying in depth and accuracy. In contrast, instructor responses might be best equaled by VR services that value quality, and are more likely than SQA to provide a thorough answer including an explanation of how the answer was found or derived.
The value of online Q & A with regard to learning outcomes remains uncertain. Although the relationship between more general online discussion and student performance has been heavily studied (e.g., Beaudoin, 2002;Davies & Graff, 2005;Palmer, Holt, & Bray, 2008;Russo & Koesten, 2005), Q & A forums have been studied far less, and been typically researched in SQA settings rather than class contexts (e.g., Ji & Cui, 2016;Jin, Li, Zhong, & Zhai, 2015;Shah & Kitzie, 2012;Zhang, Liu, Chen, & Gong, 2017). Class-based online Q & A activities differ from SQA activities in some meaningful ways, such as user motivation and interactions (Zhang et al., 2017). For instance, users in an SQA community must form and maintain their own social networks in order to sustain the online Q & A community (Jin et al., 2015;Nasehi, Sillito, Maurer, & Burns, 2012). In a class context, students, as primary contributors to and consumers of the Q & A, have instructor-defined reasons to engage in asking, answering, and viewing questions. Rather than being rewarded solely through attention and community reputation, students might be rewarded by engaging in activities and gaining knowledge that ideally will help support their academic learning and performance (Deng, Liu, & Qi, 2011). Smith IV et al. International Journal of Educational Technology in Higher Education (2020)  The aforementioned general class discussion studies consistently show both that student behavior patterns vary along with student performance in class, and that online discussion is beneficial to student learning (Beaudoin, 2002;Davies & Graff, 2005;Palmer et al., 2008;Russo & Koesten, 2005). This study expands these prior findings by examining different student behaviors and their relationships to student performance within a class-based online Q & A context. The study is situated in a large-scale introductory programming course. Online Q & A may be particularly useful in this and other STEM disciplines where the class size tends to be large and students have substantial needs for Q & A interaction (Henderson & Dancy, 2011).

Related studies
Online Question & Answer, help seeking, information seeking, and online discussion Longstanding definitional problems have muddied the understanding of several important concepts when it comes to online interaction among students, such as helpseeking, information seeking, and online discussion. Help-seeking and information seeking have been studied extensively over the last fifty years mainly in face-to-face contexts, and their boundaries become blurry in online contexts (Puustinen & Rouet, 2009). Help-seeking refers to seeking social assistance from other people, while information seeking refers to seeking information typically from books or machines (Verhoeven, Heerwegh, & De Wit, 2010). The literature on both education and computerhuman interaction in the last two decades has suggested the convergence of seeking social assistance and information (Hao, Barnes, Branch, & Wright, 2017a;Puustinen & Rouet, 2009;Stahl & Bromme, 2009). More help-seeking involves the support from machines, such as search engines or tutor systems, and more information-seeking serves the same purpose as help-seeking (Deng et al., 2011). It is worth noting that both helpseeking and information-seeking emphasize the person who seeks help or information, but not the interactions among people or machines. Online discussion, in educational contexts, refers to course-related conversations among students on discussion boards or forums (Hammond, 2005). In contrast to help-seeking and information seeking, online discussion emphasizes interactions among students (An, Shin, & Lim, 2009). Online discussion has found extensive applications in social or liberal education over the last two decades.
Online Q & A, in educational contexts, refers to interactions involving asking and answering learning questions among students and instructors (Yoo & Kim, 2014). Online Q & A shares some similarities with help-seeking, information seeking, and online discussion, but also bears some notable differences. Online Q & A approaches are similar to online discussion because both emphasize interaction among people (Hao, Barnes, Wright, & Branch, 2017b;Jain, Chen, & Parkes, 2009). However, online Q & A is comprised of three distinct interactive behaviors: asking learning questions, answering questions from others, and viewing questions and answers from others. Online Q & A interactions tend to be situated among problem-solving learning activities. For this reason, they are particularly useful in courses where students work through problem sets or are engaged in other solution-oriented work, such as in programming courses (Hao, Wright, Barnes, & Branch, 2016;Jain et al., 2009;Parnell et al., 2011).

Online Question & Answer Interaction and academic success
Many studies have explored the relationship between online discussion and student performance in the last two decades. Such studies contributed to the understanding of the effective usage of online discussions in educational settings. Beaudoin (2012) found that education majors who were active in online discussion tended to have slightly better performance, although their counterparts who were inactive in online discussion still spend time on other course tasks. Beaudoin's findings were confirmed by Davies and Graff (2005), who also noted a significant difference between students who were active and those who were inactive in online discussion. Russo and Koesten (2005) noted in communication education that students who led the discussion and those who only followed up tend to exhibit different learning styles, motivation and performance through social network analysis. The findings on the relationship between online discussion and student performance are largely consistent. However, it is worth noting that the majority of these studies were conducted in the context of liberal and social education.
Online discussion has not been as widely adopted in STEM fields as it is in the humanities and social sciences, where discursive interactions and the exploration of multiple perspectives are more highly valued. When online discussion is employed in STEM classes, participation has been found to be flat and focused on minimal, deadline-oriented participation (Palmer et al., 2008). The value of this type of participation to learning is questionable (de Jong, Sotiriou, & Gillet, 2014;Yu, Liu, & Chan, 2005). In contrast, Q & A forums have bigger application potential in education in STEM fields, given the typical large class sizes and the needs for solution-oriented communication.
Despite the potential of Q & A forums in STEM fields, there is a lack of research attention on quantifying the effects of online Q & A interactions on student performance. Nurminen, Heino, and Ihantola (2017) surveyed computer science students and found that they were most likely to seek help online from previous social connections and more experienced peers. In this sense, students are likely to mine their personal networks for formal learning purposes in much the same way as other people do for more general Q & A needs (Jin et al., 2015). Consistent with such findings, Hao et al. (2016) found that computer science students tended to ask questions online to their peers more frequently than to instructors, a trend that became more clear with experienced students. Mazzolini and Maddison (2003) investigated the roles of instructors in online Q & A interactions among astronomy students, and found that the active instructors inadvertently discourage students from interacting with peers for Q & A online, but are welcomed by students. Such studies confirmed that students in STEM fields did interact with each other online for Q & A purposes, but did not measure the effects of the interactions on student performance. To our best knowledge, Yoo and Kim's (2014) study is the only one that focuses on the relationship between Q & A interaction and student performance. Yoo and Kim (2014) studied a computer science course and found that the frequency of student answers to questions and the earlier students communicated their problems correlated positively with their project performance. Essentially, students who posted more in the Q & A also performed better on the course project. However, the extent to which this finding can be generalized remains uncertain, and a causal relationship should not be assumed from the correlation. Additional studies are needed to help confirm these findings and explore how specific types of Q & A activities relate to student performance.

Social network analysis in Online Question & Answer Interactions
Social network analysis (SNA) refers to the process of investigating social structures through the use of networks and graph theory (Borgatti, Mehra, Brass, & Labianca, 2009). SNA is a good fit for analyzing student online Q & A interactions. SNA can capture the dynamics of online interactions, and provide a more granular investigation than many conventional metrics, such as posting and viewing frequency (Hernández-García, González-González, Jiménez-Zarco, & Chaparro-Peláez, 2015). Despite the lack of application of SNA in online Q & A interactions, SNA has been applied to understand the online discussion of students. Such research may contribute to understanding online Q & A interactions from the perspective of SNA. For instance, many of these studies demonstrated that a strong sense of community can lead to a strong social network, which has a higher rate of information flow and interpersonal support (De Laat, 2002;Shen, Nuankhieo, Huang, Amelung, & Laffey, 2008). A strong sense of community can be evidenced by a set of factors such as student-instructor ratio, instructor immediateness, and student performance.
For instance, instructor immediateness contributes to a stable social network of online discussion. Instructor immediateness is a social network factor, and it refers to the extent to which instructors can provide facilitation to online interaction. As the student-instructor ratio increases, the more their ability to facilitate interactions among students is constrained. However, if an instructor becomes dominant in the social network, student-student interactions might be discouraged (Rovai, 2007). For another instance, student performance can be an important factor in determining the constructs of an online discussion network, such as network density and stability (Dawson, 2010). The higher students' academic performance is, the more likely their network is to have higher density and centrality scores. Ghadirian and his colleagues (Ghadirian, Salehi, & Ayub, 2018) corroborated the findings of Dawson and suggested that low-performing students are less willing to engage in a discussion given their innate lack of knowledge and, as a result, confidence. This condition is only worsened given their lack of interactions with high performing and well-connected members of the network who may aid them in expanding their knowledge base.

Research design and questions
This study used a quasi-experimental research design to investigate the relationship among the activity level of students and instructors using an online Q & A forum, and student performance on course assessments. Specifically, the research questions addressed by this study are:

Participants and contexts
This study was conducted in a face-to-face introductory programming course offered at a large research university in the southeastern United States. To strengthen the stability and generalizability of our findings, we collected data from the same course in both the summer and fall semesters. A total of 218 participants participated in this study, consisting of 54 students in the summer semester and 164 students in the fall semester. During both semesters the course had the same instructor, syllabus, curriculum, exams, and grading rubrics. However, in the fall semester the course duration was 50% longer than in summer, and covered the course material in a more in-depth approach.
In both semesters, an online Q & A forum, Piazza (piazza.com), was set up for students to ask learning questions to their peers, TAs and the instructor. As a Q & A forum, Pizza has been adopted widely in large-scale computer science and engineering courses (Hao, Galyardt, Barnes, Branch, & Wright, 2018). In our study, students were encouraged but not required to ask questions to their peers and instructor using Piazza. It is worth noting that student online Q & A interactions on Piazza were periodically pushed to each individual student of the same course through automated emails.

Data collection and analysis
To answer the three proposed research questions, we needed to collect student online Q & A interaction data. To achieve this, we programmed a script that mines Q & A data from Piazza. We mined the Q & A data of the two courses that were studied in this research. From the mined data, we were able to extrapolate the following online Q & A interaction behaviors and frequency data: The frequency of posting notes and clarifications only applies to the instructor and teaching assistants (TAs). The other frequency data applies to the instructor, TAs, and students.
To answer the RQ1, we applied SNA to the mined online Q & A interactions. Specifically, directed network graphs were constructed per semester. The following network factors were considered: Density: The ratio between all connections in a network and the number of possible connections. Transitivity: The extent to which interconnected transitive triads exist in a graph. Smith IV et al. International Journal of Educational Technology in Higher Education (2020) 17:20 Page 6 of 15 Closeness Centrality: This represents the average distance from one node in the network to another. Betweenness Centrality: This is a method of indicating the influence of a node in a network by the number of vertices in a given vertexes path of communication.
Eigenvector Centrality: Defines the importance of a vertex by the degree to which it is connected with other well-connected vertices.
To answer RQ2 and RQ3, we collected student performance data on both programming assignments and paper-based exams. The reason for using two different measurements is the dual focuses of programming courses, including student programming capability and conceptual knowledge on programming languages and problem-solving. Programming assignments focus on measuring student programming capability, while paper-based exams focus on measuring student conceptual knowledge. Both programming assignments and exams were graded by the course instructor with the help of two teaching assistants. Once data matching between online Q & A data and performance data occurred, all individual identifiers were cleaned from the dataset. Based on the collected data, we used blockwise regression to explore the extent to which student, instructor, and TAs' online Q & A interactions can predict student course performance.

How do students interact with their peers and instructors in online Q & a forums (RQ1)?
To answer RQ1, we used descriptive analysis and SNA to investigate how students interact with their peers, TAs, and instructors in terms of asking and answering questions. The behavior "viewing questions" was excluded from SNA given the lack of knowledge in who viewed which questions or answers. The descriptive analysis results are presented in Table 1. As student numbers increased, they tended to ask and answer more questions on average. As questions and answer numbers went up, the average view also increased substantially. Overall, the frequency data showed that students successfully formed online Q & A communities regardless of student numbers in class.
SNA uncovered a consistent theme across the two semesters: online Q & A interactions are student-driven. The evidence for this theme is that when instructors and TAs are removed from the social networks, the networks still showed strong centralization (see Fig. 1). Although instructors and teaching assistants were the most central nodes in these networks (by several metrics), their removal did not alter substantially local statistics for nodes as well as overall network statistics (e.g., density and transitivity). To quantify our observation from the visualization, we calculated all network factors and compared between with and without instructors/TAs. The results are presented in  To further understand the characteristics of students who contributed more to asking and answering questions, we separated students of the same class into three groups by their overall academic performance (total of both programming assignments and exams): high, medium and low performing students. The summary of their online interaction frequencies is presented in Table 3. The only consistent finding across the two semesters is that high-performing students with higher overall academic performance tended to view more questions and answers than low-performing students [summer: t = 3.30, p < 0.025 1 ; fall: t = 2.87, p < 0.005 2 ].
To what extent do student activity and instructor facilitation in online Q & a forums predict student performance (RQ2 and RQ3)?
To answer RQ2 and RQ3, we applied blockwise regressions to understand the extent to which online Q & A interaction activities of students, TAs, and instructors predict student performance. Given the nature of the programming courses, there were two metrics of student performance: programming assignments and paper exams. To avoid inflating Type II error, we applied Bonferroni Correction to the significance levels. The adjusted significance levels were: * p < 0.025; **p < 0.005; ***p < 0.0005. We separated all predictors of online Q & A interactions into two blocks:

Student online Q & A behaviors:
○ Ask: ask questions to peers, TAs, and instructors online When only the block of student online Q & A behaviors were used as predictors, we found that "view" as a significant predictor of student performance in programming assignments. This finding was consistent across the two semesters. The results are presented in Table 4. Overall, asking, answering, and viewing questions/answers explained 16.2% variance of programming assignment performance in the summer semester, and 11.6% variance in the fall semester.
In contrast, asking, answering, and viewing questions/answers explained 10.0% variance of exam performance in the summer semester, and 3.6% variance in the fall semester. However, neither the overall results nor any single factors were found significant in predicting student exam performance.
When the block of instructor/TA online Q & A behaviors were added to the regression models of predicting student programming assignment performance, no significant improvements were found for the summer semester [F(1,17) = 0.148, p > 0.0.25] or the fall semester [F(1, 58) = 0.674, p > 0.025]. However, it is worth noting that when the Instructor/TA behaviors were controlled, student behavior "view" was still a significant predictor of student programming assignment performance for the summer semester. A similar effect was not found for the fall semester. The results are presented in Table 5.
When the block of instructor/TA online Q & A behaviors were added to the regression models of predicting student exam performance, no significant improvements were found for the summer semester [F(1, 17) = 0.455, p > 0.025] or the fall semester [F(1, 58) = 0.003, p > 0.025]. Different from the models of predicting student programming assignment performance, no single factors were found consistently significant in predicting student exam performance. When the Instructor/TA behaviors were  controlled, student behavior "view" was found as a significant predictor of student exam performance in the summer semester. However, this finding was not replicated in the fall semester.

Discussion
We systematically investigated online student interactions in large-scale face-to-face programming courses over two semesters in this study. Of all the findings in this study, we would like to highlight and discuss three findings that were consistent across the two experiments of this study. First, low-performing students should be identified from early assessments and given more guidance/facilitation on effective usage of online Q & A. Although we investigated the predictive power of online Q & A interaction on student performance, it is important to note that our results do not imply causality between online Q & A and academic achievement. Despite the lack of causality, we found that high-performing students tended to use online Q & A forums more actively than their counterparts. This finding shares similarities with the prior studies on both online Q & A and online  discussion. Hao, et al. (Hao, Barnes, Wright, & Branch, 2017b) reached similar findings through a survey study on online Q & A in the context of computing education. Palmer et al. (2008), as well as Davies and Graff (2005), found that students who were active in online course discussions tended to outperform their counterparts in terms of the overall course grade. Compared with prior studies, this study took a further step by understanding fine-grained differences among three types of student online interactions. It is worth noting that a significant difference was only found in viewing questions/answers among students with different academic performance. In other words, high-performing students did not necessarily ask or answer more questions, but they viewed significantly more questions and answers from both peers and instructors for learning purposes. In large-scale classes, low-performing students can be identified from an early assessment. If these students are given guidance on using online Q & A (e.g., read questions and answers from peers), they may gradually learn how to utilize online Q & A effectively for learning purposes, which may lead to overall student performance improvement. To understand why students of different performance behave as such, future studies may consider an explorative qualitative study on student usage of Q & A in STEM classes. Second, effective online Q & A may not require instructors to actively answer student questions. This finding was indicated by multiple evidence accumulated when answering the three proposed research questions. The facilitation of the instructor was found insignificant in predicting student performance. More importantly, when we analyzed the social networks formed by online Q & A interactions, we noticed that the social networks remained robust and healthy when instructor/TAs were removed by all network metrics. This finding is consistent with prior studies on online discussion. For instance, high levels of instructor intervention in online discussion has been found to stifle peer interaction (Dennen, 2005) and shorten discussion or stop further conversation (Knutas, Ikonen, & Porras, 2013;Mazzolini & Maddison, 2003). In the context of online Q & A situated in large-scale face-to-face classes, the instructors' early intervention might remove the barrier to problem-solving immediately, but may also eliminate potential cooperative problem solving and knowledge construction among students (Hao et al., 2016;Mazzolini & Maddison, 2003). Given the heavy teaching loads of large-scale classes, it might be more beneficial for instructors to intervene less in online Q & A among student peers unless a question stays unanswered for some time. However, it is worth noting that the generalizability of this finding may have limits. This study is situated in a large-scale face-to-face programming course. Introductory programming courses are typically featured by a large number of students and intensive hands-on problem-solving tasks. Although online Q & A plays a significant role for students to ask questions and have their questions answered, it is by no means the only approach. Students can meet the instructors and TAs in person to ask questions. In a purely online or blended learning environment, the role of instructors in online Q & A interactions may change significantly (Bliuc, Ellis, Goodyear, & Piggott, 2011;Khine & Lourdusamy, 2003). For instance, the instructor facilitation of online Q & A interactions in a purely online learning environment may become more important than the face-to-face classes (Lynch, 2010).
Third, effective usage and engagement in online Q & A are not necessarily observable. Viewing is not deemed as active participation behavior in online discussion (Davies & Graff, 2005;Hao et al., 2016). Typically, students who only engage Smith IV et al. International Journal of Educational Technology in Higher Education (2020)  in viewing discussion content are described as "lurkers" (Rovai, 2007). Lurkers have been found to have negative impacts on the development of community networks in online discussion forums (Nonnecke & Preece, 2000;Rovai, 2007). When there is a higher percentage of lurkers, the online community may risk collapsing or never be successfully formed. However, we found that student viewing questions/ answers to be positively significant in predicting their programming assignment performance. This finding was consistent across the two semesters and with or without controlling the facilitation from instructors/TAs. On the one hand, it is surprising that a seemingly passive behavior could have any significant impacts on student performance. On the other hand, given the focus of the course is problemsolving, it is understandable that active discussion is not necessary as long as a learning question is clearly answered. When a learning question is clearly answered, all students can all benefit from reading it online. This is different from the courses in social studies or liberal studies, where the intensity of online discussion and debates is positively correlated with student performance (e.g., Hara, Bonk, & Angeli, 2000;Knowlton, 2002). Viewing questions was found to have limited predictive power on student exam performance, but students also asked significantly fewer questions on the exam than programming assignments. Given the nature of paper-based closed-book exams, it is less likely students would rely heavily on online Q & A for their studying and preparation. In contrast, when students encounter difficulties or challenges while working on their programming assignments, online Q & A can promote just-in-time learning. Given the learning benefits of viewing questions/answers, instructors may consider different information delivery approaches that best expose such information to students, such as periodically pushing information to students through emails (Warren & Meads, 2014).

Limitations
There are three major limitations of this study. First, this study was conducted in a single context, computing education. The extent to which our findings can be generalized to other fields, especially non-STEM fields, needs further investigation. Second, this study was conducted in only one higher education institute, which may limit the generalizability of the findings. To mitigate this limitation, a replication study was designed and conducted during the study period of time, and the findings were consistent across the two studies. To further verify the generalizability of our findings, future studies may consider replicating our studies across multiple higher education institutes. Third, the demographic information of the participants was not collected in this study.
Although demographic factors, such as race or gender, were found less important than student performance in terms of how student behave in online discussion, the lack of such data prevents further investigation into how students of different gender, race, or age interact with each other in online Q & A. Future studies may consider collecting such data through surveys and provide a more fine-grained analysis on the relationship between student demographics and online Q & A interaction.

Conclusion
Online Q & A forums have found wide application in large-scale STEM courses during the last few years, but few studies have investigated the effects of online Q & A interactions on student learning. This study sought to fill this gap by quantifying the effect of fine-grained online Q & A interaction behaviors on student academic performance. The findings of this study provide evidence to support that online Q & A interactions share similarities with online discussion, but also bear significant differences. Viewing questions/answers is typically deemed as a passive behavior that does not indicate active participation in online discussion. However, this study demonstrated that such passive behavior contributed to stronger student academic performance in Q & A interactions. Given the significance of viewing questions/answers in predicting student assignment performance, instructors are recommended to push such information to "the face of students" through pushing functions, which may further amplify its positive effects.