Extending an Evaluation of a Volunteer-Tutoring Program in Reading: Examining the Growth in Reading Achievement for Elementary Students with Learning Disabilities

Tutoring in reading when provided as an extension to classroom teaching, is commonly considered to be an effective way of increasing students’ academic proficiency, particularly in the area of early reading skills. Additionally, the effectiveness of one-toone tutoring as an effective reading intervention has been validated by empirical research, especially for those children who are considered at-risk for academic failure, or have been identified as having reading or learning difficulties or disabilities (Elbaum, Vaughn, Tejero-Hughes, & Watson-Moody, 2000; Hampton, Morrison, Rizza, & Osborn, 2015; Osborn et al., 2007; Slavina, Lake, Davis, & Maldena, 2011; Vaughn et al., 2009). Elementary teachers routinely identify trained, volunteer one-to-one tutoring as an optimal instructional strategy to implement in their classrooms. Yet, these teachers report that it is most challenging to implement in their classrooms due to the availability of trained tutors (Elbaum, et al.; Pullen, Lane, & Monahan, 2004). The importance of reading interventions in the primary grades is underscored by Hecht and Greenfield (2001), who note that few changes in individual reading skills occur after the third grade. As other important consequences to lack of early intervention include, poor academic outcomes, increased behavioral issues, higher probability of these students dropping out of school will lead to limited employment opportunities later in life.


Volunteer Tutoring
The implementation of tutoring programs has existed in education for several decades. Multiple reasons are frequently cited for the growth of a variety of tutoring programs in schools. The attention of the public was focused on the deficiencies of student proficiency in reading, particularly after the publication of the National Commission on Excellence in Education's, A Nation at Risk (1983). With this increased attention came a resurgence of volunteers ready to tutor students who may be struggling in reading development. Yet, despite the increased use of volunteer tutoring in schools, particularly with volunteers considered to be "non-professional" volunteers, there is not yet a sufficient body of evidence to confirm the effectiveness of the use of a diverse population of volunteer tutors (Hampton, et al., 2015;Ritter, Barnett, Denny, & Albin, 2009).
Empirical research has been conducted in a few large-scale studies aimed at improving the academic achievement of students who struggle the most in foundational reading skills. Two important studies, Reading Recovery and Success for All (Shanahan, 1998;Wasik & Slavin, 1993) are considered to be effective tutoring programs, but both require the use of professional tutors in their implementation. The most significant limitation to these programs rests in the limited availability of professional volunteers, which in turn, limits the number of struggling students who can be assisted (Ritter et al.,). In a review of 17 studies which employ volunteer tutors to improve students' reading skills, Wasik (1998) found that some of these programs could assist students who are struggling with reading development; only 2 of the 17 programs reviewed compared targeted students with a comparison group. This review has led to an increase in the evaluation of programming to meet the needs of students who struggle with reading development. Many of these studies have provided a beginning body of evidence for the effectiveness in the implementation of volunteer tutors in providing additional instructional supports for students in need of support in the area of reading ( Burns, Senesac, & Symington, 2004;Fitzgerald, 2001;Moore-Hart & Karabenick, 2009: &, Pullen, Lane, & Monaghan, 2004.The intention of continuing to build the body of evidence on the use of volunteer tutors for assisting students with reading difficulties formed the basis of this evaluation. Researchers were asked to evaluate a currently used program for using volunteer tutors to assist students with reading disabilities improve their reading development as a supplement to core reading instruction and special education services.

Rationale for Project MORE intervention
When we initiated this study, the most reliable and valid empirical support for implementing interventions for students who have reading difficulties used standardized interventions (Vaughn et al., 2009). In a review of intervention studies, Wanzek & Vaughn (2007) reports findings from 18 studies conducted with primary grade students with significant reading difficulties. None of the studies reported using problem-solving or individualized instruction based upon student needs. This report defines an intervention of having high levels of standardization within the intervention and allowing for possible adjustments based upon student performance (Vaughn et al.). Project MORE (PM) incorporates the use of standardized methodologies in each of their tutoring sessions, while allowing for a small portion of the tutoring session toward meeting the students' reading needs based upon student IEP goals. The objective of PM is not to replace general education instruction in reading, including those programs considered in Tier One Response to Intervention (RTI) programming. PM is used as a supplement to core instruction and special education services (Hampton, et al., 2015;Osborn et al., 2007).
PM has been developed and funded through a partnership between state legislators and educational leaders to create a special education initiative unfolded in 15 schools in 2000, and has since grown throughout the state (Hampton, et al., 2015;Wilson & Rychener, 2009). PM used a commercially available tutoring intervention, Reading Tutors (reading-tutors.com, 2007). This program provides lessons and materials, as well as assessments pertaining to alphabet, phonological awareness, phonics, high-frequency words, fluency and comprehension of reading. PM provides schools with over 450 tutor packets from the Reading Tutors program that are organized into the six categories stated above, covering key areas of literacy development cited by the National Reading Panel (Hampton et al., Wilson & Rychener). This study examined the growth of reading proficiency of a group of students with SLD in reading, compared to a group of non-disabled peers. We were interested in examining whether students with reading disabilities would benefit from supplemental intervention using PM, in addition to core instruction and special education services, when compared to typical nondisabled peers.

Overview of the Study
The current study was formed around replicating a previous study which examined the goal of comparing the reading gains of students in the Project MORE (Mentoring in Ohio for Reading Excellence) intervention program with the reading gains of similar students who had not received the PM intervention. The treatment condition participants in this evaluation were identified with Specific Learning Disabilities (SLD) in the area of reading using the state's common eligibility criterion (http://education.ohio.gov/Topics/Special-Education/Federal-and-State-Requirements). The participants in the comparison condition were same grade non-disabled peers. Each PM tutoring session consisted of a volunteer-tutoring program that was of 30 minutes duration delivered three times each week. The intervention was of a comprehensive design focusing on oral language skills, phonemic sound-symbol relationship, and structural analysis, as well as morphemic awareness, vocabulary building, and reading fluency.
As part of the evaluation, all second, third, and fourth graders were assigned to treatment or comparison conditions. Students in the treatment group were students identified as having SLD in reading who were further identified as being at least one grade level behind in their reading skills as confirmed by DIBELS benchmarking and progress monitoring. The treatment group was provided with PM tutoring for 30-minute sessions, three times each week over the course of an academic year. Students in the comparison group were students who were not identified as having a reading disability and were at grade level in reading at the beginning of the academic year, and did not receive the PM intervention.
This study reports the relative benefits of the PM tutoring program for use as a supplemental reading intervention for students with SLD using a regression discontinuity analysis. Using regression discontinuity analysis, we were able to examine statistically significant gains in reading proficiency for students in the treatment condition when compared to their same grade level typically performing peers. The performance of the students receiving the PM intervention will be detailed relative to outcomes of DIBELS Oral Reading Fluency given at grade level three times during the academic year. The initial benchmark score served as the pretest score and the spring benchmark score served as the posttest score.

Student Participants
The participants in this evaluation consisted of second, third, and fourth grade students from 8 schools from twelve rural districts in a Midwestern state. Two groups were formed into treatment and comparison cohorts from each grade at each school. Students in the treatment condition were identified as having SLD in reading using the state's common eligibility determination criteria. These students, across grades, are referred to as Cohort 1. Students in the comparison group were identified as being at grade-level benchmark in their respective grades, and collectively, will be referred to as Cohort 2. These groups are used to detail the demographic information; the analyses were conducted using grade level analyses of treatment and comparison groups.
In August of 2010 through spring of 2011, all students were screened using grade-level DIBELS ORF measures (Cohort 1: n=187; Cohort 2: n=291). Table 1 provides illustration of group sizes by grade. Students in Cohort 1 received the PM intervention from tutors trained in the PM program by local school staff who were trained by personnel from the county Educational Service Center (ESC) whose personnel administer and oversee the operations of PM. Detailed records were obtained over the 26 weeks of the intervention and were verified by members of the research team as well as through on-site observation and interviews with tutors and supervisory personnel at each of the 11 sites in order to assure accurate data entry. There were 187 students in Cohort 1 (96 female, 91 male), and in Cohort 2 there were 291 students (176 female, 115 male).In Cohort 1, 68 students were Caucasian, 7 were Hispanic, 16 were African American, and 3other.
In Cohort 1, 82 students received free or reduced lunch. For the students in Cohort 2, 214 were Caucasian, 21 were Hispanic, and 56 were African American. There were 169 students who received free or reduced lunch in Cohort 2.

Volunteer Tutors
For the PM intervention, volunteer tutors were used at each school site (n= 35). Volunteers were recruited from the community and varied in their age from 16-74 years old, gender, and previous education and work experience. Tutors can be high school students, retirees, college students, professionals, laborers, parents, and anyone who the school can recruit, train, and commit to the PM tutoring program (Hampton, et al., 2015;Osborn, et al 2007). Use of tutors with a diverse background is widely supported in the literature (Al-Otaiba & Pappamihiel, 2005;Caserta-Henry, 1996;Elbaum, Vaughn, Hughes, & Moody, 2000;Morris, Tyner, & Perney, J., 2000).
All tutors were trained by local school staff that was trained by supervisory personnel in charge of the state's Project MORE programming from the county ESC. Training consisted of one 2-hour session, and included treatment of fidelity in implementation of tutoring sessions conducted at the end of the training session and at the end of the first month of tutoring services. Any tutor who scored below the minimum 95% accuracy in the tutoring sessions based upon a checklist and inter-observer reliability were remediated in their training and were reassessed until they met the minimum accuracy level.

Project MORE intervention
PM provided the overall structure to deliver the reading intervention. The goal of PM is to supplement reading instruction for students who have been identified with disabilities in reading proficiency, with each participant having at least one IEP goal in reading (Hampton, et al., 2015;Osborn, et al, 2007). The program delivers structured tutoring to a student who is identified as a struggling reader via data derived from curriculum-based measurement screening at grade-level in reading. The PM intervention consisted of one-on-one tutoring by trained volunteer tutors. There were three sessions each week over 26 weeks, each lasting 30 minutes. All students in the PM intervention were given an initial screening using the Dynamic Indicators of Basic Early Literacy Skills-6 th Edition, Revised (DIBELS). Once a student's reading proficiency was determined, PM Sessions were created from the Reading Tutors reading program (reading-tutors.com, 2007). The Reading Tutors program contains 450 comprehensive lesson plans with teaching tips, instructional resource packets, formative and summative assessment, and resources in alphabet, phonological awareness, phonics, high-frequency words, fluency, and comprehension. The 450 tutor packets are organized into six categories covering six areas of literacy development cited by the National Reading Panel. They include alphabet recognition, phonological awareness, high-frequency word recognition, fluency, and reading comprehension. The Reading Tutors program conforms to the National Reading Panel's recommendations for successful reading programs (Osborn, et al, 2007). Students and tutors met in one-to-one tutoring sessions three times each week. A student may have as many as three different tutors in a week, but typically, tutors remained consistent throughout the year. The use of multiple tutors during the week could be problematic, but using an intervention system that is considered highly standardized (Vaughn & Wanzek, 2007).Tutoring sessions were generally held in a particular classroom within the school, but there were occasions in which tutoring sessions occurred in a hallway of the school.
Each session consisted of three instructional segments, with the first segment lasts for five to 10 minutes consisting of fluency building activities which included timed cold, warm, and hot reading, based on the premise of repeated reading. The second segment of each session focused attention on skills targeted at building reading comprehension by using predicting, previewing, and asking questions in advance of reading passages, followed by tutor reading aloud as the student follows along as a model of proficient reading. The final 10 segment focused on high-frequency response type reading games related to specific skill development based upon the specific needs of the student as outlined in the student's IEP. Activities included high response instructional games, manipulatives, and kits or cards to promote specific skill development. Areas of reinforced practice included work analysis, vocabulary, comprehension, study skills, and writing (Hampton, et al., 2015;Osborn, et al).

Data Collection
A team of trained graduate students traveled to each school to administer the DIBELS Oral Reading Fluency (DORF) screening were administered to each student and used as pretest and posttest scores to both the treatment and comparison groups.

DIBELS DORF. DORF is a standardized, individually administered test of accuracy and fluency with
connected text. The passages are calibrated for the goal level of reading for each grade level. Student performance was measured by having students read three passages aloud, each for one minute. ORF scores are derived from the number of correct words per minute, is considered the oral reading fluency rate. The DORF test-retest reliability over the course of several days is .90 (DIBELS Technical Reports, 2003).
The DORF is highly correlated with the Woodcock-Johnson reading subtests among students with SLD (r = .89) (DIBELS Technical Reports). Additionally, the DORF is significantly correlated with the state high-stakes annual assessment for fourth grade students (Vander Meer, Lentz, & Stoller, 2005).

Regression Discontinuity Research Design
The effectiveness of the PM intervention was evaluated using a regression-discontinuity (RD) research design. This quasi-experimental design is a particularly strong alternative to the use of randomized control research design to evaluate the efficacy of an intervention (Trochim, 1984). It is appropriate to use RD when the group receiving the condition and the comparison group are intentionally selected to differ in ability as assessed by a quantitative criterion prior to the introduction of the condition. When used with a strictly enforced cutoff score for inclusion into the condition group, RD provides a robust alternative to randomized experimental design, with additional benefit of not having to construct a comparison group by denying the intervention to those who need it (Gersten & Dimino, 2006;Vaughn et al, 2009).
The core assumption of RD is that the relationship between the pretest criterion score and the posttest outcome measure would be the same for all students. A program effect is obtained by examining the degree to which the regression line for students in the condition group differ from the expected line that is based in the pre-post relationship in the comparison group. RD can be used to determine both if a main effect exists for the condition group and if an interaction effect also exists. A main effect is evidenced when the regression line for the condition group is shifted above the expected line. This shift raises the regression line of the condition group above the line of the comparison group by a constant value. When the lines are prepared visually, the result is a vertical gap between the two lines at the location of the cutoff score on the x-axis. An interaction effect indicates that the effect of the condition differs across members of the treatment group based on their pretest scores (Gersten & Dimino, 2006).
The validity of RD is dependent on meeting five assumptions (Trochim, 2007). First, the cutoff score criteria have been consistently applied in assigning students to the condition and comparison groups. Second, the statistical model applied to the data contains sufficient terms to provide an unbiased estimate of the main and interaction effects. Third, there are a sufficient number of observations in the comparison group to reliably estimate the pre and posttest regression line.

58
Journal of Education and Human Development, Vol. 9, No. 1, March 2020 Fourth, subjects were chosen from a group with a continuous distribution of pretest scores, thereby avoiding selection bias. And lastly, the PM intervention is provided to the condition group in a uniform manner (i.e. equivalent sessions over the weeks for all participants).
Each of these assumptions has been attended to in the design and data analysis of the within study.

Results
To address our question of interest, examining whether students with reading disabilities would benefit from supplemental intervention using PM, in addition to core instruction and special education services, when compared to typical nondisabled peers. Pretest and Posttest variables included in this analysis was the appropriate grade level DORF. Each of the grade level pretest measures was administered in the spring of 2010, and the posttest measures were administered in the spring of 2011. Results of the regression discontinuity analyses are presented in the following figures by outcome and grade level. To aid in the visual presentation of the results, these data used the cutoff score which was determined from the highest PM DORF score at the fall benchmark from each grade. Each participant in the comparison group had a DORF score that was higher than those of the condition group and as students in general education; they did not receive the PM intervention. In order to find scores that are not only statistically significant, but practically advantageous, the condition group must have significantly higher reading gains (DORF) than their counterparts in the comparison group. It is generally considered difficult for an intervention to produce this type of reading gain at a statistically significant level (Gersten & Dimino, 2006;Wilson et al., 2007). To ensure the most rigorous cut score to determine the relative shift in group results, we used the highest DIBELS score at each grade level from the fall benchmark to serve as the position from which we could determine reading gains. In order to illustrate significant reading gains over an academic year for the condition group relative to the comparison group. The intervention group which has the lower pretest scores (PM students), must have significantly higher gains than the comparison group of students who had higher pretest scores. The results suggest that the DIBELS posttest scores for the condition group increased at a higher rate than for the comparison group during the academic year.

Figure 1. Second Grade Results
For students in the 2 nd grade PM condition group, fall-to-spring reading gains on the DORF increased at a statistically significant rate than those scores of the comparison group (T= 8.02, p< .00**). As shown in the scatterplot ( Figure 1) the regression line is significantly different in position 51 (the highest DORF pretest score) on the x-axis and in slope to the line for the higher responder group.

Figure 2.Third Grade Results
For students in the 3 rd grade PM condition group, fall-to-spring reading gains on the DORF as evidenced by the posttest DORF scores increased at a statistically significant rate than those scores of the comparison group (T= 4.79, p< .00**). As shown in the scatterplot ( Figure 2) the regression line is significantly different in position 71(the highest DORF pretest score) on the x-axis and in slope to the line for the higher responder group. Journal of Education and Human Development, Vol. 9, No. 1, March 2020 For students in the 4 th grade PM condition group, fall-to-spring reading gains on the DORF as evidenced by the posttest DORF scores increased at a statistically significant rate than those scores of the comparison group (T= 2.67, p< .00**). As shown in the scatter plot ( Figure 3) the regression line is significantly different in position 79 (the highest DORF pretest score) on the x-axis and in slope to the line for the higher responder group. It is important to note that the analysis of the participants in 4 th grade revealed the lowest differences in the gain scores, although they were still statistically significant.

Discussion
This evaluation reports the response to the PM reading intervention program of students with SLD in reading who were below benchmark in their respective grade level reading screening scores at the beginning of the academic year. Using a regression discontinuity design the research team evaluated the performance at the end of the academic year of students with SLD relative to students without reading difficulties at the same grade level who received the same core reading instruction in the classroom. The students in the condition group in received their special education services in addition to the core instruction, and the PM intervention was used as a supplement to those SPED services.
When comparing the ORF scores for each grade, all grade levels reported statistically significant growth in overall reading proficiency when compared to students without disabilities who did not receive the PM intervention. Students with SLD in the condition group reported the largest growth in second grade and students with SLD in the condition group in fourth grade reported the lowest growth rate, while all were statistically significant. This effect can be best seen in the scatter plots presented. In each grade-level scatter plot presented reveal a larger shift or discontinuity when plotted along the comparison group of students without disabilities who did not participate in the intervention. The group comparisons suggest that for each grade, students with SLD in reading made significant progress in their reading proficiency during the academic year over their grade level peers without reading difficulties who did not receive the PM intervention.
These comparison results may suggest important outcomes for students with reading disabilities who receive supplemental volunteer tutoring in reading during their instructional day. These students who are most in need of support in reading development as evidenced by their identification as having a SLD in reading showed significant progress after an academic year program of supplemental reading instruction provided by trained volunteer tutors. This may suggest that the delivery of supplemental instruction can be provided by trained volunteers who may not be professional intervention specialists. An interesting question that may be a result of this evaluation can be, would the format and delivery of the PM intervention by volunteer tutors constitute a special education?

Implications for Practice
One practical implication emerging from this study is the need for schools and communities to emphasize the potential benefits of utilizing a trained cadre of volunteer tutors within their schools. This would assist educators and administrators emphasize educational partnerships and cooperative associations with the community at-large that could invigorate community involvement in their local schools. Another implication for practice lies in the notion of supplemental reading programs that do not take the place of core instruction or special education services can serve to support those services, this is extremely timely as populations of students with disabilities continues to grow, and the ability for all educators to support those students in effective and sustained one-one instruction becomes more difficult. While the results of this present study represent the outcomes for students with SLD in reading, the notion of one-on-one volunteer tutoring could help students who struggle, yet are not identified, and can be seen as helpful for all students who may need extra help learning new concepts.

Limitations
There are some limitations that are present in this evaluation. Our student participants were selected from several school districts, but there was limited diversity in the population and there was limited availability of urban students. We were able to utilize both benchmark and growth data to differentiate the groups, but we did not have the benefit of developing a random-control experimental or quasi-experimental design. To counter this significant limitation we did utilize regression discontinuity design that the literature suggests is robust in analyzing data without the requirement of random assignment of condition and comparison groups (Gersten & Dimino, 2006;Trochim, 2007;Vaughn et al, 2009). Future examination using the standard criteria of experimental or quasi-experimental design while examining the PM intervention effects for students with SLD in reading and students who struggle with reading development who do not present with SLD could benefit all students in reading development.