Holly Yettick is a doctoral student in the Educational Foundations, Policy and Practice program at the School of Education at the University of Colorado in Boulder.
How much achievement does $6 million-worth of tutoring buy? Not as much as you might think.
A report submitted to the Colorado Department of Education in June received little attention at the time but is now relevant as Obama pushes for the reauthorization of No Child Left Behind. In the report, contract evaluators for the Colorado Department of Education concluded: “Across all analyses, few significant differences were found” between the achievement of Colorado students who received “supplemental educational services” (tutoring mandated by No Child Left Behind) and a comparable group of students who did not.
Other highlights of the report:
- Depending on which group provided the tutoring, costs varied widely, from $20 to $89 per hour. (The average cost was $42 per hour or $1,123 per child in federal Title 1 funds.)
- Success records also varied widely, but the majority of tutors were basically placebos in that their students performed almost identically to a comparison group of peers who were not tutored.
- Tutoring disproportionately benefitted those with fewer challenges. Native English speakers made greater gains in math than a comparison group of similar students who did not receive tutoring. English learners did not. Students who were not in special education made greater reading gains than a comparison group that received no tutoring. Special needs students did not.
“Supplemental educational services” are available to low-income students attending schools that miss making “Adequate Yearly Progress” for three years in a row. These optional sessions take place outside of school hours. Although the students who received the tutoring were enrolled in 15 different Colorado districts, more than three quarters came from Denver Public Schools. These services cost about $6 million in Colorado. Nationwide, the program serves half a million students and costs about $2 billion per year.
Nationwide, supplemental educational services are provided by a range of groups including for-profit companies, non-profits and school districts themselves. One thing that surprised me given the cost of the tutoring is that tutors are apparently not required to have four-year college degrees.
In Colorado, more than half of the 4,858 students served in 2008-09 (the most recent year for which data is available) got tutored by three for-profit companies: Tutor Train, (25%) Club Z! (20%) and Learn it Systems (10%). Yet a 2010 research synthesis conducted by The Center for Educational Partnerships at Old Dominion University found that tutoring provided by school districts cost less than commercial tutoring and produced better results.
While certainly discouraging, Colorado’s tutoring results are not unique. The Old Dominion research synthesis found that supplemental educational services had very small effects on reading and math achievement. By contrast, the Comprehensive School Reform program that has mainly been eliminated by recent reauthorizations of the Elementary and Secondary Education Act (NCLB’s official name) produced much stronger results at a much lower cost.
Considering NCLB’s emphasis on accountability for schools, it is also disturbing that the Old Dominion researchers concluded that:
Despite mounting evidence that SES is far less effective than previous Title I policies, we are not aware of a single instance in which a provider has been removed from an approved state list on the basis of failing to demonstrate positive effects on student achievement.
Under Obama’s blueprint for reauthorizing NCLB, supplemental educational services would be optional. Based on the research, I think that this is a good idea. If a specific tutoring provider is getting results, by all means, keep it on. But mandated tutoring was a good idea that just did not demonstrate good results.
Popularity: 56% [?]
Well I must admit I don’t have any insight into how Colorado administers S.E.S. but it does not appear that the state is very active in ensuring that the services are monitored. The Rand Report among other independent studies have shown S.E.S. tutoring if properly conducted has a significant impact on student learning. The author is critical but does not cite any report other that conducted in Colorado and seems to believe ( without any data) that school districts manage tutoring well; Interesting comment given the fact that the same districts often can’t reach very limited goals i.e. of 30% proficiency in math to be considered as having made progress. Approximately $300 million dollars was allocated for all S.E.S tutoring across the U. S. about $1 per year for the population or less than a penny a day. The proposed cost of the trial of one alleged terrorist in NYC $300 million, interesting set of priorities. I will not defend profit based S.E.S. companies and certainly the profit motive does or can have an negative impact however, I would caution the author to get facts and not condemn a whole program which affords free tutoring to families who cannot afford it.
Given the differential efficacy of tutoring, to me this post suggests tutoring is in and of itself a kind of alternative delivery vehicle of learning but one closely allied with traditional methods in a classroom. That challenged learners enjoyed fewer gains from tutoring suggests a whole host of probable causes–many of which may be related to lack of success in a traditional classroom environment.
But the almost 5000 students who were tutored out of a state population of 750,000 represents a pitifully small relative number of students. It could turn out the spending of $6 million is worth the effort–perhaps those 5000 would have ended up costing the state more than six million in the long run without intervention. And speaking of the intervention, all we know from this interesting study is that tutoring had a dubious impact when measured by standardized tests in a very short-term time frame. Perhaps there were differences made in the lives of students from tutoring beyond the purview of CSAP, either in quantitative measures of knowledge attainment or long-term behavioral changes.
One thing seems clear: for-profit tutors can earn a lot of money thanks to a government program. I wonder why such an organization is supposed to be better than a school or district itself offering supplemental programs with meaningful accountability measures in place. Perhaps the answer can be found in the modern American myth of belief in the free market somehow being able to offer up value-free and superior results in every field of human endeavor rather than the governmental entities staffed by congenital slackers abetted and supported by their socialist political masters.
Alan:
The RAND report did find positive results. But what I cited was a (2010) research synthesis (published by the Center for Educational Partnerships at Old Dominion University.) Here is the link again:
http://education.odu.edu/tcep/docs/FinalSESReport.pdf
I agree with you that management is a problem given the quotation in which the Old Dominion researchers noted that they knew of no providers that had been eliminated from state lists. I also agree that the tutoring should in some cases continue. I just don’t know that it should be mandated . It should be one of several options for communities and schools.
As stated above, if specific providers are getting good results, they should be kept on. Further, perhaps new providers should be required to adopt at least some of the characteristics that the Old Dominion researchers found to be associated with increased achievement: “(a) the use of school district providers; (b) experienced, well-trained tutors with four-year degrees; (c) a national or prescribed curriculum; and, (d) one-to-one tutoring for reading instruction.”
Jeffrey:
I definitely agree that tutoring could be providing benefits that don’t show up on tests. The authors of the Colorado evaluation are careful to state this in their report. As for the small numbers of students tutored, I would say that is an additional real-world implementation problem. Why aren’t more eligible kids attending? Are the schedules inconvenient with kids missing buses if they get tutoring? Why don’t more high school students participate?
As a non-profit with two decades of tutoring students and two years as an SES Provider, it is clear that a well designed and staffed tutoring program can change attitudes of students and get them again engaged in their education. It can help them from falling further behind, but catch-up growth is a different challenge.
As a professional private tutor for six years, I have seen that high quality tutoring can produce a large catch up growth in a short period of time. When I say high quality tutoring, I mean much more than just knowing the subject being tutored. There are specific skills needed to be an effective tutor. These skills are generally not taught to tutors by most SES programs. The skills of a teacher are not the skills of a tutor.
In addressing reading in the younger grades, the 10,000 hours or ten years rule described in the book “Outliers” applies. It takes a lot of effective time on task to master skills, and a fifty hour single year tutoring effort is a small fraction of the time needed to show a large benefit.
I have seen for-profit tutoring programs, and they do seem to be generally for profit.
NCLB Tutoring Much More Effective than Yettick Thinks
Holly Yettick’s “NCLB tutoring expensive, ineffective” misses the mark. She correctly cites disappointing CSAP data for the 2008 – 2009 school years’ SES participants, but fails to acknowledge that the soon-to-be-discarded CSAP lacks the sensitivity and granularity to capture progress made by some of our lowest performing students – the very students who are most likely to be enrolled in SES programs.
Summer Scholars – one of the many Colorado not-for-profit organizations offering after-school tutoring through SES – carefully tracks student outcomes by pre- and post-testing students with the nationally recognized Dynamic Indicators of Basic Early Literacy Skills (DIBELS.) This instrument measures the foundations of literacy that are frequently underdeveloped in students who are farthest behind grade level in reading. It’s particularly helpful in measuring gains for English Language Learners – again, students who are very likely to be enrolled in SES programs.
During the same 2008 – 2009 school year that CSAP scores did not show significant improvements, our DIBELS pre- and post- tests demonstrated that more than 50 percent of our students made significant gains in “oral reading fluency” – a more sensitive measurement than the CSAP’s broad-stroke proficient/ not proficient/ advanced ratings.
Data folks don’t always like anecdotes — but personal stories are anecdotal evidence of results. Just earlier this week, the mom of one of our Summer Scholars kids popped into my office to say thank you: “Jason struggled with reading in first grade, but started Summer Scholars that year – in 2008. You guys helped him so much that he’s gone from the bottom of his class to now, he’s in third grade and reading at a fifth grade level. This past year, he received an award for most improvement in reading. I’m so proud of him and so glad for Summer Scholars.”
We at Summer Scholars are proud of Jason, too, and of the SES programs we’ve been able to implement through our collaboration with Denver Public Schools. Our mission is to “narrow the achievement gap” – and our data show we do it well.