Many of you might recall an article last month in the Denver Post that discussed the release of a report from Colorado Succeeds, a new organization of business leaders focused on education issues in the state. The report was called “Proving the Possible: A case study of Florida’s K-12 education reforms and lessons for Colorado.” At the time, I was intrigued, because our Think Twice think tank review project had just asked professor Madhabi Chatterji of Columbia University to review a similar report published by the Heritage Foundation called Closing the Racial Achievement Gap: Learning from Florida’s Reforms.
Professor Chatterji’s review of the Heritage report was released this morning. Before returning to the Colorado Succeeds report, let me quote the key six paragraphs from the press release about the Heritage report:
Did a collection of Florida education policies—ranging from grade retention to school choice and virtual schools—improve achievement and narrow the test-score gap? A recent Heritage Foundation report is part of a larger campaign to convince us that the answer to that question is “yes”—but a new review finds fundamental flaws in the Heritage report that render its conclusions untenable.
The Heritage report, authored by Matthew Ladner and Lindsey Burke, contends that Florida’s “far-reaching” education policies have caused test scores to increase and the achievement gap to narrow. In particular, the report focuses on fourth grade reading scores on the National Assessment of Educational Progress (NAEP). These research claims are also made by Dr. Ladner, the Vice President of Research at Arizona’s free-market Goldwater Institute, in a dozen other reports and articles similar to the Heritage report reviewed by professor Chatterji.
The claims, however, do not withstand scrutiny. “The report’s key conclusions are unwarranted and insufficiently supported by research,” Chatterji states in her review. Most importantly, she points out the very direct effects of the state’s grade-retention policy, causing the report’s comparisons to be largely meaningless. By analogy, consider growth in height instead of growth in test scores. If two states wanted to measure the average height of their fourth graders, but one state (Florida) first identified the shortest 20% of third graders and held them back to grow an additional year before measurement, the study’s results would not be useful.
That, in brief, is the key problem that professor Chatterji identifies with the Heritage report. Florida’s retention policy, instituted in 2002, focuses on third graders, who are held back when their reading scores are low. The Heritage report focuses on NAEP fourth grade reading scores. Low scoring readers—mostly black and Hispanic—were screened out of grade 4 tests, which resulted in inflated and erroneous fourth grade scores. “Chatterji’s review explains very clearly why the simplistic comparison of fourth graders before and after Florida’s grade retention policy is a predictable and worthless exercise,” says Kevin Welner, professor of education at the University of Colorado and the director of NEPC.
The review also points out that NAEP scores at other grade levels and even NAEP scores in fourth grade math do not show the same jump as NAEP fourth grade reading. That is, the report cherry-picks the best data. Also, even if scores in Florida are in fact increasing, the report’s methods are too weak to allow for a causal inference. The report uses only descriptive test score trends to compare states and then make sweeping generalizations. Moreover, many other changes occurred in Florida during the period analyzed, including the phasing in of one of the nation’s most ambitious class-size reduction reforms—yet the report never mentions these other possible causes of any improvements.
“In sum, the report’s analyses are highly biased and of very limited value,” Chatterji concludes. “The major elements of Florida’s education reform policies are in need of continuing and more careful examination, individually and collectively, before they can be recommended for wider policy adoption.”
So that’s the Heritage report. What about the Colorado Succeeds report? It turns out that it is essentially the same as the Heritage report and, in fact, the same as similar reports published by free-market think tanks in New Mexico, Arizona, Wisconsin, Oklahoma, Utah, and Indiana, as well as by the Hoover Institution, the Pacific Research Institute, and articles in National Review Online and Foxnews.com – but with a Colorado focus added on. In each of those (but not for the Colorado Succeeds report), Dr. Ladner was listed as either author or co-author.
In sum, the Colorado Succeeds report, was (whether the organization wanted to take credit for it or not) part of a larger campaign by Dr. Ladner to market the Florida reforms. Unfortunately, as shown by Professor Chatterji, the marketing was for a seriously flawed product.
Popularity: 10% [?]