You are viewing the EdNews Blog archives.
These archives contain blog posts from before June 7, 2011
Click here to view the new First Person section of Chalkbeat Colorado

Heritage, related reports on Fla. don’t withstand scrutiny

Posted by Nov 30th, 2010.

Many of you might recall an article last month in the Denver Post that discussed the release of a report from Colorado Succeeds, a new organization of business leaders focused on education issues in the state. The report was called “Proving the Possible: A case study of Florida’s K-12 education reforms and lessons for Colorado.”  At the time, I was intrigued, because our Think Twice think tank review project had just asked professor Madhabi Chatterji of Columbia University to review a similar report published by the Heritage Foundation called Closing the Racial Achievement Gap: Learning from Florida’s Reforms.

Professor Chatterji’s review of the Heritage report was released this morning. Before returning to the Colorado Succeeds report, let me quote the key six paragraphs from the press release about the Heritage report:

Did a collection of Florida education policies—ranging from grade retention to school choice and virtual schools—improve achievement and narrow the test-score gap? A recent Heritage Foundation report is part of a larger campaign to convince us that the answer to that question is “yes”—but a new review finds fundamental flaws in the Heritage report that render its conclusions untenable.

The Heritage report, authored by Matthew Ladner and Lindsey Burke, contends that Florida’s “far-reaching” education policies have caused test scores to increase and the achievement gap to narrow. In particular, the report focuses on fourth grade reading scores on the National Assessment of Educational Progress (NAEP). These research claims are also made by Dr. Ladner, the Vice President of Research at Arizona’s free-market Goldwater Institute, in a dozen other reports and articles similar to the Heritage report reviewed by professor Chatterji.

The claims, however, do not withstand scrutiny. “The report’s key conclusions are unwarranted and insufficiently supported by research,” Chatterji states in her review. Most importantly, she points out the very direct effects of the state’s grade-retention policy, causing the report’s comparisons to be largely meaningless. By analogy, consider growth in height instead of growth in test scores. If two states wanted to measure the average height of their fourth graders, but one state (Florida) first identified the shortest 20% of third graders and held them back to grow an additional year before measurement, the study’s results would not be useful.

That, in brief, is the key problem that professor Chatterji identifies with the Heritage report. Florida’s retention policy, instituted in 2002, focuses on third graders, who are held back when their reading scores are low. The Heritage report focuses on NAEP fourth grade reading scores. Low scoring readers—mostly black and Hispanic—were screened out of grade 4 tests, which resulted in inflated and erroneous fourth grade scores. “Chatterji’s review explains very clearly why the simplistic comparison of fourth graders before and after Florida’s grade retention policy is a predictable and worthless exercise,” says Kevin Welner, professor of education at the University of Colorado and the director of NEPC.

The review also points out that NAEP scores at other grade levels and even NAEP scores in fourth grade math do not show the same jump as NAEP fourth grade reading. That is, the report cherry-picks the best data. Also, even if scores in Florida are in fact increasing, the report’s methods are too weak to allow for a causal inference. The report uses only descriptive test score trends to compare states and then make sweeping generalizations. Moreover, many other changes occurred in Florida during the period analyzed, including the phasing in of one of the nation’s most ambitious class-size reduction reforms—yet the report never mentions these other possible causes of any improvements.

“In sum, the report’s analyses are highly biased and of very limited value,” Chatterji concludes. “The major elements of Florida’s education reform policies are in need of continuing and more careful examination, individually and collectively, before they can be recommended for wider policy adoption.”

So that’s the Heritage report.  What about the Colorado Succeeds report? It turns out that it is essentially the same as the Heritage report and, in fact, the same as similar reports published by free-market think tanks in New Mexico, Arizona, Wisconsin, Oklahoma, Utah, and Indiana, as well as by the Hoover Institution, the Pacific Research Institute, and articles in National Review Online and – but with a Colorado focus added on. In each of those (but not for the Colorado Succeeds report), Dr. Ladner was listed as either author or co-author.

In sum, the Colorado Succeeds report, was (whether the organization wanted to take credit for it or not) part of a larger campaign by Dr. Ladner to market the Florida reforms. Unfortunately, as shown by Professor Chatterji, the marketing was for a seriously flawed product.

Popularity: 9% [?]

3 Responses to “Heritage, related reports on Fla. don’t withstand scrutiny”

  1. Chad Hauser says:

    It is nice to see someone looking at these “studies” that show how much radical educational changes improve education. While I agree with some of the new thinking – for instance, I think if the threat of retention were real, parents might wake up and push their children to try harder academically – you have to be very, very careful when looking at ANYTHING Heritage Foundation publishes.
    For those who are unfamiliar, the Heritage Foundation’s stated mission is to “formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.” Although this may sound good in theory, what it really means is they are a hyper-conservative group. As many in education know by now, the conservative idea of improving education is cutting spending on everything from building maintenance to teacher retirement and giving people vouchers to go to private (often religious) schools. If the Heritage Foundation can publish a study that shows that public schools are failing and “virtual schools” are doing a better job, they will certainly be happy to do so.

  2. Tim Taylor says:

    “Did a collection of Florida education policies—ranging from grade retention to school choice and virtual schools—improve achievement and narrow the test-score gap?”

    The question being asked is the right one, and needs to be answered. Mr. Welner, citing a newly-released critique by Professor Madhabi Chatterji, concludes that the answer is no. Colorado Succeeds, confident from clear student performance data, concludes that the answer is an emphatic yes. I encourage all to read our “Proving the Possible” case study to understand our reasoned, data-driven perspective and decide for themselves.

    I hope we can start with the non-controversial premise that the status quo is unacceptable. Currently in Colorado, only 72% of our students graduate from high school and only 51% of Hispanic students earn a high school degree. Our report is designed to create a conversation and explore the potential of Florida’s portfolio of reforms.

    The crux of the criticism of our report seems to be whether or not certain reforms that we highlight are legitimate and measurable factors in the dramatic increase in student achievement that Florida has enjoyed. Using NAEP data as the measuring stick for reading performance in both states, the scores speak for themselves. There is no credible challenge to the fact that, between 1998 and 2009, fourth grade reading scores for Florida’s Hispanic students increased by 25 points; in the same period, Colorado’s fourth grade reading scores increased by only 3 points. As our report indicates, at this rate it would take nearly 70 years for our Hispanic students to match their Florida peers.

    Our education system is designed so that in grades K-3, students learn to read, and from the fourth grade on, they read to learn. Florida passed a straightforward law to ensure that students move on from third grade only when they have the demonstrated grade-level reading proficiency. That means two things: Florida had to strengthen its literacy delivery in preschool through third grade; and students who do not demonstrate proficiency do not move from third to fourth grade until they do. This policy has lead to significant and lasting gains in reading proficiency.

    The assertion that low scoring readers—mostly black and Hispanic—were screened out of grade 4 tests, which resulted in inflated and erroneous fourth grade scores, is nonsense. First, very few students are actually retained under the Florida policy – and to be clear the goal is literacy not retention. Second, the students are not retained indefinitely, therefore when the students move on to fourth grade and earn high test scores, they are not inflated and erroneous – they are simply demonstrating proficiency. Finally, the fact that the improved reading proficiency holds over the long-term and illustrates its effectiveness in improving student performance. On the most practical level, let alone the moral case, it is simply the right thing to do.

    This common sense policy and the positive results it has produced is dismissed as a “worthless exercise” by Chatterji, who did not offer another means by which to ensure students have properly learned to read before they are expected to read to learn. So no, Florida’s “move on when reading” policy cannot be seriously considered a worthless exercise. If they did the same thing, they would have gotten the same result. Instead, they have seriously improved student achievement in fourth grade reading proficiency for Hispanic students. There is nothing worthless about that.

    Frankly, I glean just as much from what Chatterji did not say, as for the critical things that she did. Being the consummate professional, she is somewhat measured in her claims. Though the implication is there and her personal opinion is clear, she was careful not to say that Florida’s cadre of reforms didn’t cause these gains. This is important because we agree with her and all who say that no one policy reform is a definitive silver bullet–a cure all. But unlike Chatterji, we think it’s clear that the portfolio of reforms are to be credited to Florida’s remarkable progress, and more than any other policy, the “move on when reading” practice did the most to identify the root cause and intervene early enough to rescue students from almost sure failure.

    The real question is, what can we do in Colorado to achieve such dramatic and enduring improvement to our own broken K-12 education system? Doing nothing is not an option, nor is doing more of the same. Substantive education reform is not a request from the business community, it is a demand. Colorado Succeeds surveyed the nation for proven models of reform from which our state and our students can learn. Florida has demonstrated the most dramatic results and Colorado should be open to these ideas or present a better plan.

    Nearly thirty percent of Colorado 4th graders score below basic on the NAEP reading assessment. These children are passed-on to the 5th grade where they are expected to read to learn – a skill they do not possess. We are failing to impart even the most basic skills. How do we expect these children to compete in a competitive global economy? If this doesn’t outrage the general public it certainly terrifies the business community; our workforce pipeline is riddled with leaks and the status quo continues to defend it and wash their hands in the spray.

    I hope any futher discussion here is productive in exploring policy levers that will improve student performance, as opposed to politicizing and posturing. We don’t need any more of that. What we need are solutions that prepare our kids to succeed.

    • Kevin Welner says:

      As Mr. Taylor says, Professor Chatterji is indeed a consummate professional and measured in her claims. She provides a detailed and clear explanation for how the retention of Grade Three students invalidates the claims of Heritage (and, by extension, Colorado Succeeds) about Fourth Grade scores. During the period studied, the retention rates varied from 14% to 23% (disproportionately African American and Latino). To me, that is not, as Mr. Taylor describes it, “very few students.” And as Chatterji explains, it accounts for not just the higher 4th grade scores but the closing of the racial gap in those scores. Using the height analogy, if we hold back short Latino 3rd graders and give them another year to grow before going on to be measured as 4th graders, we are not making those kids grow any faster – we’re just delaying the measurement until they’ve grown another year. THAT is why the study is a “worthless exercise.”

      Professor Chatterji’s review only briefly mentions the research on the effects of grade retention (a policy that Mr. Taylor is promoting). But that research is longstanding and very clear: it doesn’t help learning AND it increases the likelihood of dropping out before graduation. There certainly are grade retention policies that are combined with solid interventions (e.g., support over the summer for retained students or those threatened with retention). But retention itself is harmful. Of course, even if this research weren’t so overwhelming, the debate over the merits of grade retention does not affect at all the cohort measurement problem that Chatterji identifies.

      Where I agree with Mr. Taylor – beyond his praise for Professor Chatterji – is his call to read the Colorado Succeeds report – but please also read the Chatterji review along with it. The review is of the Heritage report, not the Colorado one. But all evidence points to the conclusion that the Colorado Succeeds report is essentially an unattributed restatement of the Heritage report and the other reports from Dr. Ladner, so the Chatterji review serves as a perfect complement to the Colorado report as well.

Leave a Reply

Colorado Health Foundation Walton Family Foundation Daniels fund Pitton Foundations Donnell-Kay Foundation