You are viewing the EdNews Blog archives.
These archives contain blog posts from before June 7, 2011
Click here to view the new First Person section of Chalkbeat Colorado

Why I don’t trust education research

Posted by Feb 8th, 2011.

Research is hailed as the Holy Grail in the world of education. Staring a sentence with the words “research shows” is aimed at sticking a dagger in the heart of an opponent’s argument. Increasingly, though, I am finding reasons not to trust education research.

Over time I have noticed that many researchers’ on the left and right invariably produce studies that support their ideological beliefs. This causes me concern.

Yesterday’s release of a study by CU’s National Education Policy Center on the Los Angeles Times’ 2010 analysis of value-added ratings of L.A. teachers provides the latest example of why I’m cynical about research. I’m not saying there is anything wrong with the study, Due Diligence and the Evaluation of Teachers. Others have pointed out flaws in the newspaper’s methodology, though the Times continues to defend its work.

What bothers me is this: NEPC is undoubtedly a think tank with a progressive, left-of-center bent. Looking through a list of the center’s studies, I am hard-pressed to find one that does not reinforce the beliefs of people with that ideological inclination. Since December, I’ve received email alerts about NEPC studies casting doubt on charter school quality, teacher evaluation methodologies, school report cards and international comparisons of U.S. student performance.

So when I hear NEPC has a new study, I know generally what its conclusions will be before I read it.

I know, like and respect Kevin Welner, the professor who heads NEPC. He used to write for this blog. I do not doubt his integrity. That’s why I am puzzled by this phenomenon, which is by no means limited to NEPC.

On the other side of the ideological spectrum, I know Harvard’s Paul E. Petersen will produce research favoring vouchers and charter schools. Ditto Jay P. Greene from the University of Arkansas.

What’s wrong with this picture? I want to see one of these researchers or their think-tanks produce a study that cuts against the grain; that calls into question the beliefs of the researchers and their funders. Until that happens, I will take everything they write with an enormous grain of salt.

I may disagree with some of researcher Diane Ravitch’s conversion experience conclusions but I credit her with having the enormous courage to rethink her positions and then go very, very public with her mea culpa and her change of heart.

Popularity: 27% [?]

18 Responses to “Why I don’t trust education research”

  1. Dr. Neil Schaal says:

    Yes indeed. A grain of salt will make the medicine go down, the medicine go down, the medicine go down….. (While singing the tune in the background)

  2. Holly Yettick says:

    I would urge people to read the entire “Due Diligence and the Evaluation of Teachers” report before drawing conclusions. The lead author, Derek Briggs, is a statistician and psychometrician and one of the most careful and skeptical scholars I have met. In this report, Briggs and co-author Ben Domingue perform a valuable and time-honored scientific role in that they re-analyze the data upon which a peer has based his results. Hopefully, others will follow suit with equally careful re-analyses. That’s how science evolves. Value-added teacher evaluation models will get stronger, not weaker, as a result.

    In their report, Briggs and Domingue warn that the debate surrounding value-added teacher evaluation models has too often devolved into “two extreme, but mistaken, positions:”

    “First it is argued that unless value-added models can be shown to lead to perfect classifications of effective and ineffective teachers (however defined), they should not be incorporated into the high-stakes decisions likely to accompany teacher (and school) evaluations. This argument is a false one because, as a number of researchers have pointed out, relative to a status quo for teacher evaluations that most people regard as unacceptable, classifications driven at least in part by value-added estimates do not need to be perfect in order to constitute an improvement.

    The second mistaken position is that any critique of the inferences being ascribed to a value-added model constitutes an endorsement of the status quo. One may well agree that test scores need to be given a normative context before they can be interpreted for evaluative purposes, but not agree on what that context should be, or whether a particular value-added model has produced it. The use of standardized test scores to evaluate teachers involves making difficult choices in which there are invariably some tradeoffs between decisions that might be optimal from the perspective of estimating an unbiased causal effect, but not optimal from the perspective of crafting an educational accountability policy with a coherent theory of action. The obligation for those with expertise in statistical and econometric methods is to be explicit and transparent about these choices, so that policymakers and administrators have the information they need to weigh the costs and benefits, and so that all stakeholders have an entry point to the policy debate.”

    This is about as ideological as the report gets.( It is almost entirely a technical analysis.) To me, these statements lean neither left nor right but, rather, urge readers to avoid simplistic either/or positions on value-added teacher evaluation model.

  3. David Hazen says:

    I listened to Richard Allington describe research in reading several years ago. He described educational research as generally unreliable. What he has done in the reading world is separate the wheat from the chafe. I very much trusted his evaluations on what works and what doesn’t when it comes to reading.

  4. Derek Briggs says:

    All I ask is that you and others read the report before passing judgment or making insinuations about our motivations. Ben and I don’t have an ideological dog in this hunt. In fact, I’m supportive of the notion that student achievement on standardized tests should be part of the information used to evaluate teachers. Just read the report–we worked long and hard to produce it.

  5. Van Schoales says:

    It would be interesting to see the results if NEPC were to aim their critique guns at the research briefs that groups like NEA and other adult interests do to promote their agenda, i.e. the problem with public education is largely everything but public education…. poverty, healthcare or the lack of money.

    My concerns about what NEPC does lies less with any particular NEPC critique (most are thoughtfully done and technically correct) and more with what they choose to go after. NEPC’s focus appears totally driven by an agenda to attack any study that promotes charters, other school choice, accountability for results, money following kids, any form of tracking, turnarounds, etc.

    Hey I’m all for the discussion and debate as I’ve said many times here, I just wish that NEPC would stop claiming that their work is pure and objective. NEPC is a sophisticated advocacy group funded by the NEA.

    The other problem that I have with their work is that they never seem to have much constructive to say about what we should do to dramatically improve public education (I know it’s not their job…they are supposed to stay in the ivory tower) except of course more money and economic integration. While in many cases I’m for those things, in practice they are often impossible (particularly now) or will only make a slight improvement in achievement. What about teacher evaluation? How should we hold schools accountable for results? What are the results we should hold them accountable for? Etc, etc….

    It’s really too bad that there is not a more objective scorekeeper that can review the hundreds of briefs and policy papers that are designed to push a particular agenda whether it’s to defend the current system or promote a new way of doing public education. You’d hope that some of our universities would be better able to assist here. They do in other fields….why not in education?

  6. Alexander Ooms says:

    NEPC would help its case tremendously if it would just list the issues where its research is in opposition to the public positions of NEA and other associated groups. Even if there is little distance, at least it will be transparent in its associations, and then we can concentrate on the research.

    My biggest problem with this and similar reports (and I’ll confess I’ve just read the summary here, though I will read the body) is phrasing like these:

    1. “The [research] was demonstrably inadequate to support the published rankings”

    2. “If the L.A. Times model were perfectly accurate…”

    Both are academic in their positioning – which does not mean that they are wrong, but to school operators, authorizers and others, their context is not particularly helpful. In the first case, once one compares the LAT’s method to the existing void of any attempt to differentiate and rank the impact of teaching on student achievement, the question might be if we are better off with or without the study’s contribution? (And I would differentiate between the study itself and the decision to publish.)

    The second is equally pedantic. Virtually all sides note that the LA Times was the first real attempt to quantify a complicated and nuanced issue — so when, in the history of the world, has an initial attempt to model complex multivariable data been “perfectly accurate” on the first try?

    It’s very different slant if one says the LA Times work should be regarded as a first step, or as a dead end. The dividing point here should be if Value-Added Analysis is a promising tool to help improve public education or not. IF so, then we need to come up with increasing degrees of accuracy (as is normally the progression) which includes building on the LAT study. If not — well I have yet to hear anyone serious take the position that it has no use at all – but perhaps someone will respond as such here.

    But too often to my mind, NEPC argues strongly against the specific tree (such as describing the LA Times research as: “demonstrably inadequate” “serious weaknesses” “invalid and unreliable” — and that’s just in the first paragraph of the summary) in full knowledge that it’s report will be used politically as an argument against the full forest (Value Added Analysis thus fully flawed as well).

    Derek notes above that he is supportive of using student data to evaluate teachers — a sentiment that is missing entirely from the summary, and I’d be willing to bet from the report as well. Why not include some statement of support for the concept of VAA even as one criticizes the methodology of one of the initial studies? Would that not be more academically accurate?

    For what it is worth, this is the most balanced piece (but very general) piece I’ve seen:

    • Jeffrey Miller says:

      “Why not include some statement of support for the concept of VAA even as one criticizes the methodology of one of the initial studies? Would that not be more academically accurate?”

      No, it would not.

      “But too often to my mind, NEPC argues … in full knowledge that it’s report will be used politically as an argument against the full forest (Value Added Analysis thus fully flawed as well).”

      Why not cast similar doubt upon the political motivations of the LAT?

      “The second is equally pedantic…“perfectly accurate” on the first try?”

      In my reading their point is not pedantic. But yours might be. Their usage of “perfectly accurate” does not seem to be an indictment of failing to perform the best analysis ever; rather, it seems to be an expression of social science methodological fitness.
      I don’t find this a relevant rhetorical argument against NEPC. It is in the nature of the scientific enterprise to accept uncertainty.

      “It’s very different slant if one says the LA Times work should be regarded as a first step, or as a dead end.” Yes, it certainly would and one could argue that if most readers accepted such an estimation, the LAT work would prove a poor model for making major policy changes.

      I wouldn’t be so down on education research, Alexander. Neither of the actors in this case published in a peer reviewed journal though that is to be hoped. That fact alone does not disqualify either work for serious consideration but it does place a special burden on each to defend the value and validity of their work.

  7. Leigh Campbell-Hale says:

    It’s refreshing to read this. All research is biased, but some is more biased than others. Do you read studies from the Brookings Institute or the Cato Institute? Before now, did you really think anybody was neutral on education issues? Did you really think good, objective research was being done? With this revelation, I now hope you re-examine your intellectual love affair with Michael Johnson. He is allied with Michelle Rhee and others with a distinct agenda. They aren’t neutral, either, and they’re basing their beliefs on research funded by people with a distinct agenda to privatize public schools. In the process, they are helping to destroy teaching as a career. As an example, just look at what’s happening with north Denver schools. Under 191, any teacher who is “displaced” with that re-organization with lose her job if the district doesn’t place that teacher within two years, for no fault of her own. Who finances Rhee? Who testified for 191? Follow the money. They’re the same people who are financing the pro-voucher, pro-charter, anti-tenure research to which you refer. If you google Michelle Rhee, besides her self-promoting Students First organization (formed as a national political lobby to oppose teachers’ unions), the two most recent posts are the excerpt below from an interview where she singles out Michael Johnson as the kind of candidate she wants to fund, and the other is a judge’s decision that of the 1,000 or so teachers she fired in Washington, DC, at least 75 were fired for no grounds at all. None. And those are just the first of court cases that will follow.

    What kinds of political candidates will you back?
    Colorado is a place where a lot of human-capital issues are being addressed, and Democratic state senator Mike Johnston is an example of someone who is driving that. Another good example is Democrat Gloria Romero, who ran to be the superintendent of public instruction in California and lost. She took some bold and courageous stands, so the teachers’ unions got behind the other candidates. She got no cover.(from Jan. 12, 2011 interview with Michelle Rhee in fast company)

  8. gerald keefe says:

    Great discussion here but Alan does raise a good point about the reliability of educational research in general.

    Studies regarding the educational landscape are no doubt subject at times to the types of biases that Alan pointed out. However, it’s the high stakes nature of valued added evaluations that makes the type of discussion that has taken place here absolutely critical.

    Thanks to Derek, David and Holly for their comments.

    Oh if there ever is an educational study where I’m a contributor to the information pool take the results with a grain of salt.

  9. stephanie harvey says:

    I’m wondering if the blogger is an equal opportunity research skeptic. Does he trust pharmaceutical research that is funded by the pharmaceutical industry or energy research that is funded by the oil industry? Or does he reserve his distrust for the world of education?

    • Van Schoales says:

      Stephanie, It’s a great and fair question. Education seems particularly challenging because of the limited number of tools for quality research, the lack of useful high quality observations/data, little investment in research, a relatively low bar for granting doctorial degrees and the appearance of the lack of a strong self-critique within the education research sector. Today’s education sector appears to look similar to what medical research looked like a century or two ago. Only in education do so many with EdD’s or PhD’s refer to themselves and others with the same credentials as “doctors” regardless of whether they are engaged in rigorous research. It seems unique to education.

  10. Alan,

    I’ve long had the same frustration (though I haven’t read the NEPC study and so am in no way commenting on it specifically). As Drs. Welner and Briggs can attest, as a grad student I tried to create a methodology that would quantitatively guage the political bent behind a given study, and somehow factor that in to the interpretation of the results. I’m not sure this quite worked out, but I have wondered if such an exercise is possible.

    The discussion also brings to mind the study of scientific research in general, and whether true objectivity is ever possible. I belive we can get pretty close, but I’m not sure we’ll know when we’re there.

    Slightly off-topic (focusing on medical research, not educataion), but this recent article in the New Yorker on how research findings change over time is illuminating:
    I suspect much the same thing goes on in ed research: researchers as a group are well-meaning, competent, and objective, but there is something “else” that creeps into our work that we are often loathe to acknowledge.

  11. Stuart Buck says:

    But the problem with Ravitch is that she just flipped from being someone who predictably supported various conservati­ve positions (school choice, etc.) to someone who is 100% predictable in opposing those positions.

    By the way, here’s a study by Jay Greene that found no benefit from voucher competition: So he’s already satisfied your criterion of producing a study that cuts against the grain.

    PS: There’s a typo in Paul Peterson’s name.

  12. Stuart Buck says:

    As for changes of heart: Jay Greene wrote a recent blog post that explains why he has been changing his mind on issues like value-added modeling (which he used to support): He also wrote a blog post on why he changed his mind on merit pay (caveat: he mentions there an article co-authored with me):

    I’d contrast this to Ravitch on school choice: the evidence in support of school choice has grown substantially since the 1990s, when Ravitch was an avid supporter — we now have multiple studies by respected researchers (David Figlio, etc.) showing that vouchers in Florida force public schools to improve, that vouchers in DC increased graduation rates, that charters in Florida and Chicago improved graduation rates as well, etc. It’s implausible to claim an evidence-driven conversion when you’re going squarely against most of the new evidence.

  13. Stuart Buck says:

    Actually, Jay Greene is a more legitimate example of someone who changes his mind for sincere factual reasons. Here’s a blog post where he explains why he changed his mind about value-added modeling: And here’s a blog post (caveat: he cites a paper co-authored with me) where he explains why he changed his mind on merit pay:

  14. Alexander Ooms says:

    So after reading the study, I’m curious (if they are still reading) — did the authors also write the “two-page summary” included in the link? For the difference between the “Two Page Summary’ and the “Executive Summary” that opens the study is striking (please read for yourselves). And it’s not length – the “executive summary” is all of one additional page.

    The executive summary is academic — even a little technical — and measured and analytical in its tone. Here are the most direct critical sentences (which are buried in the second page):

    “Our other findings, however, raise serious questions about Buddin’s analysis and conclusions. In particular, we found evidence that conflicted with Buddin’s finding that traditional teacher qualifications have no association with student outcomes.”

    The “Two Page Summary” which is what most media would open first, is very different. Here are its most direct critical sentences (which open the document) with some minor editing for space :

    “The research [...] was demonstrably inadequate to support the published rankings. Using the same [data and methods, the authors] found the earlier research to have serious weaknesses making the effectiveness ratings invalid and unreliable.

    Those are pretty different sentiments. And to my ears they don’t read as if they have the same author, (or perhaps the same purpose). It’s not uncommon for someone to “sex up” material for certain consumption. But that does seem to open some questions of motive.

    I’d be interested to hear from the authors if I am wrong, and any thoughts on the differences between the two summaries.

  15. Derek Briggs says:

    For the record–I wrote the executive summary from scratch. Kevin Welner suggested a few edits to it for clarity that I accepted. Kevin composed the 2 page summary based on the exec summary and sent it to me to ok. I made a few edits and sent it back. Kevin writes the 2 page summaries so that they can be accessible to the broadest possible audience. I don’t disagree with the sentences in question:

    (“The research [...] was demonstrably inadequate to support the published rankings. Using the same [data and methods, the authors] found the earlier research to have serious weaknesses making the effectiveness ratings invalid and unreliable.” )

    I do think we show that Buddin’s white paper was “demonstrably inadequate.” I’m not crazy about the use of “invalid and unreliable” in the second sentence because it implies that validity and reliability are yes or no issues. But did we raise concerns in our report that suggest there are problems with the validity and reliability of teacher ratings? Yes. So I was willing to let this go. Note by the way, that even with such strongly worded sentiments in the 2 pager, the LA Times still hasn’t felt compelled to correct its headline covering our report which read “Separate study confirms many Los Angeles Times findings on teacher effectiveness”

  16. Alexander Ooms says:

    It is perhaps worth pointing out that there does not seem to be much agreement that either side has a definitive claim to either science or truth (I think this is a pretty neutral summary: It also seems pretty obvious to me that the summary had – as well as a different author – a clearly different purpose.

    That should bring us full circle to Alan’s original point.

Leave a Reply

Colorado Health Foundation Walton Family Foundation Daniels fund Pitton Foundations Donnell-Kay Foundation