Last week’s CSAP results were coupled with the Colorado Growth Model in a way that began to peel the onion back on school and district performance. However, one major piece was, to my mind, still missing. The growth model does not differentiate between the performances of schools or districts with vastly different percentages of Free and Reduced Lunch (FRL) students. For me, one of (if not the) primary goals of education reform is to lower and eliminate the achievement gap that persists between low-income students and their affluent peers. Unless you measure these students compared to their peers, it’s hard to see who you are helping.

As an example, EdNews helpfully provided a list of some of the larger Colorado school districts with both their status and growth scores. But missing was any sense of which districts serve the most (or least) FRL kids.

Well, here is the regression graph on FRL & status scores by school district (based on the EdNews data):

This is pretty much the same graph that has haunted public education in the US for the last 50 years.The paradigm is this: As the percentage of FRL students goes up, the percent of proficient students goes down (I took the EdNews total proficiency and divided by the three subjects). Note that this is a very consistent line – a few outliers deviate from the trend, but the pattern is stark and clear. Income, one might say, is destiny (or said another way – districts cannot improve achievement without solving poverty).

But look what happens to the data when you do a regression with FRL and Growth:

All of a sudden, the data scatters: the pattern between income and growth is deeply inconsistent, and shows some real differences by school district. In particular, Aurora, Denver and Mapleton are doing significantly better than the averages (well above the line) while a number of districts (who are below the line) do not look nearly as good.

And look at the differences even on the same horizontal or vertical line. There are roughly five school districts with FRL populations between 60-70%, but their growth scores vary from the high 30s to the mid 50s. There is something very different going on in Aurora and Denver than in Pueblo and Westminster.

Likewise look at schools with similar growth scores — particularly as the growth scores are clustered around a median of 50%, there were 11 school districts with growth scores between 50 and 55. But look at the difference in FRL numbers even at the similar growth scores: FRL percentages vary from below 10% to over 65%. How is Denver — with a FRL population of 67% able to get a growth percentile just one point less than districts with FRL percentages of 8 and 9%? Perhaps income is not destiny.

The correlation in the growth graph is much looser (I did not calculate the R2). Low-income districts (the unweighted average is 37% FRL) are still a minority of the high growth percentile districts, but there are some considerable anomalies.

A few caveats: this quick analysis does not count as statistical research (it was done, in its entirely, in a middle seat in a crowded plane and with a small data set). Better statisticians than I can crunch the numbers. And particularly with the growth numbers there is the problem of measuring the tallest pygmy — Denver and Aurora (and others) are starting from a very low base, so even this growth (as the Post tellingly reported) essentially dooms generations of kids to a life of functional illiteracy. Plus, there are bound to be even bigger discrepancies at the individual school level within districts (which would be more interesting).

But even if directionally correct, the growth data tells a story that the status data never revealed. There are big differences between the growth numbers of Colorado school districts that cannot be explained by differences in income alone.

Hopefully the second graph is a paradigm slowly starting to shift.

Popularity: 5% [?]

Alex…

Great work! I tried to do the same thing and was defeated by Xcel 2007, which is the most non-intuitive piece of software I have yet to encounter.

Given how savvy the data folks are at CDE, I’m wondering why they aren’t showing the data in the fashion you have suggested…maybe on the next go round.

They also need to:

1) Enable SchoolView so that people can copy or make page shots of the scattergrams;

2) Visually correlate ACT scores with free/reduced lunch and minority status, rather than forcing folks to work off the Xcel charts.

3) Make it easier to do statewide comparisons…if you want to know who the outliers in the entire state are (both good and bad) why not allow the filter to select all schools?

More significantly, we need a clarification and analysis of whether growth improves with student status. My original understanding of the Colorado Growth Model was that it compared cohorts…low performing kids with low performing kids, etc. Therefore, each cohort had its own median point.

But some people seem to think that the growth numbers are more general. Some also claim that high status students grow faster than the norm…perhaps because it is easier to teach them, perhaps because they are more capable of teaching themselves, etc.

Or am I asking two different questions…1) do the growth numbers compare similar cohorts, 2) do the status numbers suggest that high performing kids accelerate their own learning?

Anyone know the answers???

Great work Alex and wonderful questions David! I’m glad to know that I am not the only one that struggles with Xcel every August after the CSAP data dump. The growth data is a huge help but I worry that we don’t spend enough time looking for school quality through a complex lens than includes more than one, two or three measures of performance. It’s also helpful for folks to know value and limitations of these measures. Blood pressure is very helpful but not enough to get an accurate assessment of my health.

Thanks for focusing on FRL rather than jumping directly to ethnicity. I spent some time yesterday morning reviewing scores for my school. I got lots of tables with sliced and diced data (some growth but mostly status) but the only graph was disaggregated first by ethnicity. This is on my mind because usually people draw (jump to) conclusions more quickly with graphs than with tables.

This one shows %Proficient and Advanced against Test Year and has three lines – Caucasian/Asian, School Average, and Combined minority with the Caucasian and minority lines consistently separated by about 40 percentage points and the school average right down the middle.

This gives the impression of a bi-modal distribution in which white and Asian kids do well and other minorities do not. It makes it a little too easy for sub-conscious assumptions about individual kids to go unchallenged. I had to remind my self that there are black and brown kids who do well and white kids who do not.

I think it would be far more enlightening to disaggregate by performance level and then examine the demographic characteristics common to each. We would verify, as you have described, that it’s not the color of a child’s skin but the economic conditions under which s/he has grown up that correlates with testing outcomes.

This is not a big deal but I think it matters and I’ve seem some egregious errors of interpretation made at other schools. We’re swimming in data and there are not enough people really well trained to think about it; I sometimes wonder exactly how valuable that is.

It sort of reminds me of the derivatives market 18 months ago. The idea that this is great is out there but the underlying data set has been so meta-processed I’m not sure how many people could explain exactly what it all means. Maybe I’m just looking for a reason not to be psyched about test data.

I wanted to respond to a couple of the posts related to the Colorado Growth Model and SchoolView. Alex Oooms did a nice analysis of the relationship between growth rates and students eligible for FRL. He also wrote: “The growth model does not differentiate between the performances of schools or districts with vastly different percentages of Free and Reduced Lunch (FRL) students.”

I’m not sure but this comment may reflect a misunderstanding of the Colorado growth Model. The growth model measures each child’s growth rate based on all other kids with the same starting point (statistically similar academic history). Thus, it creates a set of growth percentiles by subject for each student that can be interpreted the same way and is not correlated with achievement level (status). This means a low achieving or high achieving student can have low or high growth. David Greenberg asked a question about this point in a later post.

We then disaggregate these growth rates based on student subgroups. The results at the state level can be found at http://www.schoolview.org/stategrowthsummaries.asp. These results show, for example, that the 2009 statewide median growth percentile for students that were eligible for free or reduce price lunch was 47 in math and 52 for students not eligible. Thus, we have a statewide gap in growth rates based on income. We also report the disaggregated growth results at the school and district level through the School and District Growth Summaries. These can be found on SchoolView off of the School Performance button (select District and School Performance Reports). These tables show three years of growth data (both normative and criterion-referenced) for each school compared to the district and the state.

When reviewing this data, we see a wide variety of growth percentiles that deviate from state averages at the school and district level and a variety of trends over time. It is these differences that Alex picks up on in his second scatter plot and the differences in school and district performance that are perhaps most interesting and leads to key conversations. Why are the growth rates for low-income students in some districts and schools higher than other students? These districts and schools are making progress in closing the achievement gap. Why are their large gaps in others? These findings are not an artifact of the measure but rather reflect something happening in these educational settings.

Lastly, David Greenberg makes a number of recommendations about improving SchoolView. Please keep these coming. We recognize that we have only made a start here and need to improve the visualization of our data rather than compelling folks to mine spreadsheets. But while we are working on improvements, we remain committed to getting the data into the public domain. You can expect us to do this with interactive displays like the Colorado growth model, simple charts like the state summary data, summary tables, and spreadsheets suitable for secondary analysis. One goal of developing the visual platform for the Colorado Growth Model is to encourage innovation in data visualization. Our next efforts will include presenting multi-year views and incorporating post-secondary readiness information.

I have great admiration for both Rich and the Colorado Growth Model, which I have often publicly said has at least or more potential to aid reform then either ProComp or the Innovation Schools Act. That said, even after reading Rich’s comments and searching through SchoolView as instructed, I don’t believe I “misunderstood” the model at all. The growth model, as Rich points out, differentiates the performances of

students(or student cohorts) based on data including FRL. It either does not comparing differences with FRL either byschoolor bydistrict, or the data has eluded me (in which case a link to a specific page would help). I’m pretty sure it is the former.The CGM has grouped students by cohort based (as Rich says) on “statistically similar academic history” (SSAH). That may be a fair way to measure growth among different populations, but it is not without its disadvantages. Grouping students into academic cohorts has the “tallest midget” problem. If a student’s SSAH is in the lowest-performing cohort, to judge his growth while in the K-12 system has its merits, but this handicapping also can create a distorted view of success.

Imagine everyone has to run a mile, and we cohort the groups by speed history: 6-8 minutes, 8-10 minutes, 10-12 minutes. It may give a better idea of a student’s growth if, after running a mile in 12 minutes for several years they improve their time to 10 minutes, but at some point the handicapping ends. Be it taking the SAT, the first year in college, or applying for a job, that individual is no longer judged solely within his cohort but across all students, and having improved to 10 minutes still means that they will finish behind all the students in the first two groups (even the ones who have not improved at all). Growth without a minimum level of status is a nice metric, but it is not a good indication of how well a school (or district) has done preparing their students for a future beyond 12th grade. All measurements have their limits.