With the DPS Board election now over, it’s back to the grind. Much of the current discussion around DPS — including the bulk of a recent A+ meeting — is focused on the School Performance Framework (SPF). Now I like the SPF, I think it is rigorous and highly useful for comparing individual schools, and I applaud the district’s honest and transparent view. However, as a snapshot of the District overall, the data is not presented particularly well — which makes sense, since that was not its original purpose.
But the SPF is increasingly used to evaluate the District as a whole. The recent A+ meeting got me thinking about what the SPF information would look like if viewed with a wider lens and in a simpler presentation. Turns out it is quite a magnified perspective.
The SPF ranks individual schools on a variety of criteria, and places each school into one of four categories. In ascending order of quality, these four categories are: On Probation, On Watch, Meets Expectations, and Distinguished. Looking at summary data for these categories, and it becomes clear that there are three significant areas where the SPF somewhat distorts the broader view due to its focus on individual schools. The data is illuminating on all three:
1. Size matters. The SPF ratings do not factor the size of student enrollment in a school. In a broader view, a good school of 500 students should be roughly equivalent to two bad schools with 250 students each. However far more of the worst DPS schools have large student bodies. In the lowest category of “On Probation” there are 10 schools (of the 20 possible) with more than 500 students. In contrast, in the highest category of “Distinguished,” just one school (of the 9 possible) have more than 500 students. So 50% of DPS’s worst schools have over 500 students, while less than 10% of its best schools have over 500 students.
2. Income matters. The SPF ratings do not factor the percentage of free and reduced lunch (FRL) students in a when ranking individual* schools. Presumably, if we are trying to close the achievement gap, these are the students we want at our best schools (or at the very minimum we want them there in equal proportions). Yet FRL students attend Denver’s worst schools in far greater proportions than their more affluent peers. About 2 of every 3 DPS students are FRL, yet in a stunning reversal only 1 of every 3 students in a Distinguished school is FRL. Our best schools literally take the district’s base demographics and flip them upside down.
3. Grade matters. The SPF does not take into account the differences between schools serving different grades. Ideally, as students move through the education system they would get better options – having proficient 12th graders is more important than having proficient 1st graders. But the data shows that the better schools overwhelmingly serve the elementary grades. In fact, the closer a student is towards graduation, the more likely they will attend a bad school. With some basic assumptions, the data shows that a student is over 3 times more likely to attend a Distinguished school in grades K-5 as they are in grades 9-12.
Lastly, all three of these factors are badly exacerbated when combined. Consider the near-deadly combination of large schools, a high percentage of low-income students, and grades 9-12. Just For example, within DPS there are** two large schools, both with On Probation status, that house over 1,850 students in grades 9-12. Of these, about 1,42o are FRL students – remember, this is in just two schools among the 20 ranked lowest. In contrast, how many total FRL students in grades 9-12 attend a Distinquished school? Roughly 180 – all of whom are at DSST (a charter school). How many FRL students are there in grades 9-12 at traditional (DPS operated) Distinguished schools? Zero. Well, how many students overall in grades 9-12 attend a traditional Distinguished school? Um, Zero. That’s not a misprint: there is just one Distinguished school serving grades 9-12, and it is not operated by DPS. If you are an FRL student, in a large school, in grades 9-12, you are likely in the vortex of public education’s perfect storm.
If you’ve gotten past those three screaming ghosts of DPS present, here is the SPF data with a view across the entire District. Again, the SPF ranks each school in ascending order of quality across four categories On Probation, On Watch, Meets Expectations, and Distinguished. Below is a graph which shows how many total students — both FRL and Non FRL — are in each school category:
Remember that the SPF is based around a median score, so that most schools will clump in the middle. Two points in this graph are striking. First is how many kids go to the district’s worst schools (10,477) versus attend the District’s best schools (3,577) — a ratio of almost 3 to 1. Again, some of the District’s worst schools are among its largest, and if we are serious about improving outcomes, we need to address these first. Being a small school does not guarantee quality (see Manual’s failed small schools experiment), but particularly in the older grades, it’s harder and harder to find large schools with significant FRL populations that are of high quality.
The second point is far worse: look at what happens to our poorest students – those that qualify for FRL status. Overall, FRL students are about 67% of the overall district. But they make up a far greater percentage of students in the worst schools. Here’s another look at the same data showing the percentage of students by income:
Denver’s worst public schools (those with On Probation status) have 83% FRL populations; its best public schools (with Distinguished status) have just 25% FRL populations. The better the school category, the smaller the percentage of FRL students who attend. Ouch.
There are about 50,000 FRL students in Denver: less then 2% go to a Distinguished school, while 17% of attend a school On Probation. There are about 23,400 Non-FRL students in Denver; 11% go to a Distinguished school, and 7% to a school On Probation. As we’ll see in a minute, this division is as true in kindergarten as it is in high school. If that does not make you think the deck is stacked against low-income kids, I don’t know what will.
What is also astounding, and what is not generally apparent from a school-based version of the SPF, is how the proportions change as kids move through the public school system. Now this is harder to ascertain from the SPF data: While schools are sorted by grades served, they do not list the distribution of kids across those grades. If we make the generous assumption that the students are evenly distributed (so that a K-8 school has the same number and FRL percentage of kids in each grade), this is what the data looks like:
There is a sliver of blue in grades 9-12 for the students who attend Distinguished schools, but you have to squint. Again, this is somewhat distorted since the distribution was smoothed, resulting in cliffs after both 5th and 8th grade, but the real numbers would likely be even less flattering (with an estimated 50% dropout rate between just grades 9-12, imagine how much the actual chart would slope down at the end). The number of kids in the bottom two school categories is somewhat consistent; it is the schools of quality that diminish.
Looking at the same data by percentages:
Under these assumptions, in K-5, a student has about a 1-in-2 chance of going to a school in the top two categories (Meets Expectations and Distinguished). By 9-12 grade, that chance is down to about 1-in-3. In K-5, about 6% of students attend a Distinguished school. By 9-12 grades, that drops to under 2%. If you did the chart with the actual numbers, I’m guessing the difference between the elementary grades and students closest to graduation would be even more stark.
And those depressing statistics are for all students regardless of income. When you start to look at the numbers by FRL, graphs no longer work because one simply can’t see the percentages as they are so small. On any reasonable scale, the FRL students who attend Distinguished schools are literally invisible. Since we can’t graph the data, here is the table. Remember again that there are far more FRL students than Non-FRL in the District:
The FRL percentage actually declines as students move through the system (which makes some unfortunate sense, as FRL students drop out of school at a much higher rate) So, again under our assumption of even grade distribution within each school, in K-5 the percentage of FRL is actually about 70%, yet within the separate categories FRL students compose 86% of On Probation schools and just 13% of Distinguished schools. By grades 9-12, the overall FRL percentage drops to about 57%. But the percentage of FRL students enrolled in On Probation schools stays high, at 74%. Low income students are far more likely to attend our worst schools – at the same time that far more of them also drop out altogether.
The top of the table bears some reflection as well. While there are a number of Distinguished schools that serve K-5 students, they include far fewer FRL students. The outlier jump in the percentage of FRL students in 6-8 attending a Distinguished school is unfortunately easily explained: West Denver Prep is a Distinguished middle school serving 93% RFL. Since the base is so low, just one school of quality can make a significant impact.
In fact, there are only three Distinguished schools with students in grades 6-12. These three schools serve approximately 1,040 students in grades 6-12. The vast majority of these – fully 85% — attend either grades 6-8 at West Denver Prep (WDP) or 6-12 at the Denver School of Science and Technology (DSST), both charter schools (the remainder are in the final years at Slavens, which is K-8 and 5% FRL). What is more alarming is that of the roughly 400 of these 6-12 students that are FRL, roughly 98% attend WDP or DSST. What this means is that there are less than 10 FRL kids in grades 6-12 attending a traditional Distinguished school. That is shocking enough where it deserves an echo: less than 10 FRL students total attend the highest quality category of Denver’s traditional schools during the second half of their public education.
This is, frankly put, the worst of all worlds. In our democratic society, public education is intended to prepare all students for college or a career, regardless of the circumstances into which they are born. To me, a functioning public education system is one of the most important tenants of a civil society. Yet this data implies that we systematically deny the equal opportunity of our best schools to the students who need it most. I support many of the current reform efforts in DPS, and I continue to believe the current leadership is the best Denver has had in generations. DPS should be commended for publishing the SPF and being honest about the difficulties they face. Their leaders are very smart people with the best intentions. But they are not in the schools, and it is in the schools where these challenges must be met.
Although I rarely get accused of underestimating the deficiencies of public education, I confess the sheer despair inherent in the summary SPF data surprised me. I tried to come up with a good ending to this post for a few days, but I can’t. In the end, the lack of a suitable finale may be appropriate. No matter how one looks for it, for the vast majority of low-income kids at DPS, there is just no good ending.
—
Notes: Two comments deserve some clarification:
* The SPF accounts for FRL status in the student growth metric in that growth is measured within a demographically-similar cohort. However the SPF does not weight the performance of an FRL student any differently than a Non-FRL student (i.e. if growth for each cohort was identical, a school serving a high percentage of FRL would be ranked exactly the same on growth as a school with a low percentage of FRL). The final blunt ranking of schools also does not differentiate between schools serving different cohort groups (it would be an interesting exercise to see the data and rankings within each school cohort, which is here).
** The original wording could be read to imply that there were just two large schools serving grades 9-12 in DPS, which is not accurate; my intent was an examination of these two specific schools, not to say that there were “just” two.
Popularity: 44% [?]
Though a bit stunned, ( yet not that surprised), I applaud the use of the date and the presentation. I was very much required to move my perspective from down right depressed, to a position more aligned with a military operation confronted with overwhelming odds and committed to survival to muck my way to the end of your article.
As a non RFL family with two DPS students, (middle and high), I am grateful for the information. What confronts those of us in the non RFL group is how do we participate? (Or we simply cross fingers and bury heads hoping our children get through before the fall.) And how do we engage the RFL families, students and the accountable staffs of all DPS schools. Math, logic, and societal, economic, migration patterns, ( maybe politics as well), will tell you that the non RFL families will only increase in the coming years.
Recently I took a group of Central European educators from former eastern bloc countries, who were here as part of a group of ROMA (Gypsy), teens and adults participating in a Leadership program. We visited a charter, ( Odyssey in Stapleton, middle school ), a DPS magnet, (Denver School of the Arts 6- 12), and Aurora Central High School, ( a large public high school). All three are near-by one another. In two of the schools, they engaged with teachers, staff and students. They drooled over the possibilities they saw in two of the schools, and literally dreamed out loud their visions as committed educators for one of the most marginalized and impoverished groups of people on the planet. Mind you well, they were not young, new or inexperienced educators.
At the third they felt almost at home but in a bit of shock by the role of the teachers. They did not engage with anyone beyond their “escort”
At all three, one of the first questions would be; “how many students?” The second would be; “Are they poor?”
Michael,
How one participates to change this structure is a very good question. For me personally, it was getting involved with a high FRL charter school. There are a number of other worthy organizations to which I could help point you, and I hope others will chip in with their suggestions.
Alex,
Thanks for presenting this information. Very interesting, but I’d like to present a different perspective on the data.
1. Yes, grade matters. Why? Because it’s correlated with size and with another factor you do not specifically mention: homogeneity or heterogeneity of students in a school. So, yes, a larger proportion of DPS high schools have poor scores than do DPS elementary schools. However, if you think carefully and critically about it, such data do not establish that high schools are performing more poorly in educating their students. High schools are larger than elementary schools and, as you note and as some research shows, larger schools, when all else is held constant, tend to do have lower scores or, as I prefer to say, result in less learning. High schools are also more heterogeneous than elementary schools because they draw from a larger catchment area. As such, they are more likely to have students from a wider variety of neighborhoods. And, yes, income does matter. So, elementary schools that draw from a smaller catchment area and have a relatively homogeneous student population with few low-income students are more likely to have a distinguished rating. This result is only partially, if at all, the result of teaching at that elementary school. It’s largely due to the students the school receives. So, grade matters but only because of the factors I have discussed, not because DPS high schools are worse than its elementary schools. Any city that combines schools as students move up in grades, as most cities do, will show the same effect.
2. Yes, income matters and it always has. Not all kids from wealthy homes learn easily and not all kids from poor homes struggle with learning, but there has always been a correlation. You would need to read more research than I have time for to explain why that is so, but I’ll just say that I support reducing or closing the achievement gap (which is primarily one of income), but much of the press and the writing on this issue implies that it is new. I’m just glad that we’re finally giving attention to the achievement gap, but it is not a new trend. We have always had an achievement gap. If someone would present longitudinal data occasionally, we might even find that we are doing better in some areas, i.e., graduation rates, than we were in the past when many poor students dropped out at earlier grades. But, the fact that school scores differ by the income of the students is not necessarily because teachers in those schools are teaching better. In fact, many teachers in students serving low-income students may be teaching better than those in high-income schools. We just don’t have the data, or present it, in a way that permits us to examine the link between quality of teaching or quality of a school and student scores.
3. A couple of comments on statistics: (a) You write, “Remember that the SPF is based around a median score so that most schools will clump in the middle.” You’re confusing central tendency (median) with variability here. Clumping in the middle means there’s little variability in the data. A median score is the middle score and has no implications for how spread out the scores are. (b) “Data” is the plural form of “datum”. We rarely see the singular word, but it’s still generally accepted as a plural word.
I’m not meaning to sound harsh. I’m thrilled that you’re giving us some data and prompting us to think about what the meaning. Numbers are useful, but there are many ways to interpret them. I just wanted to add my interpretations!
Thanks,
Jody,
I don’t think you sound harsh at all, but I’m also not sure what you mean on several of your points. It sounds a bit like you find the results for DPS entirely reasonable given their demographics and a history of achievement gaps between income groups. I most emphatically do not.
Here’s where we might disagree:
1. “elementary schools that draw from a smaller catchment area and have a relatively homogeneous student population with few low-income students are more likely to have a distinguished rating.”
Sure, but the primary factor here is that they have “few low-income students,” NOT that the geography base results in a homogenous population. If a school had a homogenous population of high FRL, it would be even less likely to have a distinguished rating. Homogeneity is not the critical factor.
2. “This result [a distinguished rating] is only partially, if at all, the result of teaching at that elementary school. It’s largely due to the students the school receives.”
Um, “If at all”? Forgive me, but I find this just another way of saying that poor students just can’t be taught. I don’t believe that at all, and there are lots of schools that show that to be false, and I would hope lots of elementary teachers with high FRL populations who would disagree.
3. “So, grade matters but only because of the factors I have discussed, not because DPS high schools are worse than its elementary schools.”
By pretty much any metric DPS high school are worse than their elementary schools. While I don’t necessarily agree with your claim that certain inputs matter (i.e. homogeneous populations), the conclusion does not change. Grade matters precisely because DPS high schools are worse than elementary schools. During the latter half of public K-12 education in DPS, learning diminishes.
4. I don’t think (or argue) that the achievement gap is new. Others can chime in if it is getting worse or not. But clearly the impact of a substandard education in an increasingly service-based and global workforce is far worse. I expect the career and income prospects for a high-school dropout are far less compelling now than a generation or two ago when there was more of a manufacturing sector. The achievement gap has a greater impact now than it did previously.
5. “But, the fact that school scores differ by the income of the students is not necessarily because teachers in those schools are teaching better. In fact, many teachers in [schools] serving low-income students may be teaching better than those in high-income schools.”
The SPF weights cohort academic growth more than any other factor, so yes, I think the difference in scores is likely related to teacher quality and that student demographics are taken into account. Moreover there are plenty of studies that show that unwanted and low-performance teachers get dumped in high-poverty schools.
6. “We just don’t have the data, or present it, in a way that permits us to examine the link between quality of teaching or quality of a school and student scores.”
No offense, but this is patently absurd. There is an entire industry of well-regarded academics doing exactly that. You might argue about the degree of accuracy, but not that we don’t have the ability, or that it is a worthy exercise.
7. I don’t think I am confusing central tendency at all, although I probably did not explain my point very well. The SPF gives each school a growth rating that is a relative, not absolute score (and median growth is set at 50%). This grouping produces a bell curve, whose natural shape is “clumping in the middle.” Were DPS to do a growth assessment on an absolute (and not relative) scale, I expect it would look different.
8. I appreciate the reminder that “data” has a singular form, but I find its use in a generalist, contemporary format both archaic and pedantic. Each his/her own.
I’m glad to have your interest and the conversation. I think these are important distinctions.
Jody, you said:
“So grade matters but only because of the factors I have discussed, not because DPS high schools are worse than its elementary schools.”
I’m not sure I’d agree with your conclusion.
Alas, while appreciating your spoonful of sugar that helped Alex’s medicine go down, I’m still left with the bitter aftertaste of the raw data he has presented.
Please don’t take it personally that I’m not altogether comforted by your own rational interpretation of the numbers! — but from my perspective, there’s only so much that can be done to make palatable what I think has been shown to be cod liver oil to begin with.
And really old oil, to boot. Blecch!
Not only did you fail to find a way to end your analysis on an Up note, Alex, but the beginning and middle weren’t so tasty either. If truth is its own reward, you’re to be commended for presenting the undisguised oil on the cold spoon of reality today. Blecch! er, I mean, “Thanks a lot!”
Alex–
I won’t dare tread upon your general conclusions. Things are crap, and not getting less crap-like very quickly.
But there are some inherent problems in aggregating the SPF’s over the district.
Did you look closely at the SPF rubric? The scoring in many categories (the AYP comes most to mind) is very arbitrary and distorts the results. In other words, the SPF creates ordinal variable out of continuous ones. Now this distortion may come out in the wash, when aggregated. I don’t know. It would take a multi-variate analysis on the full data set (which DPS does not make available, even at th school level — you could not take the materials given to a principal and reconstruct the SPF scoring), and then individually t-tested them (maybe, I’m not sure that approach would work) against the SPF scoring. It was clear from the very first video Jaime Acquino did (May 2007?) explaining the SPF that: A) there was a lot of subjective decisions made; and b) outcome could be gamed from the system.
I have tried to understand why the SPF paints a high-achieving (20points above DPS average), low growth school like Edison so poorly, so I went through and looked at every single contributing factor in the SPF rubric. And frankly, I still don’t understand how/why the district reaches its conclusions on AYP and keep-up growth, and why they differ with CDE’s conclusions. The other issue, which may not be small, is how to track delta’s in the rate of student growth, as they move between schools-elem to middle to high.
ALEX –Please take your Student Progress by SPF category, and re-weight it for constant school size!! No doubt your conclusion is on the mark, but I have a hunch you are overstating the effect, which is what Jody is writing aobut…
I’ve yet to hear a totally convincing answer as to why reform efforts exemplify marginal utility as students progress, but you’ve illustrated it. We get some results in primary grades, and then they erode inexorably over time. It is as if schools can close part of the kindergarten preparedness gap, then slip backwards over the years.
HINT: It ain’t all about the classroom experience…
Interesting analysis, Alex. A couple of comments..Another reason why FRL rates drop as kids get older is that kids are less likely to sign up for FRL once they get to high school for a variety of reasons–they’re embarrassed, they spend lunchtime off campus with friends, etc. So FRL may not be as accurate a measure of poverty in a high school as in an elementary school. None of this negates any of your analysis, of course. Second, DPS is not the only place where high school achievement does not appear to be as good as elementary achievement. This phenomenon can also be observed in Colorado’s mean CSAP scores (especially in math) as well as in National Assessment of Educational Progress (NAEP) scores. A bigger question is..why is this happening in our state and our nation as a whole?