This post is kind of a work in progress, and more of a test to see what people think. Let me know if you have questions, please write comments.
I’ve got beef with the CAASPP website.
They give the same score reports as they did last year, but now they also include this neat little feature where you can see how “classes” did across grades. For example – here is what they provide for KIPP Raices, one of the highest performing schools in LAUSD’s area.
So if you add it up, they went from 83% meeting or exceeding standards to 79% meeting or exceeding standards. In theory, they dipped by 4%.
My problem is this: a 4% drop isn’t the same across all scores. It is the NOT the same if you go from 15% to 11%. You can’t compare them, they are just complete different situations.
It is completely unfair to compare schools using this kind of measurement of growth.
The truly fair way to compare scores is by using similar starting scores. Only compare 4th grade classes that started (in 3rd grade) at 83% with other 4th grade classes that started at 83%. Then we can set up percentiles to measure how the did only compared to similar schools.
That way, you compare like with like.
I will admit this right now: there are a couple of problems with the analysis I am proposing that are addressed in a footnote (see below). But this is what we got when it comes to public data, so let’s get going.
First, before you look at the scores, remember, that in this analysis a score of 50 is Average – it means the school performed better than 50% and worse that 50% of schools. So a 50 is not a bad score. A score of 90 means they did better than 90% of similar schools. And a score of 10 means they scores better than only 10% of similar schools.
When you click the link below, you will see comparison scores for all 4th grade classes in LA County. There’s a search function on top. I am just going to leave it out there, so you can look it up. Enjoy.
4th Grade Math Comparison Scores
Notes: First of all, it works much better with micro-data, which I don’t have access to (because I am not super official at this or anything). When you use the public data, it isn’t great, because there simply aren’t enough data points. There are only about 5,000 schools that have 3rd and 4th grade scores, and it would work a lot better if there were more like 100,000 scores. Second, this also has the problem that some schools simply don’t have enough similar schools to compare them to. When that is the case, I have listed it as “comparison score not available”.
Hello!
I like your effort to do something like a student growth percentile (SGP) at the grade level. Have you thought about using a regression to create a residual growth model? A regression could produce a predicted score for each school. The residual (predicted score minus actual score) would tell you much more or less the school grew than expected, given all the changes that occurred statewide. No measure is perfect, but it would be interesting to see how similar residual growth is to your SGP-like measure.
One big benefit of residual growth is that you could use mean scale score instead of % Met. That provides much more nuanced information, and it avoids the problems of only looking at whether students are above or below one threshold.
Best, Dominic
On Tue, Sep 13, 2016 at 6:13 AM, School Data Nerd wrote:
> schooldatanerd posted: “This post is kind of a work in progress, and more > of a test to see what people think. Let me know if you have questions, > please write comments. I’ve got beef with the CAASPP website. They give > the same score reports as they did last year, but now they ” >