Next Wednesday, at 10 A.M., the results of the most recent state tests will become available on the CDE’s website. Ahead of that data dump, I thought it would be important to give some reminders about how a data nerd might interpret this information:
Proficiency Levels Are Not The Best Measure of Effectiveness
The ideal school adds value to students’ education. That is, students improve (or maintain high) performance year-over-year. So, ideal data would tell us how much a school improved student outcomes.
That is not the data we will receive from the Department of Education.
Instead, the state will publish proficiency levels for each grade level. You may recall – this is the heart of the problem. Proficiency is often simply a reflection of the cards the school was dealt. If a school was given a set of students that arrived with a high proficiency level, and they then get high test scores, that is not particularly impressive.
Instead, we have to twist the data and contort it to get meaning out of it.
Like last year, I will put out a list of most improved schools. This year, I am also going to analyze individual grade level changes by school. And then in October, the California Charter Schools Association will run a rigorous regression with poverty and ethnicity data to rank schools on their Similar Students Measure.
Ideally, the state would provide more individual student analysis, but it doesn’t look like that is coming any time soon.
All of this is to say: the scores you see on Wednesday are important, but they tell only a sliver of the picture. More likely, they are simply a mural of our unjust and segregated educational system.
Many Subjects are Not Tested
Students are tested in English language arts and mathematics. Next year, the new science test will also have publicly released scores.
But notice all that is missing: We have no data on History! Nor we do we have data on those dual immersion programs that seem to be so popular. We have no data on technology skills or coding, even though STEM schools abound.
Many people, including my union, think that we test students too much. I think we under-test students. Parents have the right to know whether a dual-language immersion program will actually result in bilingual fluency. Parents have the right to know if their STEM school is actually teaching coding and engineering skills, or is just plopping the students in front of code.org for an hour and calling it a day. Schools need to be tested on the claims they make.
We don’t have to do all these tests all at once, we can spread them out – and we would get a much more robust picture of school performance. Which brings me to my last point…
Every School Takes the Test Differently
While there are a set of rules that govern how the state test is to be taken, there are a lot of decisions about the test taking time frame that give schools freedom to adjust test taking calendars and procedures.
Students have the ability to start and stop the test and take it over many days. Some KIPP schools, for example, break the English test into 4 days, the performance task into 3 days. On the opposite end of the spectrum, many schools simply give their students one extra long sitting to complete the entire English test.
There is no available data on whether this is beneficial for students, but in your gut, you have to believe that it can’t hurt. Instead of subjecting students to 3 to 4 hours of grueling testing in one sitting, breaking it up seems much more pleasant.
Why wouldn’t a school choose to break up the time? Sometimes this has to do with technology constraints, bandwidth constraints or even scheduling conflicts. But other times, it has to do with a lack of understanding of how the test works. I honestly think that many people do not know that the test can be taken over multiple days. And I think that if they knew that, more schools would use it to their students’ advantage
As a result, sometimes a test is more than just a measure of students’ proficiency or growth. It may also be measuring a school’s ability to manage its resources, think creatively and teach students self-awareness and perseverance.
I know that this is a bit of a stretch, but it comes from my personal experience with schools. Testing is just one of the many complex tasks that a school has to complete – and if the school gives the students an advantage through their test taking organization, I think that its a fair reflection of the school that should rightly be factored into the score.