Via Lisa Falkenberg on Facebook, SBOE member Thomas Ratliff uses the philosophy from Moneyball to analyze the accountability system for Texas public schools.
The poster boy for the book
The book says, “One absolutely cannot tell, by watching, the difference between a .300 hitter and a .275 hitter. The difference is one hit every two weeks.”
In Texas public schools, you absolutely cannot tell the difference between an exemplary school district, a recognized district or an acceptable district simply by watching. The difference can be the performance of a small subset of students on one test on one day in the 180-day school year. This is a byproduct of our accountability system.
The book says, “The problem is that baseball statistics are not pure accomplishments of men against other men, which is what we are in the habit of seeing them as. They are accomplishments of men in combination with their circumstances.”
The accountability system doesn’t care about circumstances. It generates a report that shows how students did on a test, period. This is measuring the accomplishments of students against other students. We must change our accountability system to measure student performance in combination with their circumstances. Not all children enter or exit public schools with the same circumstances. We absolutely cannot have the same expectations for all of them, nor should we measure them all in the same manner. There are different definitions of success that involve academics, athletics, career and technology, community service, the arts, and the list goes on and on.
The book says, “I am a mechanic with numbers, tinkering with the records of baseball games to see how the machinery of the baseball offense works. I do not start with the numbers any more than a mechanic starts with a monkey wrench. I start with the game, with the things that I can see and the things people say, and I ask: Is it true?”
Our accountability system is designed to measure career and college readiness. The question is, “Is it true?” Does it predict career and college readiness? I believe it does not. My proof? To my knowledge, there are very few, if any, colleges or universities in the United States that look at TAKS test scores as part of a student’s application. If the accountability system and the state’s standardized test measured college readiness, wouldn’t you think colleges would look at it? Similarly, I’m not aware of a single business in the state of Texas that asks for TAKS test scores as part of the job application process. Again, if the system predicted career readiness, wouldn’t Texas employers use this as a part of evaluating prospective employees? We need an accountability system that takes a broader look at a student’s K-12 education and provides a measurement that will be useful to colleges, universities and employers.
Just for the record, it was the movie Bull Durham that first made the observation about being a .300 hitter. Be that as it may, a couple of points. One, while everyone talks about the statistics when discussing “Moneyball”, the central insight that Billy Beane had wasn’t just that on-base percentage and slugging average correlated better to winning than batting average does, it’s that (at the time, at least) those skills were valued less in the marketplace than batting average was. As a low-budget team, the A’s needed to take advantage of market inefficiencies like that to overcome their financial disadvantage. That’s beyond the scope of Ratliff’s analogy, but as this was the most misunderstood part of the book, it needs to be said.
Now that I’ve gotten that out of the way, I think Ratliff is on to something here. Is it true that TAKS scores correlate to success in college? More to the point, do TAKS scores correlate better than other available measures to success in college? I don’t know, and it’s not clear to me that anyone else does, or at least that anyone in a position of authority does. This is an easy enough question to answer, if we’re tracking how students ultimately fare in college. Let’s crunch the numbers and see what we get. Maybe TAKS scores are a good metric. Maybe there’s something else, like writing ability or extracurricular participation, that correlates better. Maybe we’ll find that external factors like a family’s income level and prior educational attainment are better predictors than any standardized test we can come up with. We won’t know until we hold our accountability systems accountable.
The difference between .250 and .300
Just as a foil, I offer this link to St. Louis Cardinals outgoing manager Tony La Russa’s review of the movie Moneyball. His main point is that the truth is only half there – that a careful use of the data helped highlight positive opportunities, but that an equal dose of natural talent and old fashioned baseball know-how was also at play. Translated, a child’s life “circumstance” matters. Here’s the link…
http://www.nytimes.com/2011/10/28/sports/baseball/la-russa-disagrees-with-emphasis-of-moneyball.html
Allegedly, SAT scores are supposed to be predictive of how a student will perform in his/her first year of college, but I have serious doubts about this. I would be surprised if the R squared on that was more than 0.3. I’ve been told, by one of the people who built it, that Georgia Tech uses a “really accurate” predictive model for how a high school student will perform in college. I can tell you that my department’s admissions does not use a specific model and just looks at several numbers in the application. Our department admissions is about graduate students though, so I’m pretty sure TAKS results are not included.