Off the Kuff Rotating Header Image

TAKS

The “Moneyball” approach to public education

Via Lisa Falkenberg on Facebook, SBOE member Thomas Ratliff uses the philosophy from Moneyball to analyze the accountability system for Texas public schools.

The poster boy for the book

The book says, “One absolutely cannot tell, by watching, the difference between a .300 hitter and a .275 hitter. The difference is one hit every two weeks.”

In Texas public schools, you absolutely cannot tell the difference between an exemplary school district, a recognized district or an acceptable district simply by watching. The difference can be the performance of a small subset of students on one test on one day in the 180-day school year. This is a byproduct of our accountability system.

The book says, “The problem is that baseball statistics are not pure accomplishments of men against other men, which is what we are in the habit of seeing them as. They are accomplishments of men in combination with their circumstances.”

The accountability system doesn’t care about circumstances. It generates a report that shows how students did on a test, period. This is measuring the accomplishments of students against other students. We must change our accountability system to measure student performance in combination with their circumstances. Not all children enter or exit public schools with the same circumstances. We absolutely cannot have the same expectations for all of them, nor should we measure them all in the same manner. There are different definitions of success that involve academics, athletics, career and technology, community service, the arts, and the list goes on and on.

The book says, “I am a mechanic with numbers, tinkering with the records of baseball games to see how the machinery of the baseball offense works. I do not start with the numbers any more than a mechanic starts with a monkey wrench. I start with the game, with the things that I can see and the things people say, and I ask: Is it true?”

Our accountability system is designed to measure career and college readiness. The question is, “Is it true?” Does it predict career and college readiness? I believe it does not. My proof? To my knowledge, there are very few, if any, colleges or universities in the United States that look at TAKS test scores as part of a student’s application. If the accountability system and the state’s standardized test measured college readiness, wouldn’t you think colleges would look at it? Similarly, I’m not aware of a single business in the state of Texas that asks for TAKS test scores as part of the job application process. Again, if the system predicted career readiness, wouldn’t Texas employers use this as a part of evaluating prospective employees? We need an accountability system that takes a broader look at a student’s K-12 education and provides a measurement that will be useful to colleges, universities and employers.

Just for the record, it was the movie Bull Durham that first made the observation about being a .300 hitter. Be that as it may, a couple of points. One, while everyone talks about the statistics when discussing “Moneyball”, the central insight that Billy Beane had wasn’t just that on-base percentage and slugging average correlated better to winning than batting average does, it’s that (at the time, at least) those skills were valued less in the marketplace than batting average was. As a low-budget team, the A’s needed to take advantage of market inefficiencies like that to overcome their financial disadvantage. That’s beyond the scope of Ratliff’s analogy, but as this was the most misunderstood part of the book, it needs to be said.

Now that I’ve gotten that out of the way, I think Ratliff is on to something here. Is it true that TAKS scores correlate to success in college? More to the point, do TAKS scores correlate better than other available measures to success in college? I don’t know, and it’s not clear to me that anyone else does, or at least that anyone in a position of authority does. This is an easy enough question to answer, if we’re tracking how students ultimately fare in college. Let’s crunch the numbers and see what we get. Maybe TAKS scores are a good metric. Maybe there’s something else, like writing ability or extracurricular participation, that correlates better. Maybe we’ll find that external factors like a family’s income level and prior educational attainment are better predictors than any standardized test we can come up with. We won’t know until we hold our accountability systems accountable.

Higher standards mean lower ratings

Schools across the state have seen their academic ratings drop as a result of changes made in how the Texas Education Agency computes them.

The new accountability ratings released Friday for public school campuses in the state’s 1,228 districts and charter schools present a “far more accurate look” at academic performance, Texas Education Commissioner Robert Scott said.

They are also markedly lower — with far fewer schools achieving the highest ratings than last year. Instead, most schools fall in the middle “acceptable” category.

Many districts find themselves with lower ratings even though their student achievement has remained the same. That’s because the formula used to calculate the ratings, based primarily on students’ standardized test scores, no longer includes a mechanism called the Texas Projection Measure. The TPM gauged students’ future test scores based on a campus-wide average instead of using their actual test scores and had the effect of giving schools credit for students passing when they hadn’t.

In April, Scott announced he would discontinue the measure after state lawmakers took a unanimous vote against it during debate on a testing bill.

The Chron gives the local picture.

In the Houston Independent School District, the “unacceptable” campuses more than tripled to 25 — or 9 percent of its rated schools.

Statewide, about 7 percent of schools netted the lowest rating this year. The unacceptable list grew from 104 schools to 569.

[…]

The ratings, from best to worst, are exemplary, recognized, acceptable and unacceptable.

Cypress-Fairbanks ISD, honored last year as the state’s largest “recognized” district, dropped to “acceptable.” HISD, the biggest district, also was “acceptable.”

Among the area’s other large districts, Katy, Pasadena, Conroe, Alief, Klein, Clear Creek, Humble, Lamar Consolidated, Galena Park and Pearland earned “recognized” status.

You can see ratings for all HISD schools here, and for all school districts in Texas here. It is important to remember that last year’s ratings were basically bogus. If you do keep that in mind, HISD actually showed some improvement.

The news that Houston ISD’s number of exemplary schools dropped from 101 in 2010 to 59 in 2011, according to the Texas Education Agency’s figures just released at 1 p.m. today, could only add more fuel to the fire of critics who are certain Superintendent Terry Grier is destroying HISD.

Except that if the now discarded and discredited Texas Projection Measure (a method of giving extra points to schools by predicting that certain kids who failed the Texas Assessment of Knowledge and Skills actually would pass in the next year) was removed from last year’s results, and other new “accountability measures” were factored in, according to HISD, then in 2010 there were 46 HISD schools that were really exemplary.

Which would make this year — at least in the exemplary category — an improvement. And Terry Grier a hero (or at least not a complete goat)?

Elsewhere in the annual ratings, the number of HISD’s academically recognized schools in 2011 was 106 (107 last year with the TPM), and academically acceptable increased to 79 (from 49 with TPM).

The number of academically unacceptable schools soared to 21 from last year’s 7 — but HISD’s recalculation last year’s effort says it would have been 23 — so hey, put another one in the win column.

In addition to the dropping of TPM, there were other ways in which the accountability system was made more difficult. Special ed kids were counted for the first time, and the standards for kids with limited English proficiency and math scores were raised. And before you get too used to this new/old way of scoring things, get ready for them to change again.

This is the last year for the TAKS testing program, which began in 2003. Schools will get a one-year reprieve from ratings as students take the new exams, expected to be more challenging.

Test scores traditionally rise over the years as teachers and students get used to the format of an exam. Statewide, at least 90 percent of students passed the TAKS in reading, writing and social studies this year. At least 80 percent passed in math and science.

HISD saw its scores remain mostly flat this year. The district’s passing rate in math rose two points to 83 percent, while writing dropped two points to 91 percent.

“Schools have a pretty good routine based on the TAKS,” said state Rep. Rob Eissler, a Republican from The Woodlands who chairs the House Public Education Committee. “It will change when we get to the (new) end-of-course exams and the STAAR tests.”

The forthcoming STAAR standard is already causing a lot of anxiety in school districts. The good news for Houston teachers is that they will be cut some slack in their evaluations.

​In a startling reversal of previous statements and his own avowed philosophies, Houston ISD Superintendent Terry Grier today released a statement that he will recommend to the school board that teachers not be evaluated by their students’ test scores this next school year.

It was only in May that trustees — urged on by Grier — voted 7-2 (Carol Galloway and Juliet Stipeche dissenting) to include student test scores in the formal list of criteria used to evaluate a teacher’s performance.

The May vote came after several months of entreaty from HISD teachers who argued that it would be especially inappropriate this coming year to judge teachers on their students’ test scores given that the state was introducing a new standardized test system that is replacing the Texas Assessment of Knowledge and Skills. (Historically, student test scores drop after a new test is adopted.)

But Grier and his administration had remained adamant that it was inconceivable that the district have this information — student test scores — and not use it to evaluate the effectiveness of teachers in the classroom. They have repeatedly said that teachers are the most crucial element in whether a child succeeds or fails in school.

It is difficult to understand what new information became available in the two months since May that would change Grier’s position on this. In his statement, he references “feedback we’ve heard this summer from teachers about taking on these challenges,” but he certainly heard plenty of this feedback before school was out.

Better late than never. How did your school do?

Lege loosens graduation requirements

A sign of the times.

The Texas House tentatively approved legislation Wednesday to make it easier for high school students to pass end-of-course exams, a move critics called “a substantial retreat” from school accountability.

“This bill creates a clear, understandable path to graduation,” House Public Education Chair Rob Eissler, R-The Woodlands, said of his bill, HB 500.

Business and education reform groups complained the legislation would weaken efforts to make sure all high school graduates are college- or career-ready.

Here’s HB500, which received final passage by a 138-5 margin on Thursday. Here’s the Trib on some key aspects of the bill:

The idea behind Eissler’s bill is to provide a transition period for students as schools move from the TAKS to the STAAR tests — whose more rigorous standards some believe could lead to large numbers of students failing to meet graduation requirements. Right now, students can’t graduate unless they get a certain cumulative score across all the year-end tests. Fifteen percent of their final grades is based on how well they do on those tests. HB 500 does away with those requirements, instead allowing districts to set their own policy on how end-of-course exams weigh in student assessment. Eissler, who chairs the House Public Education Committee, said his bill was about “trying to get out of the micromanaging of school business from Austin” and vehemently denied accusations from his colleagues that it weakened school standards.

Three amendments from state Rep. Scott Hochberg, D-Houston, were adopted from the floor. One prevented double-testing for fifth and eighth grade students taking advanced courses. Another, in an allusion to this summer’s Texas Projection Measure kerfuffle, specified that the Texas Education Agency could not use a projected achievement level to measure student growth. The last allows districts to opt into a pilot program to study whether students are “overtested.” Hochberg said that there is “pretty clear data” that show that if students pass a test one year, they are more than likely to pass it the next. “If we know they are going to pass that test, why are we going to continue to test them?” he asked. (Hochberg’s HB 233, co-sponsored by state Rep. Dan Huberty, R-Houston, would implement this policy statewide.)

I’ve blogged about HB233 before. It’s a good idea. Abby Rapoport has some more details and context.

School districts had fought for the bill—House Bill 500—which was carried by Public Education Committee chair Rob Eissler, R-The Woodlands. But amendments prompted heated discussions about just what role testing should play in school assessments. And the coalitions for and against were anything but predictable.

The measure centered around the new STAAR tests, the soon-to-be-implemented statewide school assessments set to replace the current TAKS tests. Eissler’s bill would give school districts an opportunity to cut students some slack while students adjust to the new testing system. If the districts so chose, for a transitional period, a student’s STARR test performance wouldn’t necessarily count toward their final grade in a course. Districts could set their own policy on just how much the assessments count for a student’s grade. The bill also allows districts to suspend a new graduation requirement that students maintain a cumulative passing rate on 12 exams in four subject areas. Instead students would only have to pass four exams total—English III and algebra, specifically as well as one in science and one in social studies.

I don’t have a fully formed opinion of this bill yet. Mostly, I agree with Rapoport in that this bill won’t have nearly the effect on student performance that the budget will. Maybe when we’re at a point of fully funding education again, we can revisit this and see if it’s still needed. Until then, the budget is the cause and everything else is effects.

Should we do away with school police forces?

Grits makes the case.

If public school budgets will be radically cut in Texas, a prospect which for the moment appears all but inevitable, which employees should be eliminated first? Judging from the ongoing debate, maybe campus cops. Jason Embry at the Austin Statesman describes some of the debates surrounding school budgets thusly:

One of the most important dividing lines in the discussion about the state’s budget crisis separates those who think Texas schools need more money and those who think schools just need to make better spending decisions.

Those in the second group have some powerful numbers on their side. In a December report, Comptroller Susan Combs found that per-student spending increased 63 percent over the previous decade. That growth rate was nearly twice as fast as inflation, as measured by the Consumer Price Index, and it points to a Texas school system that isn’t starving for cash.

Another statistic in wide circulation these days says Texas school districts employ about as many nonteachers as teachers. This has led many to suggest that, even as lawmakers consider billions of dollars’ worth of funding cuts to schools, local education officials can balance the books without shedding teachers.

I’ve not seen hard data, but based on anecdotal accounts I’d suggest that the growing number and size of school-based police forces likely account for a big chunk of growth among nonteacher school employees in the last decade. Shouldn’t they be among the first to get the budget axe? They’re the only sizable class of school employees we know for sure they can do without because schools did so for most of their history in Texas and elsewhere. The phenomenon of campus-based police departments is something that’s really only arisen en masse in the last 20 or so years in Texas public schools.

He notes that Sen. John Whitmire has advocated greatly reducing the amount of tickets that school cops write, which would fit well with this idea. For what it’s worth, I don’t think there’s a whole lot of savings in this – as I reported before, according the HISD Trustee Anna Eastman, HISD budgets $13.5 million for its police force, of which 95% is personnel costs. That ain’t nothing – it’s 270 teachers, assuming $50K per year in salaries – but it’s less than eight percent of the optimistic-case $171 million projected shortfall. Maybe it would be more in some other ISDs, I don’t know. I think there’s merit to the idea, and not just for budgetary reasons, I’m just trying to keep perspective on it. What do you think?

On a side note, I can’t leave this subject without pointing you to Martha’s posts about why schools need more support staff, not less, and why gutting educational service centers are a bad idea. That Jason Embry article linked by Grits also gets down to it:

In 2000, 49 percent of Texas students were considered economically disadvantaged. In 2010, that number reached 59 percent. These students often need extra attention as they move through the system.

As the student population has changed, Texas has continued to pile more demands on schools, and it costs money to meet those demands. Schools began giving the Texas Assessment of Knowledge and Skills, a much tougher exam than its predecessor, in 2003, and began that year to require students in the third grade to pass the reading section of the test to advance to fourth grade. Today the test is tied to promotion in grades five and eight. In addition, students who used to graduate from Texas high schools with three credits in math and three in science now must have four credits in each. To meet these demands, schools have spent more on student remediation, teacher training and the renovation of science labs.

Schools are preparing to give a new test next year, the State of Texas Assessment of Academic Readiness, which the Texas Education Agency has promised “will be significantly more rigorous than previous tests.” And let’s not forget that, led by our last governor, the federal government created an additional set of accountability measures for schools to meet during the past 10 years.

The increasing demands on students have put more demands on teachers and principals, particularly considering the state’s heavy emphasis on standardized testing to judge schools.

Districts across the state have therefore decided to hire instructional coordinators, curriculum specialists and others to give students extra attention and to help teachers make sure their lessons help students meet the escalating expectations.

In other words, yes, schools spend more than they used to. But the people of Texas also ask their schools to do more than they used to.

Funny how that latter part always seems to get overlooked by the “schools have too many administrators” crowd. In addition, as BOR notes, the cost of administering TAKS tests in Texas increased tenfold from 1999 to 2009. There’s been way too much talk in this debate about what schools do or don’t need by people who probably haven’t stepped foot in a public school in forty years, and it’s drowning out those who are there every day trying their best to make it all work. Martha’s a fine example of the latter, so please go see what she has to say.

Cutting the budget means cutting education

No two ways around it.

As the single biggest consumer of state money, the Texas public education system stands to lose millions of dollars as the state grapples with a looming budget shortfall.

Education Commissioner Robert Scott has suggested more than $260 million in cuts from the state’s almost $40 billion education budget for the next two years. Some of those would reach into the classroom, eliminating money for new science labs, textbooks and teacher development. Those recommendations have infuriated teachers.

Gov. Rick Perry’s “budgetary policies are wrecking the public schools and jeopardizing our children’s future,” said Rita Haecker, president of the Texas State Teachers Association. “The governor can talk all he wants about school savings … but most districts and educators are already stretched so thin, there is little, if anything, left to save.”

The budget proposal for the Texas Education Agency would ax millions of dollars for a teacher mentoring program and other continuing education opportunities for teachers. It also would cut $35 million that was set aside in the previous budget to help schools build new science labs to go along with a new requirement that high school students take four years of science classes.

The reality is very simple. Texas has a young and growing population. A large and increasing number of public school students come from poor and/or immigrant families. School districts are completely strapped, thanks to the economy and the property tax cuts from the 2006 special session. How much more cutbacks can schools take? And why won’t Commissioner Scott show up at legislative hearings to answer these questions?

I’ll say it again, for the umpty-umpth time, that what we have here is first and foremost a revenue problem. At least some members of the Republican leadership are willing to admit that, even if they won’t admit that they caused this problem in the first place by supporting that ginormous unaffordable property tax cut from 2006. The system they want to scrap now is the one they created before as the solution to the previous system that they said needed to be scrapped. How many times are we going to repeat the same mistakes before we try a different approach?

It’s true, as Rep. Scott Hochberg discussed in his interview with me that there are savings to be found in the public school budget. They involve reallocating resources, not resorting to the lazy tactic of across-the-board cuts, as if no item in the budget is more important than any other. It does have the advantage of being easier than thinking, though.

Speaking of thinking, it would be a good idea if we all did some about the new end of course exams, their potential effect on graduation rates, and how we can best equip our teachers to get students ready for them. I expect exactly nothing on this from Governor Perry or Robert Scott, so it’ll be up to the rest of us to figure it out.

How the schools are really doing

I think I’m just going to let the picture tell the story:

If you want the words, go read the Trib story. I get that the TPM is supposed to measure growth, and that growth can and does occur with students who didn’t pass their assessment tests. But if that growth that the TPM says is occurring really is occurring, then that should show up in subsequent test results. Students that the TPM predicts will pass should eventually pass. I don’t know about you, but I don’t see much evidence of that in these data.

TEA Commissioner Scott defends Texas Projection Measure

Texas Education Agency Commissioner Robert Scott takes advantage of a friendly audience to lash out at critics of the Texas Projection Measure.

Scott, speaking to the State Board of Education, said the so-called Texas Projection Measure has been misunderstood and misrepresented by critics who contend the policy gives a false impression of school performance.

The complex formula allows schools and districts to count as passing some students who actually fail the Texas Assessment of Knowledge and Skills if the projection measure shows they are likely to pass in a future year.

“There is a little bit of election year politics going on here,” Scott said. “It is very easy to demagogue. It is very easy for someone to say they gave students credit for failing.”

Too bad he didn’t have the guts to say this to Scott Hochberg. It would have been nice to know how he would have answered those questions, instead of leaving his assistants to hang out to dry.

The commissioner also pointed to scores of e-mails from superintendents, principals and teachers across the state who wrote that the projection measure was beneficial for their students and schools — and should be retained. The Dallas Morning News obtained copies of all e-mails received by the Texas Education Agency through the beginning of this week.

“Please keep TPM and do not suspend the use of the TPM for school accountability ratings,” said Lewisville High School Principal Brad Burns, reflecting the viewpoints of numerous principals in Texas.

“Whether TPM was good, bad or in-between, we had children for the first time in their lives that experienced success,” wrote Temple schools Superintendent Robin Wuebker-Battershell. “Retool it if necessary, but don’t surrender the concept.”

And Weatherford High School Principal David Belding urged Scott to please “not dismantle a system that gives schools with more difficult student groups to educate the chance to be recognized for moving those students forward. That is what TPM does.”

Look, nobody is attacking the idea of a means to measure growth. My understanding is that such a thing is required by No Child Left Behind, so totally scrapping it isn’t an option. The problem is that as a way to measure the growth of students who are not already passing their tests – that is to say, to measure the growth of the students it was really designed to measure – TPM sucks. In mathematical terms, it’s a lousy model. Pointing that out isn’t politics, but distorting that criticism is. Can we please focus on the real issue, so that we have accurate data about our teachers, students, and school districts and so that the real progress they have made doesn’t get lost under the weight of a bad metric? Thanks.

Big education bills pass

Well, the SBOE may be doing its best to destroy public education in Texas, but the Lege took a step forward to make it better by passing omnibus school reform bills in each chamber.

Crafted by the education leaders in the Senate and House of Representatives, the bills aim to reduce the role of standardized tests, give schools more flexibility to help struggling students and focus education on readying students for college or the workplace.

Gone are many of the school reforms ushered in by then-Gov. George W. Bush, such as a prohibition on promoting a student to the next grade if he or she failed to pass the Texas Assessment of Knowledge and Skills.

That promotion decision will now be left to the school and parents.

[House Public Education Committee Chairman Rob Eissler, R-The Woodlands] said the overhaul will end the one-size-fits-all approach of the current system and allow for schools to be judged on more than just performance on a single test.

The bills are HB3 and SB3, which are very similar but not identical and thus will go through a reconciliation. Both passed unanimously, so it ought to be relatively smooth sailing. EoW has more.

Is the TAKS test at the end of the line?

If so, there’s a lot of people who won’t be sorry to see it go.

“We have counted on testing and testing only. And it’s caused a lot of angst in the schools,” Senate Public Education Chair Florence Shapiro, R-Plano, said Wednesday about the Texas Assessment of Knowledge and Skills.

“We’ll still test, but we’re using other variables to give us the results that we need.”

Shapiro and House Public Education Chair Rob Eissler, R-The Woodlands, plan to file the school accountability legislation on Thursday. The changes — which would start in the 2011-12 school year — aim to gradually elevate Texas into the top 10 states when it comes to preparing students for college or equipping them with workforce skills.

Texas ranked 46th in the country last year in the Scholastic Assessment Test scores and last among all states in the percentage of adult population with a high school diploma.

The bills in question are HB3 (Eissler) and SB3 (Shapiro). That’s quite a lofty goal they’ve set for this legislation, but a worthwhile one.

The legislative proposal contemplates a “Texas diploma” for college-bound students and a “standard diploma” for those seeking skilled workforce training and a related career. The standard diploma would require three years of English and one year of algebra.

“This diploma will be in a field that says you are certified and are skilled workforce ready,” Shapiro said.

Students would be measured by individual improvement instead of a single test score. Existing “exemplary” “recognized” and “acceptable” ratings for schools and school districts will be eliminated and replaced by an “accreditation tier” focused on individual student achievement based on readiness for college or career.

High school, middle school and elementary school campuses also can earn distinctions for excellence in a variety of areas, such as growth in student achievement, workforce readiness, second language learning, fine arts and physical fitness.

Student testing “will cover more than minimum skills,” Eissler said. Tests will be given in each grade level in an effort to get “an instant growth indicator,” Eissler said, measuring a student’s academic improvement from one year to the next.

We’ll have to see what the details are, but I like the general concept. The purpose of school is to prepare you for what comes next, and I think it makes more sense to evaluate them on that kind of metric than on a standardized test one, which is easy to game and doesn’t really measure anything useful. This is going to be a lot trickier to do, and I’ve no doubt there will be problems and disagreements with the implementation. But the direction strikes me as the right one, and so I hope this makes it through. EoW has more.