Putting lipstick on a catcher

I nearly choked on my lunch today when I saw the Chron not only address the grievous shortcomings of Brad Ausmus as an offensive player, but try to make up for it by the use of Catcher ERA as a measure of how good he really is for those who know where to look. "Catcher ERA", for those who are unfamiliar, is the combined earned run average for the innings that each pitcher throws while a given receiver is behind the plate. And at first glance for Ausmus, it looks pretty good:

Since the start of 2001, the Astros have a 3.75 ERA when Ausmus is behind the plate. The team ERA with anybody else catching: 4.66. Ausmus, in other words, connects with pitchers of all sizes, shapes and persuasions.

That's pretty damned impressive, especially given that it's considered an open question whether or not Catcher ERA is a real effect that carries over from one season to another and is a true measure of differences in ability. The Ausmus result would be a clear indicator that something is really happening.

Except for one nagging little doubt that can be best expressed by reading the next paragraph:

He helped coax a 38-18 record and 2.40 ERA out of the emotional Clemens the past three seasons. Roy Oswalt has fashioned a 98-47 record since reaching the majors in 2001 with a demeanor so stoic, Ausmus says, "You wouldn't be able to tell if Roy was in the middle of Game 7 of the World Series or watching Archie Bunker on TV."

What that suggests to me is that Ausmus has spent most of his time catching the Astros' best pitchers. This stands to reason - whatever one may think of the stats here, Ausmus does have a great reputation in the clubhouse, and you can be sure that the staff aces want him behind the plate when it's their turn on the mound. What's more, at least for last season, when the Stros had a 3.80 ERA in 1124 Ausmus-caught innings and a 5.00 in 344 innings without him, I can prove it to a certain extent.

Thanks to the magic of Retrosheet, I can tell you that catchers other than Brad Ausmus (mostly Eric Munson) caught 127.1 of Wandy Rodriguez's 135.2 innings, and that Rodriguez's ERA was 5.86 for that time. (For the remaining 217.2 non-Ausmus innings, other hurlers had a 4.55 ERA.) Of course you're going to look good by this metric if you're catching Roy Oswalt and Roger Clemens while everybody else gets Wandy Rodriguez. Cliff Johnson and Mitch Melusky would look good, too.

I don't have the time to be completely obsessive about this, but I can tell you that it's the same basic scenario in 2005, too - Ausmus caught exactly one game that Wandy Rodriguez started. That's a smaller fraction of the non-Ausmus innings - about 120 of the 377 - but again, Wandy's ERA was in the 5.50 range for those innings. Bottom line is the same: Ausmus gets the wheat, the rest get the chaff.

I don't mean to pick on either Wandy Rodriguez or Brad Ausmus here, but the point I'm making is simply that for this statistic - for any statistic - to mean something, it has to be an accurate measure of something. The question is not "How much better does the Astros staff do with Ausmus catching", but "How much better does each individual pitcher do with Ausmus catching". That's why Catcher ERA is such a shaky metric - the relative sample sizes (i.e., Clemens with Ausmus versus Clemens without Ausmus) are too small to be meaningful. And as we see, aggregating to compensate for that has other issues. It may well be that the Stros throwers do better with Brad Ausmus as their batterymate. All I'm saying is that you can't tell that from the information given in this story.

Posted by Charles Kuffner on March 12, 2007 to Baseball