Last week, when I wrote about the anti-red light camera folks turning in their petition signatures, I noted that the Chron story referenced an update to the January 2009 study about the effect of cameras on the collision rate at the monitored intersections. That study reported an overall increase in collisions at all intersections, whether monitored by a red light camera or not, with the monitored intersections showing a smaller increase than the unmonitored ones. This result was both puzzling – How is it that there was an increase in collisions in Houston when the data for the state as a whole showed a drop in the collision rate? – and controversial – ZOMG! Red light cameras meant more crashes! – but at least there was to be a followup study, which hopefully might shed some light on that.
That study was completed in November of 2009. I was sent a copy of it, which you can see here. The results this time were very different.
In January of 2009, we released a report analyzing the effect of red light cameras at the 50 DARLEP [Digital Automated Red Light Enforcement Program] intersections. The report concluded that red light cameras were mitigating a general increase in collisions at the monitored intersections. We based this conclusion on the fact that collisions occurring on intersection approaches with red light cameras were rising at a significantly slower rate than collisions occurring on approaches without camera monitoring. This conclusion was based on data drawn from a collection of individual incident reports provided by the Houston Police Department (HPD).
In the spring of 2009, the Texas Department of Transportation (TxDOT) released an updated statewide database of collisions digitizing all paper incident reports available. The database is known as the Crash Record Information System (CRIS). In theory, the CRIS data for the 50 DARLEP intersections and the original HPD data should be identical as they are both based on the same incident reports. However, in a comparison of the two datasets, we found CRIS reported over 250% more collisions during the before-camera period and over 175% more collisions during the after-camera period. From the comparison of CRIS to the HPD data and after consultation with HPD, we determined the original data in first report was inaccurate as a result of a substantial undercounting of collisions in both the before- and after-camera periods. We then conducted an analysis similar to the original report, but with the new CRIS data. We compared the rate of collisions before the red light cameras were installed to the rate of collisions after the cameras were installed. Because the cameras were installed on only one approach at each intersection1, we separated the data into those approaches that were not monitored by red light cameras and those approaches that were monitored by red light cameras.
The comparison of collisions at monitored and unmonitored approaches leads us to conclude that the Houston red light camera program is reducing collisions at the 50 DARLEP intersections (see Exhibit 1). After the implementation of red light cameras, collisions per month at monitored approaches decreased by 11%. This decline was statistically significant – that is, not due to random variations in the data, with over 90% confidence. The number of collisions per month at unmonitored approaches increased by approximately 5%. This difference from the before-camera period was not, however, statistically significant; the probability that the observed change did not occur due to chance was less than 90%.
The main point to understand here is that the original study was done with incomplete data. I had the chance to speak to Drs. Bob Stein and Tim Lomax about this, and what they told me was that they used HPD’s accident reports for the initial study. These reports were all on paper, and came from various HPD locations. It turned out that a sizable number of the reports were not provided at that time because they were in offsite storage facilities, and nobody they were working with knew about that. Stein and Lomax stressed to me that they had no problems with HPD, they cooperated fully and provided all the data they thought they had, it was just that there was quite a bit more than that.
Anyway, once they had their hands on the CRIS data, which was fully digitized, from TxDOT, it became apparent that there had been no increase in accidents, there had just been a disparity in the number of paper reports they had from before and after the camera installations, which had made it look like there had been an increase. Doing the study on this complete data set yielded the results above, which are much more in line with the original expectation that there would be fewer collisions at monitored intersections.
Unfortunately, that’s not the end of the story. TxDOT has since announced that there were some issue with the CRIS data, in particular with GPS information. This matters because without being confident in the exact location of a crash, you might classify a collision away from an intersection as being in the intersection, or vice versa. TxDOT will be issuing an updated data set in the next few weeks that will supersede the one on which this study is based. Because of all that, Drs. Stein and Lomax told me that they no longer have any confidence in the reliability of the November 2009 study, and that no conclusions should be drawn from it. Here is the memo expressing their concerns, which was sent to HPD Assistant Chief Tim Oettmeier last week:
We have identified several issues with our revised report dated November 2009. These issues and their potential effects on our analysis are outlined below:
1. TXDOT advised us that they would be reprocessing existing crash datato correct data errors, append current roadway data, and update crash location information.
2. As we have refined our data processing, we discovered potentially incorrect data that will require further analysis (e.g. JFK/Greens Rd.).
3. The November 2009 report uses a 500 ft. inclusion standard. Upon further review of the literature, we have decided that a 150-200 ft. inclusion standard is appropriate.
1. Collisions are relatively rare events. Even a small change in the number of collisions can have a significant effect on the results of our analysis. For this reason, we must be sure we are using the “cleanest” data possible. The reprocessing of the Crash Records Information Systems (CRIS) data has the potential to significantly alter the results of the November 2009 report and we believe it is best to withhold judgment until the new TXDOT data is available. We cannot be sure of the reliability of the underlying data in the report.
2. When we collected/processed the CRIS data, there was an error in our geolocation of crashes at the JFK/Greens intersection. This error needs to be corrected and we are planning to do so with the new August data (which will include data through 2009). The error adversely affects the reliability of the report itself.
3. Upon further discussion with transportation experts and additional review of the extant literature, we have discovered that the 500 ft. inclusion standard in the November 2009 report was potentially an overly broad standard for collisions included in the dataset. We erred and are correcting this error in a report to be released soon after the revised CRIS data is available.
When taken individually, a given issue may not be insurmountable. However, the compound nature of the effects prevents us from affirming the reliability of the November 2009 report. Erring on the side of caution, we believe it is best to issue a corrected report once we have an opportunity to utilize updated CRIS data (availability of which is anticipated later this month).
I will report back after I’ve received a copy of the revised study. The main point to take away here is that the original January 2009 study, which is regularly cited by camera opponents as evidence of their ineffectiveness, was based on incomplete and inaccurate data, neither of which was known at the time. We should finally have an idea of what the data really tells us after this third study is done.
Two other points of interest. One is that according to Stein and Lomax, theirs is the first study of red light cameras in Texas that utilizes the CRIS data. I hope someone will perform similar studies in other red light camera-enabled cities with this data – once we’re sure it’s as clean as it’s going to get, of course – so we can have a true apples to apples comparison across cities. There’s no indication who did the study cited in the Grits link above or what data they used, so I can’t offer a critique of it. Clearly, it’s a tough issue to wrap your arms around.
Second, I asked Stein and Lomax why it was that I hadn’t seen any references to that November 2009 study before now. They said that was a question for the city – it was their job to produce the study, not to publicize it. I’ll just leave it at that.