Industry stakeholders commend GAO report on CSA scoring system

A report issued this week by the Government Accountability Office (GAO) calls for various changes to the much maligned Federal Motor Carrier Safety Administration’s (FMCSA) Compliance, Safety, Accountability (CSA) program. 

CSA was designed to weed out as many as 5 percent—or 150,000—of the nation’s 3 million or so long-haul truck drivers that the feds believe are involved in a disproportionately high number of truck accidents and fatalities. CSA uses a complex scoring system to rate the nation’s nearly 700,000 DOT-registered interstate trucking entities on seven “Behavior Analysis and Safety Improvement Categories,” known as “BASICs.” The seven BASICs are driving, fatigued driving, driver fitness, alcohol and drugs, vehicle maintenance, cargo security (now HM Compliance) and crash history. Carriers are given “scores” in each category—higher the score, worse the performance. So-called “warning letters” go out to fleets with scores above 65 (which means that only 35 percent of carriers in their class have worse scores). For hazmat carriers, the cutoff score is 60.

The GAO report focuses on CSA’s Safety Measurement System (SMS), which uses carrier performance data collected from roadside inspections or crash inspections to identify high-risk carriers for intervention by analyzing relative safety scores in various categories, including Unsafe Driving and Vehicle Maintenance.

In December 2012, FMCSA rolled out multiple CSA SMS changes that it said were based in part on public comments received from a preview of proposed changed to its SMS Web site, and they noted at the time that in the first year of CSA enforcement violations per roadside inspection and driver violations per inspection decreased by their largest amounts in ten years at 8 percent and ten percent, respectively, coupled with 30 million visits to the SMS site. And they added that these changes, which were developed over several months with feedback from public and private industry stakeholders, would enhance FMCSA’s ability to identify and take action against trucks and buses with safety and compliance concerns.

The SMS changes that took effect in December 2012 include:
-strengthening the Vehicle Maintenance BASIC (Behavior Analysis and Safety Improvement Categories) by incorporating cargo/load securement violations from today’s Cargo-Related BASIC;
-changing the Cargo-related BASIC to the HM (hazardous materials) Compliance BASIC to better identify HM-related safety and compliance problems;
-better aligning the SMS with Intermodal Equipment Provider regulations; and
-more accurately identifying carriers that transport significant quantities of HM, among others

Even well before theses changes took effect, industry reaction to CSA has been frosty at best. In many trucking circles, the general consensus is that CSA is an ill-conceived plan that is doing nothing more than making it difficult to get things done in an already-challenging environment.

The American Trucking Associations has said that CSA scores are unreliable, have a loose and sometimes inverse connection to crash risk, coupled with the fact that they say the FMCSA is demonstrating an unwillingness to discuss CSA’s flaws.

In its report, the GAO explained that FMCSA has at least two key challenges in reliably assessing safety risk for most carriers:
1 -for SMS to be effective in identifying carriers more likely to crash, the violations that FMCSA uses to calculate SMS scores should have a strong predictive relationship with crashes, but based on GAO’s analysis of available information, most regulations used to calculate SMS scores are not violated often enough to strongly associate them with crash risk for individual carriers; and
2- most carriers lack sufficient safety performance data to ensure that FMCSA can reliably compare them with other carriers

SMS scores based on violation rates calculated for each carrier and then compares rates to other carriers, but with most carriers operating few vehicles and inspected infrequently there is insufficient information to produce reliable SMS scores, said GAO. What’s more, it added that by only scoring carriers with more information, FMCSA could better identify high-risk carriers likely to be involved in crashes. And even though this methodology would assign SMS scores to fewer carriers, these scores would be more reliable and useful.

In its recommendation for executive action, GAO said “To improve the CSA program, the Secretary of Transportation should direct the FMCSA Administrator to revise the SMS methodology to better account for limitations in drawing comparisons of safety performance information across carriers; in doing so, the Secretary of Transportation should direct the FMCSA Administrator to conduct a formal analysis that specifically identifies: (1) limitations in the data used to calculate SMS scores including variability in the carrier population and the quality and quantity of data available for carrier safety performance assessments, and (2) limitations in the resulting SMS scores including their precision, confidence, and reliability for the purposes for which they are used.”

These recommendations were soundly endorsed by Jeff Tucker, CEO & Founder at QualifiedCarriers.com & CEO Tucker Company Worldwide Inc.

“This is welcome news, and not a surprise to anyone who is objectively looking at CSA,” he said. “Academia, truckers, brokers, shippers, some people within FMCSA, former FMCSA leadership, Congress and others are vindicated. Top FMCSA leadership have been downright obstinate in their refusal to call CSA what it is—a good improvement on SafeStat, that needs a lot more work. FMCSA is guilty of being data-blind and not a bit humble or realistic in the usefulness of CSA.”

Instead of taking stakeholder feedback, and improving their subjective and data-starved, fledgling program, Tucker said FMCSA has acted like CSA is a mature program above reproach. Before improving CSA for their own use, and demonstrating its effectiveness over time, he said FMCSA told shippers, brokers and carriers from the start, to select carriers using CSA.  He explained two of the seven BASIC scores are purposely hidden from the pubic, two more, of the five visible BASICS are admittedly broken, and definitely confusing.

“That makes 4 of 7 BASICs useless,” he said. “FMCSA has tons more data like driver records and other data are also hidden; its own CSA Subcommittee pointed much of this out to FMCSA in April 2013. This doesn’t cover most of the obvious problems with CSA. Still, FMCSA still has the audacity to speak to industry groups and state shippers and brokers should use the data for carrier selection. The notion is laughable to everyone but FMCSA and accident chasing lawyers, and to some naïve safety advocates. Thank you GAO.”

The ATA also applauded the GAO’s findings.

“Given GAO’s findings, FMCSA should remove all carriers’ scores from public view,” said Dave Osiecki, ATA executive vice president and chief of national advocacy, in a statement. “Since scores are so often unreliable, third parties are prone to making erroneous judgments based on inaccurate data, an inequity that can only be solved in the near term by removing the scores from public view.”