Control Systems and the Great Toyota Fail

Feb. 25, 2010

"Those whom the gods would destroy they first make mad."

"Those whom the gods would destroy they first make mad."

We’ve all been treated to a grand spectacle with all the hallmarks of an ancient Greek tragedy. As General Motors and Chrysler crumbled, Toyota rose in a seemingly effortless fashion to be the largest automobile maker in the world. And then came the revelations about quality failures, and Toyota’s unwillingness to confront their problems openly, disclose them and fix them. Memos were uncovered that revealed that Toyota wasn’t any better at honesty and quality thinking than the bean counters and engineers at Ford and Audi were decades ago. As Euripides said, “Those whom the gods would destroy, they first make mad.”

Entertaining as this has been, there are some serious and specific cautionary tales here for control systems and automation professionals. For decades now, automobiles have been operated as drive-by-wire systems with many digital control systems and sub-systems. Instruments on the dashboard that appear to be analog are in reality CanBus digital indicators. Things like acceleration, braking, shifting, and even steering are actually actuator-and-sensor loops. Many of those loops use versions of PID control, just like processes in a refinery, chemical plant, food, pharma, or water or wastewater plant do.

The big difference is that the control loops in automobiles, and the control systems themselves supposedly have been engineered to be completely transparent to the user—so rugged and durable and mistake-proof that they would seem to operate exactly the same way as the old electromechanical systems they replaced. The accelerator computer would act exactly like the old mechanical linkage to the throttle from the pedal. The shifter would act exactly as if there was a linkage from the lever in your hand to the mechanical shift mechanism on the transmission.

Well, now we know that after more than twenty years, across multiple companies, the engineers who design these systems still haven’t been able to get them right. Toyota is simply the latest in a long line of auto companies whose controllers weren’t as bulletproof as advertised. GM had engine controller problems in the late 1980s. Audi had acceleration and braking issues also in the 1980s. Ford had its Pinto, and Chrysler had controller problems with the k-cars.

In the engineers’ defense, complex control systems and real-time software are almost impossible to completely troubleshoot and bug-fix before release. And the number of failures that have happened to any of them, including Toyota, graphed as a percentage of the number of systems (cars) shipped is well within the rules for six sigma quality, and the lean manufacturing principles that made the Toyota Manufacturing System so admired and so imitated.

A congressman thundered at Akio Toyoda that if a Camry or a Prius were an airplane, they’d be grounded. But if a Camry or a Prius were truly as reliable as the systems in a Boeing or Airbus aircraft, Camrys would cost $10 million, and Priuses even more. As the great Robert A. Heinlein coined the phrase, “TAANSTAFL. There ain’t no such thing as a free lunch.”

We are not willing to accept that equation. We, as all the auto makers know, aren’t willing to pay enough for failures not to happen.

And this is also true in process manufacturing. We are not willing to pay to make our control systems safe enough to prevent the long litany of fatalities of the past 100 years. We are not willing to pay operators enough to get the quality of staff needed to safely run plants. We are not willing to pay enough to design control rooms, safety systems, security systems and alarm management systems to be as bullet-proof as a Toyota’s control systems.

But it is easier to pillory Toyota as the fail-of-the-month than to accept that what modern humans do is still as dangerous as hunting leopards with stone spears. In order to decide whether we want to do more than have the president of Toyota figuratively disembowel himself on C-Span, we need to recognize that while we all piously want safety and security, based on results we aren’t willing to pay for it.