The phone rang at 4:40 a.m. “The plant is down and half the DCS and ESD workstations aren’t working,” said the voice on the line. One of two uninterruptible power supply (UPS) systems had become “interrupted,” and no one onsite knew where the bypass switch was to switch to line power. A critical piece of rotating machinery tripped, placing expensive catalyst at risk and bringing production to a halt. With a pair of redundant, 15 KVA “uninterruptible” power supplies, this chaos was supposed to be contained in a way that kept the process online.
Emil brought a PC or two back to life by replacing a power supply. Even when his spouse’s car was exhibiting curious faults and diagnostics, he replaced the battery—all the check-engine and VSA-disabled trouble lights, among others, went off, and the vehicle functioned normally. The precision scale he used for measuring hop additions for his homebrew became erratic, but cured when he swapped the batteries. But his skepticism about DC power wasn’t in his mind when it came to the three-phase, 15-KVA UPS supplying half the DC power and battery backup for his plant’s control system.
All of Emil’s basic process controls accommodated redundant, 120 VAC power from disparate sources. For both the DCS and the ESD system, the downstream AC-to-DC (typically 24 VDC or 12 VDC) power supplies were wired to rectifiers, where either DC power supply could bear the entire load of downstream consumers, switching in a bumpless fashion when one or the other’s output voltage waned. Even in remote panels, the modular, DIN-mount power supplies from a prominent manufacturer were already designed with built-in diode rectifiers to facilitate dual redundancy. The rotating machinery’s vibration, velocity and displacement-sensor I/O racks had a pair of power supplies, aiming to be redundant, which Emil ensured were fed from each of the two UPS system circuits.
Get your subscription to Control's tri-weekly newsletter.
It was common for modern servers, which provide engineering interfaces, database servers, domain controllers, historians and asset management platforms for the DCS, to have two redundant AC-to-DC power supplies that Emil ensured were fed from both redundant UPS systems. The typical DCS workstation has only one power supply, so the system was engineered to feed every other workstation in the control room with AC power from one or the other UPS. What he missed, possibly, was that each workstation’s monitors also needed UPS power from the same UPS as its workstation, as well as any line-powered speakers—useful or necessary to continue annunciating alarms during an upset. The control room’s workstation components had been plugged into outlets under the floor, fitted with twist-lock receptacles to discourage the random drill or toaster oven from corrupting the critical power source.
Emil’s scheme also left the door open to randomize the power source accessed by each component. Consequently, a workstation might keep running, but its monitors would go dark.
Such consequences were back-of- mind, at best, if anyone gave them any thought at all. The UPS was, after all, uninterruptible.
As control professionals, we might be inclined to treat our UPS—if we’re fortunate enough to have a culture that’s chosen to invest in one—as a “black box.” A 15-KVA UPS fed by three-phase, 480-VAC power is more in the realm of an electrical specialist.
Is the electrical engineer, tuned into our robustness assumptions, enough to challenge how we might choose to distribute the power supply he or she provided us? A UPS has a finite lifetime, as Emil found out, and its final throes might be erratic. High amperage semiconductors are integral to rectifiers and inverters, and large capacitors are likewise subject to degradation, and will eventually fail.
While OSHA regulations compel us to keep P&IDs up to date, there’s no clear specification or regulation encouraging us to review and document all permutations of the control system’s power infrastructure. Likewise, it’s best if we’re not surprised when an aging UPS eventually fails to have some training and procedures in place to minimize the chaos.
