"How do we want to respond to a DeltaV upset? Things happen, then there's a reboot. How do you know about it?" asked Anthony DeJohn, R&D process systems group leader at the Bristol-Myers Squibb research facility in New Brunswick, New Jersey. "DeltaV diagnostics are good, but we needed more. It's my job to make sure operations has information immediately. They can't fix a problem they don't know about."
DeJohn spoke on October 9 to a standing-room-only crowd at the Emerson Global Users Exchange in Orlando, Florida. His presentation won a "Best in Track" award in the event's Operations and Management track.
DeJohn's facility is an R&D site that develops processes and produces pharmaceuticals for trials. "Our group develops software applications on DeltaV to produce pharmaceuticals," he said. "Ours is a regulated environment, so of course we need data to meet regulations. We also need data for our scientists, and our data must have integrity—we do not want to release a product that puts the public at risk."
The facility has multiple DeltaV systems. "Faults could be minor to major," DeJohn said. "They can impact data or production and range from network upsets to service hang-ups to hardware failures. An undetected fault could lead to investigations and potential delays."
The question was, "How do we respond?" DeJohn said. "When and how do we detect problems? How long will it take to determine the extent of the impact? Do we know who to contact? Do we know what to tell them? Is there a repeatable response to upsets? Are we relying on a process, or on exceptional talent?"
The plant's prior approach was a reflection on the available tools. "The only program was DeltaV Diagnostics," DeJohn said. "Se we had defined a set of parameters that define a healthy DeltaV system, but we did not have enough instruction on how to communicate or proceed in the case of a fault. Without instruction, I was always amazed at who they would tell—it might be someone three levels up in management, or two levels too low. "It was too reactive. DeltaV Diagnostics is great for troubleshooting, but it doesn't tell anyone else."
To get diagnostics into mainstream communications, the facility now uses DeltaV Control Studio. "We use a control module, our traditional workspace," DeJohn said. "For any parameter of interest, you can copy and paste the ‘diagnostics parameter path' from Diagnostics into Control Studio."
The approach opened "a ton of possibilities," DeJohn said. "We can check nodes for health, communications, time offset, time synchronization. We can check our batch historians—one primary, one backup—to know if one is not working. DeltaV is now watching itself 24/7, whether someone's there or not."
The hard part is to decide what is really important. "Decide that, and it's just a little work to put it in a module," DeJohn said. Now, instead of scanning many screens in Diagnostics, the control system will let you know if there's an exception. Small problems like minor time shifts can be easily seen and corrected before they become big problems. By setting alarms, significant problems can get immediate attention, he said. "If a historian is down or the alarms are not talking, call the Ghostbusters, we have a real problem."
And f you want to use PI data to make decisions, you can make sure it's current by observing the heartbeat tag. "If I want to make a real-time call, I'd better be sure the data is up to date," DeJohn said. It's easy to set alarms using "out-of-the-box DeltaV stuff," DeJohn said. "What's important to you? You can set an alarm and be notified."
The plant added an OPC-based cellular paging system that sends a message to operators if a system needs attention, and calls DeJohn's group if something needs their attention. The system also allows DeJohn to call in to check status. "If I'm on a lacrosse field on a Saturday afternoon and there's a power outage in New Brunswick, I'll get a call asking ‘How's DeltaV?'" he said. "In about a minute, I can find out.
"If you think about the dollar value of what's in our tanks, it's understandable that people want to know right away if anything is going wrong."
Once the diagnostics are in Control Studio, DeJohn can access them from a button on the control screen. "I can take a quick look and if it's all green, no problem. I can scan 20 screens' worth of diagnostics in 20 seconds. I can see right away what's working and what's not working, and I can drill down into why.
"One screen shows information that would take me half a day to tabulate, and it's real-time. If I tabulated it, it would be out of date before I got done."
Alarm triggers are used to visualize a form of "uptime," DeJohn said. "How's the historian doing compared to last year?" For example, more triggers might indicate that a hardware problem is developing on a server. The facility already had the logic for this application built in Excel. "I just put in the start and end times, and Excel goes and gets the data and makes the graphics," he said.
In essence, the work instruction procedure becomes "consistent everywhere," DeJohn said. It changes from "look at the diagnostics" to actionable information—who to tell, what to tell them—and from reactive to proactive," he said. "So you can take proper action when things aren't going right."