The Holy Grail—that mythical object of desire that is the subject of so much legend and storytelling. It is the object that heroes and heroines go through multiple trials, ordeals, difficulties and dangers to achieve, and frustratingly, it always seems to be just out of reach.
Asset management (AM) and maintenance systems, whatever acronym you choose to give them, have their own "Holy Grail," which, like the cup of legend, seems close, but still just out of reach. For maintenance folks, this grail would look something like this: a system that would
- Detect a problem,
- Automatically propagate a bill of materials,
- Cost out the project,
- Assign the work order to the appropriate person,
- Schedule it,
- Get the info into the control system to let it know that certain piece of equipment is going to be down for repair, and account for that in its operation. Oh, and do all this before the problem brings down the process, but no sooner than is necessary.
We've been on the search for this Holy Grail for decades now, and the trip has been long and arduous, littered with numerous failed initiatives, half-completed projects and frustrating disappointments.
The nagging question remains: Why is this so hard?
The Fog Rolls In
Part of the difficulty with any integrated asset management project is that "Asset management is a lot of different things to a lot of different people," observes Herman Storey, formerly of Shell and now chief technology officer at Herman Storey Consulting LLC. "It can mean instrument management, configuration management, inventory management, work order management and much, much more."
Also, as is often the case on journeys, not all the people you talk to speak the same language you do. Andrew Soignier, North American director of sales for oil, gas and chemicals at enterprise software vendor Ventyx (www.ventyx.com), an ABB company, explains, "One of the biggest problems in this industry is that automation and IT both use the term ‘asset management,' but they mean different things. From the top level, when we look at widely accepted definitions of enterprise asset management (EAM) and computerized maintenance management systems (CMMSs) they constitute everything from the supply chain on down that's tied to the view of the asset. The traditional automation view looks at it from more of a site or facility level. Automation defines an instrument as an asset. At the top level, it's not."
But these difficulties aren't stopping some companies from starting on their individual asset Holy Grail quests. The operations farthest along the route to the AM Holy Grail are mapping a way forward. In fact, it's not a single map at all, but different for different companies, depending on their particular needs. None of these routes is a six-lane freeway, and that entire, integrated grail vision is still tantalizingly in the future, but the routes are getting clearer.
The DCS Route
Take, for example, the case of Colorado Springs Utilities (CSU). It took the integration-to-the-DCS route to improving its maintenance. CSU provides electricity, natural gas, water and wastewater services to nearly 600,000 people in central Colorado. To help deliver these services, it operates four hydroelectric plants, the oldest of which, in Manitou, Colo., a remote town in the foothills of Pikes Peak, is over 100 years old. In spite of their age, these hydro units are essential to CSU's operations. For one thing, hydro units can generate electricity for a third of the cost of one CSU's coal-fired plants, but to take full advantage of this low-cost power, the company needed to address key issues with them: system availability and maintainability.
CSU's biggest maintenance challenge is the remoteness of some of these units. It needed to increase system availability at Manitou and at its Ruxton hydro unit, which hovered at 50%. Making matters worse, 20% of the outages were unplanned. CSU's operators couldn't remotely access the hydro units from a central location, so it was always necessary to travel to the sites to assess the cause of each forced outage, resulting, at a minimum, in several hours of downtime—but usually several days of lost production.
Furthermore, Ruxton, CSU's remotest unit, also relied on old equipment that was failing at an alarming rate. CSU's maintenance engineers would journey up Pikes Peak using the Cog Railway an average of three times each month to address malfunctioning equipment. While scenic, the trips were far from profitable. It takes 20 minutes to reach Manitou and two and a half hours to reach Ruxton. After arriving at the facilities, engineers would diagnose the problem and secure parts to repair it, which sometimes required multiple trips up and down the mountainside, extending downtime for days.
The solution was to replace CSU's aging DCS systems and standardize on one control architecture — Rockwell
Automation's PlanPAx (www.rockwellautomation.com) process automation system. CSU included the hydro units in the upgrade. The company began this process with Ruxton, and then migrated the control system at Manitou.
As part of the PlantPAx system, CSU installed an Allen-Bradley ControlLogix programmable automation controller on each of the plant's generators. The ControlLogix controller sends diagnostics via EtherNet/IP to a microwave station at the base of the mountain back to a ControlLogix controller at CSU's more convenient Birdsall facility. FactoryTalk AssetCentre software consolidates human-machine interface (HMI) data from the hydro units, allowing CSU's operators to centrally monitor and manage system parameters, such as water pressure, and adjust controller setpoints. By evaluating diagnostics, such as I/O modules and field devices, operators can also remotely troubleshoot the system.
"When maintenance is necessary, we can analyze the situation from the Birdsall location, gather all the necessary parts and route water to one of CSU's other hydro stations during the repair to maintain power production," says Brent Richardson, plant manager, Colorado Springs Utilities.
Prior to the upgrade, it could take three or more days to recover the Manitou and Ruxton systems from a forced outage. Since then, CSU has experienced nearly 100% availability. CSU's operators remotely monitor each plant and diagnose issues as they arise, often well before the issues cause downtime.
With downtime costing $100 dollars per hour, CSU estimates that the increase in availability is saving the company $50,000 annually. Furthermore, CSU can now optimize renewable energy production by identifying the highest efficiency hydro units. Prior to the upgrade, CSU could produce 3% renewable energy—when the hydro units were functioning properly. Today, CSU can reliably produce 6%, and is well ahead of schedule for meeting the 2015 state mandate of 10%.
Figure 1. A system upgrade at Colorado Springs Utilities' aging Manitou hydro-electric plant increased uptime and system availablity, and made it easier to service the remote operation.
The Focused Route
For some companies, the real need is for a system that simplifies and automates a critical, but time-consuming part of the overall maintenance task. In the case of pharmaceutical giant AstraZeneca's (www.astrazeneca.com) Sweden Operations group, calibration management was the challenge.
Calibration is critically important in the pharmaceutical industry. The FDA's 21 CFR, Part 11 regulates how the calibration certificate for each instrument is documented and signed electronically. All the equipment and instruments in the production process must be calibrated regularly. This involves testing sensors to make sure they give the correct result in various conditions, such as temperature, atmospheric pressure, air humidity, etc. The job is exacting and time-consuming, and when done manually, is not only tedious, but prone to error.
Eight years ago, the Sweden Operations group embarked on a project with calibrator supplier Beamex (www.beamex.com) to automate the company's entire calibration process. They implemented a completely new calibration system using Beamex's CMX Calibration Management software with the Pocket PC option and documenting Beamex MC5 multifunction calibrators.
Tomas Wahlgren, business relationship manager for maintenence and engineering at AstraZeneca, explains, "Previously, the calibration was primarily done with manual and paper-based process. The certificate was on paper, which was signed and stored in binders. After implementing the new process, the entire process takes place digitally, from measurement to signing and archiving. We perform about 22,000 calibrations annually, which previously engaged about 50 employees. Today, the same work can be accomplished with only 15 people."
The roll-out was by no means plug-and-play. To justify the $600,000 investment, the system needed to reduce time and costs for calibration itself and for the line-down period in production. With 22,000 calibration certificates to produce, the ability to justify a reduction in calibration frequency was essential. Astra Zeneca also wanted quick and easy access to data from any desktop computer in the company. Essentially, what the implementation team needed to do was unify the entire company—used to working with 12 different databases and several different workflows—around a single system.
While the payoff was quick—one year—the full benefits of the system did not kick in until later. When the toughest work around unifying the entire operation around one calibration system was done, around 2007, the team settled in to gather and analyze data. "Thanks to this data cleansing process, we have become more and more aware of how to optimize the amount of data, which data to keep and which to dispose of," says Wahlgren. It took until 2010 to stabilize the data and only then, based on the data, could AstraZeneca start reducing the calibration frequency on certain devices and instruments.
The Devil Is in the Data
As illustrated by the AstraZeneca project, the asset management Holy Grail is all about the data. "The real success is going to be dependent on how good your database is. The problem with instrument asset management systems is how well or poorly they do is linked with the plant asset management system," says Ian Verhappen, director, Industrial Automation Networks Inc. (www.industrialautomationnetworks.com). "MIMOSA, B2MML, that's why they're there—to get data from instruments to the AM system."
And it's more than just linking one system to another. "How do you make sure you have the system set up to make sure you're getting an accurate picture?" asks Mark Leroux, marketing manager for collaborative production management at ABB (www.abb.com). "You have to look at this from a variety of perspectives. You have to match the system's nomenclature. That's a big issue. When you configure the bills of materials and the costing, that has to match what's in the real world. There is software out there that allows you to do that, but it's still a lot of work."
Configuration alone is a challenge—one that is often overlooked. Storey observes, "We've got smart devices. We've got smart systems. We've got condition-based monitoring, which tells us continuously what's going on. It compares an asset's measured condition with its real condition and can raise a flag if necessary. And we have stranded diagnostics. You can buy an asset management system to go with the DCS to receive these diagnostics. But we really haven't configured the instruments properly."
Most companies don't do this configuration because it's so complex, he says. "For a majority of companies, [the worker] takes the device, takes the handheld and puts in configuration parameters. It works, and life is good. Some configuration parameters allow you send information. But we're also talking about tags, etc. If it only sends out 4-20 mA, everyone assumes it's correct. But it isn't really. You find a lot of mismanagement of configuration. Data that goes into configuration may not be managed well. Things get out of synch."
But this is only one aspect of asset management, he warns. "We're also looking at the loop level. Is it controlling? Is it in manual? Is it responding properly? Then there's maintaining alerts and alarms. There's a lot of soft configuration that's all part of asset management," he continues. "You have to invest time and energy into finding out how to match up the instrument information with information about how critical the device is. You have to set individual priorities, based on the application of the device as well as its condition, and most projects don't want to deal with this. [The data] comes to the asset management system this way and is kind of useless because people haven't done the work. Then they don't have the work processes to deal with it."
Abandon Hope, All Ye Who Enter Here?
So is there any hope? Why even bother if the job is so tough? After, all, we've kept our facilities running all this time the old way.
As with the Holy Grail of legend, the benefits of achieving it are just too great to let it go. End users and vendors are still out there trying multiple paths.
A few hardy souls are embarking on a new project to map out a way through the configuration part of the Holy Grail maze. Conversations have been ongoing for some time, and formalized action to develop some standards around configuration management began with a meeting last month at the ARC Advisory Group (www.arcweb.com) Forum in Orlando, Fla., according to Storey. He continues, "This stuff is tough to do alone. It needs vendor and user-community help to push the technology along faster."
The Fieldbus Foundation and ISA were part of this meeting in February. Shell has offered up some materials for managing instrument configuration and work processes that the company developed in-house over the course of several years, adds Storey. FDT and the HART Foundation have also expressed an interest in the efforts.
"We're trying to promote some standards on just pieces [of the asset management system]. "We're focusing on instrumentation and systems stuff, systems nodes and networks," adds Storey.
Meanwhile, individual vendors are tackling the quest in their own ways. For example, Endress+Hauser (www.us.endress.com) just released an instrumentation support program called the Installed Base Audit. The program begins with a plant audit by an E+H instrumentation expert that puts all the device and application data into E+H's W@M Life Cycle Management database. Then, working with plant personnel, he or she helps assign a criticality level to each instrument. Based on the data, the plant gets recommendations for improving maintenance activities, as well as the populated database. The system is open in that it includes, not just Endress + Hauser instruments, but those from other vendors as well.
Meanwhile in late February, Beamex released a new multi-function calibrator, MC6, that combines simple metering, calibration, a documenting calibrator, a communications module and a data logger. The calibration module can be used with all types of fieldbus instruments, not just pressure and temperature transmitters. It can be used as a HART, Foundation fieldbus or Profibus PA communicator.
The big process automation system vendors, including Emerson Process Management, (www.emersonprocess.com), ABB and Invensys Operations Management (http://iom.invensys.com) and Rockwell Automation, all have asset management modules that can easily integrate with their control systems, and all offer service programs for helping you manage your maintenance systems.
Maybe the secret of finding the asset management Holy Grail is that there is no secret. You get there the same way you get to any other goal—a step at a time. And, you treat the journey the same way you would treat any other major technology upgrade project—or any road trip. You have to assess where you are, decide where you want to be in the end, and then figure out a route that makes the most sense for you.
Kim Custeau, director of product marketing for the Invensys Avantis asset management suite, explains the map. "Start with the basics. What assets do you have? What parts do you have? What processes do you have? Go step by step and learn about what these parts and processes are doing. Look at the system holistically. Then move to monitoring technologies."
From there, look at the asset and see how performance can degrade. Then what happens? Somebody has to go and do something, so you overlay the monitoring system with a workflow. Create a process by which, when performance starts to degrade, the maintenance technician gets a message to go check things out."
Then you keep refining your database, setting benchmarks, and improving the system as you go.