CT0504_CVR_0
CT0504_CVR_0
CT0504_CVR_0
CT0504_CVR_0
CT0504_CVR_0

How to run a plant when expertise is lost

March 31, 2005
When plant institutional knowledge leaves, process control soon follows. How do you keep your plant running if nobody remembers how? Senior Technical Editor Rich Merritt reports.
 By Rich Merritt, Senior Technical Editor

L

ayoffs, cutbacks, outsourcing, retirements, forced retirements, mergers and consolidations are stripping plants of instrument engineers, control engineers, technicians, maintenance people and operators. Once gone, their accumulated knowledge of the plant&rsquos processes goes with them, leaving companies without in-house expertise. Companies are turning to vendors for help.

If current trends continue, most of you will no longer be working as a control engineer for an end user company 10 years from now. More likely you will be working for a systems integrator, a process control vendor, or a service company of some kind. Soon, there may be nobody left in the plants, your accountant-bosses will have subcontracted all the work out, and process control will have become another purchased utility, like electricity and water.

That&rsquos what happened to the engineering and technical staff who worked at the Danube Refinery, Százhalombatta, Hungary. The refinery is the biggest production facility of the MOL Group, a huge oil conglomerate in Europe. Eight years ago, 130 engineers and technicians worked in the refinery. Today, after downsizing, outsourcing and restructuring, all of them are gone.

&ldquoAs the former instrumentation department disappeared, most of its services were taken over by vendors like Honeywell,&rdquo says an MOL manager, who asked to remain anonymous. Another 45 employees were eliminated when the refinery contracted out operation of its wastewater and waste incineration facilities to EarthTech in the UK.

The refinery also gave Emerson $10 million to design and install a new control system as the main automation contractor (MAC) for a control project. As the MAC, Emerson will perform front-end engineering design, applications engineering and project management, install and commission the automation solution, and supply operator training.

That doesn&rsquot leave much work for the refinery&rsquos control people. &ldquoThe company tries to hire back or subcontract former MOL employees when it needs specific knowledge,&rdquo says the anonymous manager.

This is happening at smaller companies, too. &ldquoOur chief instrument engineer left for greener pastures, and was never replaced,&rdquo laments a manager of a small, specialty organic chemical manufacturer in the Midwest. &ldquoThen another instrument specialist left. Now we are relying on an engineering service to support our PLC programming, HMI programming and instrument specifications, and relying on process engineers to tune controllers. The few remaining instrument technicians we have do not have the time to tune controllers since there are too many other fires to put out.&rdquo

The power industry is an excellent example of what happens when knowledge leaves. As we point out in this month&rsquos Control Report (p31), no coal-fired power plants have been built in 25 years, all the people who knew how to control coal-related process have retired, and there is no one left–except a few vendors–who know how to control a coal-fired power plant. Many in the process industries are going down that road, too, as knowledge leaves the plants.

One reason for loss of institutional knowledge is the aging and shrinking workforce in our business. A CONTROL reader poll shows that 71% of you are 41–50 years old, and 12% of you are over 50. Only 3% of you are under 25 and ready to step into their shoes and learn the business. Few young engineers choose to enter our profession, maybe because their fathers warn them away. Or maybe young engineers avoid control because of the disturbing trend of process companies to contract out major parts of a control engineer&rsquos work and shut down entire instrumentation departments, as MOL did.

A Boon to Suppliers
There&rsquos no question that the loss of knowledge in plants is benefiting suppliers, such as process control vendors, systems integrators and service companies. &ldquoWe are seeing a very large upswing in engineering services,&rdquo says Bill Robertson, director of global services and support for process systems and solutions division of Emerson Process Management. &ldquoCustomers are increasingly outsourcing engineering services on new projects, migrations and, to some degree, ongoing system maintenance and operations.&rdquo

Honeywell says it takes deep pockets for a process company to keep up these days. &ldquoUnless a plant has the luxury of being able to overstaff and over-train, they will have to address the issue through the strategic use of suppliers who combine expert knowledge with technology that allows that knowledge to be disseminated out over a broad customer set,&rdquo says Andy Drexler, Global Marketing Leader, Services, Honeywell Process Solutions. Drexler, of course, means companies like Honeywell.

&ldquoTo prevent the impact of lost expertise, plants must carefully select partners whose expertise they can use on an ‘as needed&rsquo basis and who offer the technology to help less experienced workers quickly reach a level of proficiency that once required years of experience,&rdquo he adds.

All the big process control companies are aware of the outsourcing trend, and are gearing up to provide services ranging from design to installation to operations to maintenance. John Berra, president of Emerson Process Management, agrees that Emerson is capable of doing everything in a plant, including running it. &ldquoWe have no plans to offer such a service,&rdquo says Berra, &ldquoand I doubt that any company would want us to actually run their plant.&rdquo

Nevertheless, the vendors are developing new hardware and software that gives them the capability to take over more and more responsibilities, such as automated monitoring, documentation, tuning, diagnostics and so on.

&ldquoAdvanced diagnostic capabilities, such as our Loop Scout /Alarm Scout Service, enable the Experion control system to quickly identify and isolate problem areas in the large sea of control loop and alarm performance data,&rdquo says Drexler. &ldquoThese services automate and optimize plant work process that would have required significant time and effort from expert personnel in the past.&rdquo

           

VIBRATION ANALYZER IN A CAN

           

Analysts who understand vibration data are leaving plants for greener pastures, and are not being replaced. Emerson&rsquos CSI 9210 machinery health analyzer diagnoses problems with rotating machinery and tells operators and maintenance exactly what is wrong and how long before the motor-pump fails completely. Source: Emerson


Emerson just announced its CSI 9210 machinery health transmitter (right),which automates rotating machine diagnostics. The system predicts when a motor-pump set needs maintenance. &ldquoAnalysts who know how to interpret vibration data are leaving the plants,&rdquo says Brian Humes, VP of Emerson. He says they are retiring or forming their own engineering companies, and the plants are not hiring anyone to take over the jobs. Their knowledge is not being passed on. &ldquoThe CSI 9210 performs the necessary analysis and identifies the problem, so maintenance can take care of it without an analyst.&rdquoUnintended ConsequencesNo one can blame vendors for taking advantage of the opportunity. Plants are losing or divesting their control systems professionals for various reasons, some good, some bad. Sometimes the motivation is a clear attempt to shake up and modernize plant and organizational systems that are stuck in the mud. Sometimes the motivation is clearly to appease Wall Street analysts. Sometimes, the motivation is just plainly to provide more income for the top management of the company. Companies do what their management believes they have to do. And so we meet the law of unintended consequences.In many cases, plants that divest themselves of engineers have nowhere else to turn for help but to their vendors. The vendors have the necessary expertise, experience, process knowledge and a staff of engineers. However, rumors persist that the vendors are stretched very thin these days, and are not keeping up.Glenn Givens, a control systems specialist at Innovention Industries in Burlington, Ontario, Canada, certainly isn&rsquot impressed by vendor expertise. &ldquoIf you ask one of their technical people a question, they usually know less than I do and their answer is ‘I don't know, why don't you try it?&rsquo," says Givens. &ldquoWhen I find an important bug and I have hard data, they first say they're interested, and then nothing is done. I don't bother telling them any more. I see their controls expertise as ‘entry level&rsquo or perhaps slightly more advanced than the average instrument technician.&rdquoThe manager of the small specialty chemical company says help eventually comes. &ldquoWhen we scream for help, so far we have been able to get help from our engineering service,&rdquo he says. &ldquoBut it comes at a high price, and sometimes with a delay.&rdquoTo keep from getting overwhelmed by the increasing requirements of plants, the vendors are coming up with ways to simplify their lives.&ldquoWe&rsquove systematically engineered complexity out of our systems, at times eliminating tasks that used to take up valuable engineering time, like configuring highway addresses,&rdquo says Robertson. &ldquoAdvanced control is now easy enough for the basic automation engineer and some configuration tasks can be accomplished by operators instead of engineers.&rdquoBerra says they&rsquove seen this trend at Emerson. &ldquoAs our diagnostic tools do more of the low-level work, we&rsquove been moving up our engineers into more responsible jobs,&rdquo he says. &ldquoOur support engineers spend more time on bigger customer issues, and less time diagnosing equipment problems.&rdquoPerhaps such a realignment of work is the future for control engineers. Instead of working with the actual equipment, tuning loops and diagnosing problems, you will let software do that work while you operate at a higher level–such as approving invoices from vendors.Keep the KnowledgeWhen experienced engineers leave the plant, they take valuable knowledge with them, such as the reasons for certain types of loop-tuning parameters, why PLC logic was done a certain way, or why batch sequences are done in a particular order. Therefore, you should make every effort to capture that knowledge before it is sent away. Good documentation right from the git-go helps, of course, but if that&rsquos lacking, you need to create it. To justify the expense of documenting your existing system, you can explain to your senior management that such documentation will reduce the cost of future upgrades.&ldquoOften the lack of knowledge of existing systems becomes obvious when customers attempt to migrate to newer generation equipment,&rdquo says Robertson. &ldquoThe first and sometimes painful step along this path is to fully document what the customer has. We provide a wide array of services and technology to make the process of untangling the spaghetti code that we occasionally find for the customer so they can migrate their valuable configurations.&rdquoGivens has known this for years. &ldquoWe configure and build controls, write software, diagnose process problems, tune controls and, occasionally, train technicians,&rdquo he says. &ldquoWe have detailed records of every single bump test done to every single loop since the day we went into business. I can find the data files for every one and tell you how I calculated the tuning parameters, how I determined how much backlash/stiction was present, and so on.&rdquoA great deal of new software is available to help you document your system. Wonderware&rsquos Industrial Application Server, for example, allows control and system engineers to standardize system implementation and encapsulate knowledge on ArchestrA-based systems, says Steve Lewarne, Wonderware's vice president of product marketing.  &ldquoArchestrA provides a structured model and approach to designing supervisory automation and information systems,&rdquo he explains. &ldquoArchestrA is also self-documenting with respect to initial configuration and modifications made throughout the life cycle of the system. These allow system engineers to easily design, build, deploy, and maintain secure and sustainable automation and information applications.&rdquoYokogawa has something similar. &ldquoYokogawa has developed a software solution called Exapilot for capturing this fleeting best-operations knowledge,&rdquo says Fred Woolfrey, Productivity Solutions Consultant at Yokogawa. &ldquoExapilot is a Microsoft Windows-based operation efficiency improvement software package that allows operating personnel to standardize and automate what would normally be manual procedures and incorporate the know-how and skill of expert operators. Operating personnel can build and modify these procedures with an icon based interface. Users can easily configure and test complex procedures. Procedures built with Exapilot can be used to standardize and automate manual procedures, improve plant efficiency, and improve the safety of plant operations.&rdquoBecause vendors must document what they do when developing and configuring control systems so they can support them later, they&rsquove developed such tools for their own use. Check with your control system vendor to see what&rsquos available.Train and Keep Maintenance
An I/E Supervisor at a power company in Texas trains his technicians and spreads the knowledge around, so he&rsquos not crippled if a key person leaves. He&rsquos also less dependent upon vendors. &ldquoWhen we purchase an upgrade or a new system, we add in cost for training in the price of the equipment,&rdquo he explains. &ldquoIf it is not cost effective to send all the techs, then the ones that do attend the training are required to train the rest of the group. I also rotate areas of responsibilities with the techs to keep them current in the understanding, Troubleshooting and repair of the equipment. This helps these individuals become more of an asset to the company and it makes my life a whole lot easier. You can never train enough.&rdquoHe loves the new open control systems, because it gets him away from proprietary hardware. &ldquoControl system manufacturers used to build and supply every component, but now they buy from third parties to build their systems,&rdquo he says. &ldquoBelieve it or not, this is really a cost savings to the customer. For example, if I lose a hard drive on my engineers console, I run down to Radio Shack and buy one at a quarter of the cost than when you had to buy from a single source.&rdquoIt&rsquos not just economics. He says the vendors have lost some of their expertise in repairing their own equipment. All the vendors responding to my question of how much they support their older equipment said they go back to the dawn of time, but that&rsquos what you would expect them to say. They did not say they had experts doing the work. Therefore, you can probably build a case against letting a vendor take over maintenance on your system on the basis of cost and performance. Cruising the automation lists on the Web would probably get you enough anecdotal evidence to even make a flint-nosed bean counter think twice about outsourcing maintenance. Keeping equipment knowledge in-house helps keep jobs for you and your techs, but you also have to be cost effective.Analyze Those Loops Yourself
The process control vendors and service companies have built powerful analysis and diagnostic capabilities into their new software systems. They have the ability to monitor every process variable, put all the data into a process historian, and analyze it on a loop-by-loop basis for efficiency from remote service centers or locally, in your plant.You might want to take advantage of the same tools to preserve your job. PlantTriage from ExperTune and similar software from other vendors can perform the same service that the process control companies offer. In fact, the vendors might be using the same software, because it works.&ldquoA performance monitoring package like PlantTriage prioritizes the loops that need attention and automatically diagnoses the problems,&rdquo says John Gerry, president of ExperTune. &ldquoThe only way to survive and be competitive is to use software tools like this.&rdquo It also helps you preserve process knowledge, because it records all PID tuning changes and allows you to enter notes as to why you tuned it that way, he adds.Run Your Own Server FarmWith server-based systems soon to be announced, the big control vendors will be able to run your plant from centralized server &ldquofarms,&rdquo where specialists of all kinds will be available to diagnose problems, retune loops, decide when maintenance needs to be done, and even operate the plant from afar. In other words, the entire control system and all its engineering and tech support can be outsourced.There is no reason why a forward-thinking control engineer such as yourself could not set up a similar server farm in your company; that is, a centralized system that monitors and controls all your company&rsquos process operations around the world. In other words, instead of having instrumentation departments in each plant, set up a central Instrument Engineering department at company headquarters, and run all your plants from there.Such a system documents and preserves all the process knowledge in one place. A centralized system also preserves the jobs of your company&rsquos expert control engineers, techs, and operations people, and keeps all their knowledge in house. The wave of the future is to consolidate all the control, maintenance and diagnostics work in a plant anyway, so just ride the wave, set up your own central operation, and keep your knowledge in house. Otherwise, you might do better to shop your resume to whoever will have to pick up the ball as the knowledge disappears from your plant.Adversity always breeds opportunity. When your business accounting management starts looking for ways to replace engineers and techs with outside services, show them that you can do the same job at a lower cost by using the same knowledge management tools that the outside suppliers will use if you don&rsquot.