zzzz646
zzzz646
zzzz646
zzzz646
zzzz646

Unlock Plant Secrets

Aug. 28, 2003
Manage engineering and operations knowledge to handle abnormal situations

In the process control world, most things run smoothly. Control systems and field devices are more reliable than ever, intelligent equipment communicates up and down the line, automatic diagnostics take care of many problems, and human operators are reduced to a life that often consists of 99% boredom and 1% panic.

Equipment is so reliable, we sometimes forget how to use it, start it up, shut it down, or make adjustments. Problems occur so rarely, we forget how to fix them. All the information needed is in the plant somewhere, buried in a manual, a set of handwritten instructions, or an operator's head. Knowledge management (KM) is one way to organize all this data.

KM can gather all the printed documentation, handwritten notes, operator knowledge, real-time data, diagnostic tests, tuning instructions, "perfect batch" recipes, and other plant secrets. KM can then grind it all up, put it away into a database, and produce the needed information on demand.

If it works, that is.

Unfortunately, KM has proven to be a huge bomb over in the information technology (IT) world, as it failed to live up to such promises. But that's not the first time KM bombed.

The first time KM failed was millions of years ago, on the remote planet Altair IV. As science fiction fans know, the Krell stored the knowledge of their entire race in a gigantic computer for the benefit of Krellkind. Alas, as told in the movie Forbidden Planet (MGM, 1956), the KM system destroyed the Krell race and threatened to kill off actors Leslie Nielsen, Walter Pidgeon, and Anne Francis, plus Robbie the Robot, a few million years later.

A little knowledge can be a dangerous thing.

KM reached its zenith in 2000, when the Internet was booming. KM was all the rage, software vendors promised wondrous things, and company execs told their information technology (IT) people to go out and buy one.

Alas, KM's time seems to have come and gone, almost in the blink of an eye. "It looks like KM is fading fast," says Mark Clark, my local IT and ERP consultant here in Cedar Rapids. "A lot of the links I find are dead and key publications seem to have ceased publication. Several deep thinkers are still actively engaged but the fire seems to have subsided."

Such a shame, because the technique held such promise. "If done right, KM is supposed to create a more collaborative environment, cut down on duplication of effort, and encourage knowledge sharing"saving time and money in the process," says Eric Berkman, a writer for Darwin magazine. "The problem is, in many cases KM devolved into a purely technical process, resulting in expensive software implementations sitting unused by oblivious, fearful, or resentful employees." Sounds like an ERP or CRM system, doesn't it?

Now, KM has reached the process control and automation world. Process control vendors hawk the wonders of KM, just like the IT folks did back in 2000. Will KM bomb over here, too?

If you do a little analysis of your situation beforehand, you can find tools that will let you manage the process knowledge available to you. We found several examples of plants that are using various versions of KM, including quite a few home-grown systems that seem to work very well.

What's KM?

Nobody knows the definition of knowledge management. "There is no useful academic definition for KM," says Clark. "Instead, it has become a marketing buzzword. Companies in the software industry have discovered KM is a hot topic, so suddenly any piece of software that deals with data qualifies to be called KM. This includes databases, process historians, document management, and software that does collaboration, data mining, content management, or data warehousing."

In some cases, KM is just a revisit with old artificial intelligence (AI), expert systems, and rule-based logic that failed in the late 1980s and mid 1990s process control marketplace. On the other hand, today's modern information gathering systems ,“ including process historians, SCADA and HMI systems, XML-based communications, Ethernet-based networks, fieldbuses, smart instrumentation, and all the other advanced hardware and software available to us ,“fulfill the promises of the 1980s, because they make it possible for the AI, expert systems, predictive maintenance, condition-based monitoring, and all the other advanced asset management software to get all the data they need.

Some practical definitions might be when such data is used by expert system software to help tune a system, by RCM software to diagnose an equipment malfunction, or by an engineer or operator to gain insight or direction into how to deal with a alarm situation. Duncan Schleiss, director of marketing at Emerson Process Management (www.emersonprocess.com), puts it simply: "Knowledge management is getting the right information to the right person (or system) at precisely the right time."

For example, suppose you have to start up a heat exchanger that's been on line for the past five years. It's been running continuously, the company that built it is out of business, the operator who used to run the system retired to Florida, and you had to shut down the exchanger for maintenance. How do you start it back up? If you can open up a notebook and take out a set of written instructions, then that's knowledge management. The data exists, and you know where to find it.

At the other extreme, you go to your PC, link into your KM software, and enter: "Heat exchanger XYZ startup procedure." The KM software searches the Internet, the original vendor's site, procedures stored in memory, captured keyboard strokes from five years ago, a transcribed conversation with the old operator, and it eventually finds the procedure buried somewhere under a pile of bytes. That's knowledge management, too.

Dave Novak, controls engineer at BASF, Monaca, Pa., wishes he had any kind of KM system. "We lost one of our computer hard drives that was installed about three years ago. We had all of the data backed up, but we couldn't get the software to install again. We had operating system conflicts, revision conflicts, and we couldn't find the person who installed it. I am in the process now of putting together documents on how to recover all of our special computers. It's taking two to three weeks to re-invent the process per machine. In the meantime I lost another computer that's five years old, and I am trying to find the design engineer to help."

When he gets through putting those recovery documents together, Novak will have performed a little knowledge management. The next time a computer fails, he'll be prepared.

KM for Peanuts

Some of the reasons KM bombed in the IT world included unrealistic expectations and overselling the concept. Berkman says KM gained a dubious reputation among many business executives, industry analysts, and even KM industry insiders. "At worst, KM has been a total bust; at best, it hasn't lived up to the considerable hoopla surrounding it," he says. "Like many a business concept, KM has evolved from a hot buzzword to a phrase that now evokes more skepticism than enthusiasm."

In other words, company executives thought KM was the answer to all their problems, so they spent millions of dollars willy-nilly on software, and got no return. Problem was, they didn't think through their problem.

Figure 1: Tricks of the Trade

Advanced Automation Associates, a systems integrator, built an Access database to share the accumulated knowledge of its 65 engineers. (Source: AAA)

Some of the best KM systems don't even use software. At least, very little. "We instrument control technicians communicate in the shop among ourselves about discoveries and shortcuts to troubleshoot specific equipment and systems," says Timothy Brennan, instrument technician at Rohm & Haas, Charlotte, N.C. "If an idiosyncrasy of programming or a special code is needed, then I mark this information inside a cabinet or enclosure so it is on hand but not public. I laminate prints and drawings needed in the field, complete the planned maintenance system (PMC2000) work order, and input what was wrong and what was done to correct it into the computer program. This way, it is possible to call up all previous work on that specific equipment and see what was done before. This retrieval of information is very cumbersome but it does work."

Jay Munro, director of the information group at Advanced Automation Associates, a systems integrator in Exton, Pa., goes a step farther. His company built a Visual Basic front end to a Microsoft Access database for knowledge management. "Several years ago, we relied on information being passed between engineers by word of mouth," says Munro, who manages 65 engineers. "As we grew, finding the right person to ask became a more difficult task. If that connection was not made, an engineer would build from scratch a solution that already existed in house.

"We realized managing the combined knowledge of 20 years of system integration was going to be a key factor in our success as a company. We looked to gather these technical tips' in one central location," he continues. "The database points engineers to similar solutions within our library of applications. Engineers can find which job used an Intellution SCADA with a Modicon PLC. The database also highlights valuable tricks of the trade' like getting a Pocket PC to talk to VB over wireless Ethernet, or getting a flowmeter to talk to an AB ControlLogix PLC over Modbus" (Figure 1).

Equally straightforward is the Excel spreadsheet system used by Karl Baldwin, data processing manager at USA Tacom-Ardec, a supplier of military commodities in Picatinny, N.J. Baldwin also uses an SAP system and some specialized software for PDAs. "Our KM system is home grown," he says. "In our office, process knowledge is all the data on an item over its cradle-to-grave lifecycle. It covers data card information, waivers, deviations, inspections, renovation data, serviceability test results, type classification actions, and final disposition information."

USA Tacom-Ardec keeps 30-40 years of data on several hundred commodities and uses the data for trend analysis, malfunction reporting and investigation, defect reporting and analysis, demilitarization planning, and logistical analysis.

These examples illustrate how to make KM a success: Think through your problem, decide what you need to know, and then find an appropriate software tool, such as Excel, Access, a maintenance management system, Lotus Notes, or other simple software to make it work.

Real KM software (not just data collection tools) is becoming available. GE Plastics, BP Amoco, Eastman Chemical, and GlaxoSmithKline are using a KM system from TheBrain Technologies Corp. (www.thebrain.com). You can try a single-user version for free. See the sidebar, "Build Your Own KM System."

VI Engineering (www.vieng.com) has a Technical Data Management (TDM) product that appears to do KM. TDM uses hardware and software from National Instruments to collect data, store it in databases and file servers, and make it available for data management, analysis, and reporting.

As noted above, KM has become a buzzword, and it's being applied to a lot of different software. So it's sometimes difficult to tell what software is actually KM and what's not.

Accumulating Knowledge

The total amount of process knowledge available in a typical plant staggers the mind. It is available from dozens if not thousands of sources. Control engineer Michael Paulonis says Eastman Chemical Co., Kingsport, Tenn., uses these categories:

  • Operating and maintenance procedures
  • Best practices
  • Historical process and laboratory data
  • Operating targets
  • Alarm limits, tuning constants, and controller performance metrics
  • Empirical and rigorous process models
  • Logs, notes, comments, lessons learned, etc.

Matt Bothe, a controls engineer at CRB Consulting Engineers, Kansas City, Mo., says it also includes all of the experiences accumulated over time by an operator through training, repeated tasks, conflict resolution, process upset recovery, troubleshooting, and emotion. "The more practice and the longer the association with a particular process unit, the more experienced an operator becomes," he says. "Tuning procedures, trending, alarm entries, and screen schematics are all tools of the trade for an operator."

Where can you keep all this data? Eastman uses commercial databases and creates its own schema and interfaces needed to hold the data and get it in and out. "Procedures can be stored in word processing documents within a content management system, or can be steps recorded in a database, such as Oracle or SQL Server," says Paulonis. "Best practices can be captured in an expert system such as Intellix, and process data can be archived in a historian, such as OSISoft's PI. In many cases, history of a golden batch' would be noted in the historian. Operating targets, alarm limits, logs, and notes are likely to be stored in a database. Process models are almost always stored as files in the specialized modeling software used to create the models, such as Umetrics SIMCA or Aspen Plus."

Atlantic Copper in Hueva, Spain, integrated its PI data historian and its Thermo Electron SampleManager LIMS (laboratory information management system), so that analytical information became available in real time to the control systems and human managers that run the manufacturing process. It also simplified testing procedures, data reports, and meeting regulatory requirements.

Bayer's new aspirin-making plant in Pilar, Argentina, went even further: It integrated a process historian, ERP, MES, and control system into a plant-wide knowledge system (see sidebar, "Tracking Pills").

Storing data is easy, but Bothe says acquiring and using human operator knowledge probably will require an expert system or AI.

Veterans of AI remember the classic tale of "Aldo in a Can," one of the first and most successful expert systems. Back in the early 1980s, Campbell Soup relied on one engineer, Aldo Cimino, to deal with its toughest problems. They flew Cimino all over the country to solve problems with cookers, hydrostatic sterilizers, and similar equipment. After 45 years, Cimino was retiring, so Campbell wanted to preserve all his knowledge.

Campbell worked with Texas Instruments to develop an AI system that could diagnose cooker problems based on Aldo's experience. They developed 150 heuristic rules, tested it in the field, and installed the system in all of Campbell's canneries. It took two years to develop and implement this expert system, but it seems to work well.

Aging operators and equipment are becoming a big problem, says Emerson's Schleiss. He says the average age of many plant operators, especially in Europe, is 45 to 50, so retirement is looming. "Also, the time for replacing equipment, such as workstations, is increasing from four to six years," he says. "Chances are, the people who installed the original equipment won't be around to help install new systems." Capturing the knowledge of such plant experts can become vital.

Not much is heard about AI and expert systems these days, but the software is still around. "AI requires enormous quantities of memory and disk I/O," advises Bothe. "A redundant data server is required, along with entry and monitoring devices. A neural network approach might work, or a classic procedure-based expert system may be applied."

Diagnosing the Situation

Part of the operator's job involves panic. That's when the process gets upset, abnormal conditions occur, alarms go off, equipment is failing, or strange conditions create calamity.

A large part of a panic mode involves figuring out what is going on, and then taking appropriate action. Here's where diagnostics and KM can help. In the midst of control room insanity, it would be nice if a display popped up and told the operator the cause of the problem and offered some suggestions on how to deal with it.

Figure 2: Process Unplugged

Blockage in a process transmitter (above) can be detected by analyzing the variability in the noise signal. (Source: Emerson Process Management)

Unfortunately, the diagnostics built into some control systems are abysmal. "Failure diagnostics usually require expert operators or maintenance personnel to use," says Doug Taylor, control engineer at Concept Systems, a systems integrator in Albany, Ore. "Usually, this becomes obvious after the machine is placed into production and the hardware begins to fail due to normal wear and tear."

Taylor says it is far better to take diagnostics seriously from the beginning. "We design HMI layouts that lend themselves to diagnosing problems. During the final acceptance test, the diagnostics will be tested, forcing a level of training that otherwise may be missed. The end result is a system that is easier to understand and makes personnel much more knowledgeable about the machine's inner workings."

He proposes six levels of diagnostics, each requiring more knowledge:

  1. Alarm annunciation on a large display board or HMI terminals. Operators can see a problem exists.
  2. Graphical HMI application that attempts to categorize the problem. The HMI screen suggests possible causes of the problem.
  3. HMI screens with manual methods that can be used to test the failed system after the errant equipment has been identified. Includes step-by-step instructions for running tests.
  4. Automatic diagnostics, performed by the control system on the failed equipment.
  5. Self-diagnostics running continuously on field equipment and control systems.
  6. Voting system, similar to the space shuttle control system, where independent systems diagnose each other.

"Unfortunately, most systems stop at the first level of diagnostics," laments Taylor. "When customers take diagnostics seriously, equipment manufacturers will also take it seriously."

Just sounding alarms is not the answer. One large chemical plant, according to Yokogawa, was generating 50,000 alarms per day, more than the operators could handle. Audible alarming had to be disabled in the plant. Fortunately, the plant kept all its alarm logs in a database, so the alarms could be analyzed. Using Yokogawa's Exaplog event analysis software, the plant identified and eliminated alarms from ordinary operator messages, such as the progress of a batch; alarms from instruments that alarmed because they operated at the edge of a measurement range, and needed to be re-ranged; and alarms that were unnecessary. The plant reduced the number of alarms by 98%.

Phil O'Hara, senior instrument electrical design coordinator for Petro-Canada's refinery in Edmonton, Alberta, reports similar success. "Over the years, the number of alarms in the DCS system grew exponentially," he reported in a paper delivered at a Honeywell user symposium this year. "Alarms were being used as a general alert system, and operators were missing or ignoring important alarms." In one month, more than 15,000 alarms went off.

By using expertise developed by Honeywell's Abnormal Situation Management Consortium, buying the Honeywell AMO suite of software tools, and taking training in alarm management, O'Hara and Petro-Canada developed a consistent alarm standard for all areas in the refinery. "The number of alarms has dropped dramatically and operators can do a better job of identifying and responding to potential problems," says O'Hara.

Automatic diagnostics in Johanson's alarm levels 4 and 5 are certainly within easy reach, as we pointed out in "Center on Reliability," [CONTROL"June '03, p57]. Much equipment today comes with built-in self diagnostics, while HART and fieldbus networks make it possible to collect diagnostic information from field instruments in real time.

"Digital transmitters can detect their sense lines plugging by examining the variability of noise on a real-time basis [Figure 2]," explains Ram Ramachandran, director of marketing for Emerson Process Systems. "This real-time process information is communicated directly to a DeltaV system, and then to a cell phone, pager, or other alerting device."

Loop analysis software, such as ExperTune's Plant Triage, continuously monitors the health of a plant, performs performance assessments, and identifies trouble spots. "We use database technology, oscillation detection, and rules-based software to detect, diagnose, and present process control issues to the user in an easy to understand format," says John Gerry, president of ExperTune. The software takes all the data it needs from the assorted databases and process historians in a typical plant.

Finally, plant knowledge can be gathered, ground up, and digested by various software products to produce operational equipment efficiency (OEE) numbers, which determine how well the plant is running and where there are opportunities for improvement. "OEE allows users to identify and eliminate the causes of process inefficiencies, bottlenecks, or machine downtime," says Scott Miller, business manager at Rockwell Software (www.software.rockwell.com). "Rockwell's RSBizWare PlantMetrics, for example, gathers information on the three components of OEE ,“ machine availability, throughput, and quality ,“ to determine the level of manufacturing efficiency."

ABB agrees. "OEE is a key performance indicator of how assets, production lines, or processes are performing," says Bert Mijten, product manager at ABB (www.abb.com). "The overall performance of a piece of equipment or the entire plant will be governed by the cumulative effect of these three categories."

ABB installed its OtimizeIT software to analyze OEE in a polyethylene plant in Brazil to diagnose speed losses in a filling process. "The firm was able to capture on-line data and analyze the reason for the speed problem," says Mijten. "Payback materialized in the first few hours of production following the incorporation of a number of improvements."

The ideal diagnostics probably would be a combination of an "Aldo in a Can" rule-based expert system, condition-based monitoring, self-diagnostics in fieldbus instrumentation, loop analysis software, and OEE calculations. All this is certainly possible with KM.

Who's in Charge, Anyway?

Science fiction is full of stories about how fully automated systems run amuck and destroy the world. From Forbidden Planet to 2001: A Space Odyssey to Terminator, the same theme prevails: When given too much automation and authority, machines fail.

"Perhaps one surprising finding from our use of KM is that a greater degree of automation is sometimes questioned," says Paulonis of Eastman Chemical. "If everything is automated, operators lose opportunities to work with the process and maintain a feel for it. It may not be optimal if the only time they need to take charge is during an upset or emergency."

Consulting engineer Bothe agrees. "Information itself is not intelligence, but the methods by which the information is processed, presented, interpreted, and acted upon are," he cautions. "Remember, data is meaningless unless used by a human or artificial intelligence to formulate relationships, patterns, and trends."

All the software and knowledge management in the world can't replace an experienced operator or engineer. One of my heroes continues to be Jack Godell, the control engineer in The China Syndrome (1979), who knew there was something wrong with the Fontana nuclear plant, no matter what all the data said.

In 1979, I lived in Coatesville, Pa., about 50 miles downwind from Three Mile Island. Twelve days after the movie was released, we had our own China Syndrome, when the automatic systems failed. I am glad that TMI had its own Jack Godell, or I might be glowing in the dark right now.

Although KM and modern control systems make it possible for a plant to run in full automatic mode, is that what we really want? As Taylor puts it, "Nobody would fly in an airplane that had a single control pushbutton marked Push to Fly,' but this is exactly the kind of control system some people expect operators to use."

Maybe there is some way to enter the lessons of sci-fi movies into expert systems, so the software knows when it should call upon human help.

Build Your Own KM System

You can experiment with knowledge management for free. Just go to www.thebrain.com and download the free trial version.

The Brain lets you store documents, text, pictures, drawings, e-mails, web sites, links, and all sorts of information. You can store it under any topic you like, in whatever way makes sense to you. Later, you can search the Brain database by keyword, general topic, category, or any other means.

The free version lets you set up a personal database on your PC, organized any way you want. It takes a bit of fiddling to get comfortable with the concept, but after a while it becomes second nature to take whatever you happen to be looking at on the screen--such as a web page, e-mail message, HMI display, alarm message, heat exchanger startup instructions, programming idiosyncrasies, or notes on how to reset the individual channels on a HSC card--and dispatch it into the Brain's database under whatever category seems suitable.

Later, when you are looking for, say, the heat exchanger startup instructions, Brain will find them with a keyword or topic search.

The Brain can be expanded enterprise wide. The system is called Enterprise Knowledge Platform, BrainEKP. Terminals all over the plant can store information in a central Brain database. For example, an instrument control technician can enter discoveries and shortcuts to troubleshooting specific equipment and systems at the nearest terminal, and it will get stored in the main database server for access by all.

Although the Brain is not specific to the process industry, it seems to be as useful as the other generic tools being used for knowledge management--such as Access, Lotus Notes, and handwritten instructions in a looseleaf notebook. Prices start at $80 for a single user version.