The instrument shop: How to optimize this key to a safe, efficient and productive plant

It’s become apparent that, while the primary goals of our instrumentation and control systems is to keep process operations under reliable, safe and accurate control, the amount of information they have made available has reached an almost overwhelming level, and there is no end in sight. This has put considerable pressure on the performance of the instrument shop.

Further muddling the shop’s complex task are rapidly changing hardware and software technology, the coming of the Cloud and big data analytics, the Industrial Internet of Things (IIoT), artificial intelligence (AI), machine learning (ML), augmented and mixed reality, cybersecurity concerns and digital twin technology, which all to a large extent depend on plant instrumentation for their informational foundation.

Let’s discuss some of the challenges facing current and future instrument shops and how to achieve the necessary performance. Here, the term “instrument shop” is generic, covering the traditional instrument shop, an I&E shop to include the electrical maintenance aspect, the process control group, the DCS digital specialists or analysts, and other maintenance or engineering groups that service and maintain our plant instruments, control systems and safety systems. Some of the indicators you may have problems in the instrument shop are given in Table 1.

Goals for greatness

Table 1: Signs of an instrument shop in trouble

1.	Large number of bad actors (instrument systems that are repeat offenders or large consumers of instrument shop resources).
2.	Large number of instruments out of service or waiting for work.
3.	Large amount of repeat work.
4.	Maintenance budget growing but plant is not.
5.	Large work backlog.
6.	Instrumentation in general in run-down condition or in disrepair.
7.	Poor operator opinion of instrumentation operation, availability, accuracy, poor MTTR, regular complaints about service.
8.	Poor shop morale.
9.	Large personnel turnover.
10.	Low shop experience level or abnormal experience distribution.
11.	Poor safety record.
12.	Safety systems not tested on time.
13.	Poor plant instrumentation documentation (e.g. not correct, out of date, poor quality, etc.).
14.	Backlog of calibrations, testing and inspections.
15.	Small training budget, limited training.
16.	The instrument shop staff is who other people at your site consider difficult people to work with. Poor relationships with other plant groups.

One of the obvious goals for an instrument shop is above-average performance, with a stretch goal of best-in-class performance, to provide high availability and dependability of the plant instrumentation and control systems, to provide the requisite instrumentation accuracy for good quality products, to maintain the safety integrity of all safety-related instrumentation, and to have a low mean time to repair (MTTR). Some goals that are not so visible are good maintenance discipline, efficient use of resources, maintenance budget in line with the required tasks, good relationships with other plant groups, and ensuring personnel and plant safety.

To achieve best-in-class performance, plant management must support and facilitate the necessary actions. Unfortunately, many times, good performance is the result of good people figuring out how to work around the obstacles placed by management that prevent them from doing a good job. It’s not the most efficient method, and while it’s often successful, it’s sometimes painful.

For an instrument shop to properly maintain the plant instrumentation in the condition necessary for safe and efficient operations, it must have maintenance budgets that reflect the work to be done. The company management must realize that the maintenance of the plant requires x amount of work to achieve safe and efficient operation, which requires a certain budget amount based on the capability and performance of the maintenance group. Cutting the budget without considering these things typically results in a poor level of maintenance, and in the case of the instrumentation, compromised control and information-gathering as well as potential safety consequences.

It’s far better to improve maintenance capability, performance and reliability by design of the plant. One look at the BP 2005 Texas City refinery accident shows what reducing maintenance budgets without concern about the work to be done can get you.

Plan for the future

An instrument shop may have an adequate or even excellent mix of experience and capability today, but things change, and tomorrow or in five years they may be different. The instrument shop will be the most affected maintenance shop when it comes to changing technology. There is also typically a lot more turnover these days due to changing demographics and company polices that do not encourage company loyalty, making it necessary to ensure a good mix of experience, capability, and competence today and tomorrow. You certainly don’t want to be caught in the “hit by a beer truck” scenario, where a substantial amount of your expertise is embedded in one person (particularly true of small plants), and that person steps off the curb and gets hit by a beer truck. Failure to plan for the future is planning to fail.

Competence includes a combination of good experience, knowledge, skills, abilities, behavior and attitude. Competence distribution tends follow the experience distribution if the experience quality is good, there is a good training program, and the technicians have the ability to perform the given tasks. Maintaining competence also requires maintaining the existing experience, training for current and projected future tasks, providing experience with the new tasks as they become available, maintaining tribal knowledge and intellectual property, and planning for projected changes in competence and experience.

Capability is also a key requirement that must be managed as many current and future instruments are complex, high-technology systems where new and unique problems can emerge almost daily, and maintenance can be quite demanding. Capability includes the ability and willingness to take on new challenges, which requires people who are flexible and can embrace new ideas.

Ensuring the availability of competent people requires both short-term and long-term planning. You must ensure that your competence and experience (the old guys who know everything and where all the bodies are buried) doesn’t walk out the door without planning for the loss and recouping as much as possible of that experience and knowledge. If you don’t plan for replacing that experience (via retirement, turnover, etc.), you are planning to fail.

Take and give ownership

One of the typical ways to improve performance is to give technicians “ownership” or responsibility for instruments or group of instruments (and they accept that responsibility), because people tend to work harder when they have a clear sense of accomplishment and are rewarded when above-average performance is achieved. It may just be a pat on the back, but it’s always appreciated.

The commonplace work order system is great for organizing maintenance, but can depersonalize instrument maintenance because the technicians operate from work order to work order with no responsibility beyond the work order activities. This typically doesn’t give the instrument technicians a good sense of their contribution to the whole and little ownership or pride in their efforts.

Ownership is essentially responsibility with benefits, and works to cash in on most people’s natural desire to accomplish things and do a job that they can be proud of. It doesn’t work for everybody, but it works for many people and, surprisingly, it can get some of the rest to do a better job by example and group dynamics.

The benefits can be achieved by giving people responsibility for groups of instruments and providing benefits for high performance. It may take creativity in places where singling people out based on performance is frowned upon (e.g., many union shops). The technician can still work on other instrumentation, but they will have primary responsibility for their group of instruments.

Keep tabs on morale

Besides competence and organizational skills, good shop morale is a big indicator of how a shop will perform. Happy people are hardworking and loyal people who will work together for common goals. I saw firsthand the effect of morale on unit performance when I was a U.S. marine. Good morale comes from fair treatment, giving credit where credit is due, providing benefits for performance, all people pulling their own load and helping others when the going gets tough, a good company/employee relationship (something missing in many companies today because of bottom-line management), and good group dynamics. The instrument shop supervisor is a key player, managing and facilitating the group dynamics with a light touch, and protecting the group from things that fall from above.

Use CMMS and AMS

We live in an increasingly complex world in modern process plants, and can hardly do without instrument-capable computerized maintenance (CMMS) and asset management systems (AMS). This world isn't going to get any simpler, and there’s a high probability that it will get more complex. Plants can generally achieve significant reductions in operating costs and downtime as a result of implementing an effective asset management strategy (with the key word being "effective").

One of the differences between now and the old days is the number of instruments in a plant and the quantity of information available from them. An instrument asset management system is a must to manage it, as well as the information that will become available in the future through increased use of condition monitoring instrumentation, the IIoT, the cloud and big data analytics, the plant digital twin, and machine learning/AI systems. Success requires a computerized AMS. These systems typically provide network access via HART or fieldbus, and can help simplify instrument maintenance and documentation. (For a further discussion, read Instrument asset management systems).

A CMMS is also typically needed to manage the work process and to better track your instrumentation maintenance resources (such as labor, materials and equipment) to improve maintenance budget predictions. Effective use of both systems are required to achieve best-in-class performance.

Invest in good tools

The many same technologies that are giving us challenges in instrument maintenance also give us new tools to help us. We have new test equipment, such as documenting process calibrators like those from Beamex, Fluke, and other manufacturers; portable oscilloscopes; network analyzers and instrument communicators. Fluke and FLIR give us digital infrared cameras and imagers that can help troubleshoot equipment and process problems indicated by high temperature or temperature change.

Equipment monitoring, as well as some process monitoring, has entered a new era, with wireless allowing monitoring sensors in many places where they couldn’t go before due to location or cost, which can improve troubleshooting capability. Emerson and ABB are a couple of the available suppliers of these new wireless condition monitors, which can provide additional information that allows detection and, in some cases, prediction of potential problems to help maintenance plan its actions.

Eliminate bad actors

Instrument loop “bad actors” devour instrument shop resources. They’re the loops that require repeated maintenance actions or instruments whose failure causes a large expenditure of maintenance resources (e.g. a maintenance hog). They’re typically the result of a poor design or misapplication. Once identified, if the instrument technicians can’t resolve the issue in a reasonable amount of time, the instrument shop should enlist the help of a competent plant instrument engineer to resolve the issue.

The key here is to have good relations with the instrument engineering group so each group can contribute their best skills and knowledge to solving the problem. Besides repeat offenders, another good place to look for bad actors is in the control room. Look at how many loops are in manual (and why); how many alarms are defeated, disabled or manually bypassed; and if any interlocks or safety systems are regularly bypassed during operations. Ask the operators what problems they’re having with the instrumentation.

Unfortunately, the work order system generates work only when someone requests it, and requests are typically to solve an immediate problem. The result is a very narrow view of instrument maintenance, and typically the minimum amount of work is expended to clear the work order. Above-average performance can’t be achieved, and certainly not best-in class-performance, if only the minimum amount of work is expended on a consistent basis. One solution is for an experienced I&E improvement engineer to be assigned to the plant, dedicated to improving the instrument and electrical systems and to helping solve longstanding instrumentation and electrical problems. I have seen the best technicians assigned this duty successfully (with some help from Engineering).

Out-of-service instruments, on the other hand, typically represent bad actors that the instrument shop or operations have given up on, or whose maintenance is not considered a priority. They can also represent a lack of resources (time, money, personnel availability, expertise, etc.). One of the most terrible accidents in recent times, the 2005 BP Texas City explosion, had contributing factors including instruments on the Raffinate Splitter tower being out of service (level gauges), instruments not working (high level alarms), level transmitters not properly calibrated, and instruments not fully tested prior to startup. The BP instrument technicians described the unit instrumentation as being rundown and in disrepair. That was an accident waiting to happen, and it did.

It’s easy to forget about maintaining the field indicators when we have all these fancy instruments that send their data to the DCS. But one has only to observe what the board operators do when they have an abnormal situation to see their importance—they call the field operator to go check the level gauges, pressure gauges and other field indicators to verify that what the DCS is telling them is actually happening. It goes to an old Russian proverb that quantifies one of the operator’s main operating principles (paraphrased here): “Trust your instruments, but verify what you can from your field instruments.” In the 2005 BP Texas City accident, which was essentially a tower overfill situation, the tower had its level gauges out of service. One has to wonder what would have happened if the level gauges had been functional.

The operator’s opinion of the instrumentation is also important because they live with it 24/7. If they don’t trust some of instruments, there is probably a good reason that deserves investigation.

Ensure cybersecurity

Table 2: Cybersecurity steps

1.	Have the procedures and discipline to ensure that the instrument write-protect jumper or switch is engaged after maintenance.
2.	Disable or control access via thumb drive to the instrument shop computers.
3.	Control access to asset management systems.
4.	Don't use any default passwords and only provide password access to qualified people.
5.	Control any software updates and revisions that might introduce a cyber threat.
6.	Use industrial-grade ICS cybersecurity appliances for any external connection to the control system, and industrial-grade firewalls for instrument shop computer connections to external systems.
7.	Remember, Ethernet is a walk in the park for computer hackers.
8.	Never leave your PLC or SIS logic solvers in the program mode.
9.	Limit and tightly control any remote access into the instrument, control and electrical systems.
10.	Be aware of any abnormal operation or anomalies in the instrument systems that affect multiple instruments or systems (that is not the result of grounding problems).

Cybersecurity is starting to become a big concern for process plants and manipulation of plant information is the goal of many cyber threats, which puts the instrument shop on the front lines. The push to connect all the instrumentation to the IIoT opens a large door to cyber threats.

The instrument shop also uses computers for asset management, PLC programming, and other uses. Many times, these computers are connected to the enterprise Ethernet, which is connected to the Internet, which introduces another cyber vulnerability.

Updates from manufacturers are often downloaded via the Internet, and it’s possible that malware will also be downloaded. The concept of using cloud connectivity to provide a plant digital twin with big data analytics will be just piling on potential vulnerabilities.

Non-business programs uploaded from a thumb drive (memory stick) also can be a threat. The modern instrument shop should be aware of these vulnerabilities and protect against them. Table 2 provides some things to be considered to protect the plant, instrumentation and control systems against cyber threats.

Embrace big data

Much of the future of maintenance in general and the instrument shop in particular lies in the realm of AI, ML and mixed/augmented/virtual reality. AI and ML technologies will use information gathered from the plant instrumentation and plant operations to learn about the plant, and will use big data analytics and machine learning to detect anomalies and failure patterns of plant equipment, unit operations and instrumentation. These capabilities can provide increased levels of predictive maintenance and AI-assisted maintenance, and help minimize unscheduled downtime.

Mixed reality will provide instrument technicians better access to data, information to help troubleshoot and maintain systems, the ability to visualize their instrumentation systems in new ways, assistance and expertise on an on-demand basis. They will also require a lot more condition- and process-monitoring instrumentation to provide the necessary information to AI, ML and cloud-based analytics. The information coming from the plant instrumentation will play a very big part of this and will require that the instrument shop be a high-performance organization to help to ensure the success of this endeavor. Is your instrument shop up for it?