"Ask the Experts" is moderated by Béla Lipták, (http://belaliptakpe.com) process control consultant and editor of the Instrument Engineer's Handbook (IEH). He is currently recruiting contributors for the 5th edition. If you are qualified to contribute or answer questions in this column, or want to ask a question, write to [email protected].
Q: We are having an internal argument as to who loads the DeltaV machines when it comes to the Windows software. My question is should we allow the "Business IT" group to do it, or is this the "Controls" group's responsibility? Are there any regulations about this?
Robert Angelloz
[email protected]
A: Over the years, I've received 322 questions for this column, but none was more important than yours. It's so important because it asks about the domains of information technology (IT) and automation technology (AT), and if a "firewall" is needed between them. This is important because, running any industry, it is essential to clearly understand who is responsible for what activity in order to make that industry safe, reliable and efficient, while complying with all regulations and producing globally competitive, high-quality products.
It must also be clearly understood that one cannot control a process without fully understanding it, and that it takes education and decades of experience before one can obtain the required level of understanding. A good process control (AT) engineer fully understands the process he/she is operating. This is an important distinction because a ChE, EE, ME or IT professional understands only one aspect of the plant design, but not the total process.
In the distant past, computers were new, and only the programmers and IT engineers understood them. As our DCS systems began to look like desktop computers, the IT departments got involved with them, but this period is long past, and modern management clearly understands that this age is over. They now understand that IT is for business and AT is for plant operation. They understand that on the outside the DCS systems might look like computer equipment, which the IT people used to work on, but inside them is the combined know-how of the AT profession about which IT people know nothing. Therefore, the domain of the AT department must include all that has to do with plant operation (Level 1–dealing with measurement; Level 2– dealing with control; and Level 3–covering manufacturing execution and operations management, or MES and MOM).
AT is different from IT because the control of industrial production operates 24/7 and in real time! For example, if the operating pressure is dangerously high, AT immediately opens a relief valve. (This is not what the people at Fukushima did, who waited seven hours, while they debated business and PR implications of an accident, and thereby caused hydrogen explosions). AT is also different from IT because society as a whole is entering the age of cyber terrorism. Remember what the slammer virus did at the Davis-Besse nuclear plant, or what Israel did to the Iranian centrifuges. Therefore, while very hard to achieve and inconvenient to top management, the only foolproof protection against virus attacks on industrial plants is to disconnect the AT domain from the outside world.
This is a basic difference from IT, which can and must use the Internet all the time, and must also secure the company business systems. Naturally, while the AT and IT domains must be totally separated, IT display screens can also be present in the control room to keep plant operators abreast of business related information.
Today, the "magic" of computers is gone, so IT should be involved only with the business and its associated networking needs in order to collect data and assist management in scheduling production, estimating market demands and making decisions on personnel management, salaries, job titles, etc. IT should not be involved in process operations because they don't take place in the "business environment." For example, IT people usually do not understand that one cannot just load and update software, work on cybersecurity, or do patching at any time they feel like it because doing the wrong thing at the wrong time can shut down the plant or worse. Therefore, such in-plant activity must always be directed by the AT department, which (in larger plants) should also have at least one IT specialist in it.
Today, unfortunately there are no regulations that clearly separate the AT and the IT domains, and the few standards that do exist (ISA-95, TÜV) are rather vague. What the process control industry needs is a "firewall" between the IT (business) and AT (operations) domains, and universal standards that clearly define where the IT domain ends and the AT domain begins.
Béla Lipták
[email protected]
A: My response is that no critical control computer should ever be on the network and thus subject to IT control.
Bruce Land
[email protected]
A: This historically has been a very important question, although perhaps less so today as process automation groups have been assimilated into the IT organizations in many companies.
I recall vividly the tensions that existed between process automation and IT groups in the company where I worked. In general, IT groups did not understand the special challenges and needs of control manufacturing processes in real time. Process automation groups had to correct program code errors in their real-time process control computers ASAP—in minutes, if possible—while IT groups required an extensive, bureaucratic, change control procedure that sometimes took weeks to complete. There are two paradigms that have been used by some companies.
- Process Automation (process control) is responsible for those computer systems that have "real-time" operating systems, and generally need to be available 24/7. IT usually has everything else. Such real-time computers typically are those that control manufacturing/pilot plant operations, and, in many cases, also include laboratory automation systems, especially those systems performing assays.
- Process Automation is responsible for those computers and components that exist within levels 0, 1 and 2 in the computer layer architecture defined by S-95. IT has all the higher levels.
Regardless of where you draw the line in differentiating process automation from IT, the fact of life today is that these two domains are quickly evolving to where they need to interface with one another, including exchanging information and data. This is also apparent when reading and understanding the S-95 standard because it is clear that the different layers need to interact with one another.
Process automation and IT groups need to work together.
Joe Alford
[email protected]
A: Not an easy question to answer, and I expect that all companies will have their own view on this. The approach of my employer was to develop the "Converged Plantwide Ethernet Design and Implementation Guide," in conjunction with Cisco, to help define domains of responsibility between automation and IT. You can find the document at www.ab.com/networks/architectures2.html. It won't answer your question explicitly, but it might give you some food for thought.
Vivek Hajarnavis
[email protected]
A: The design should segregate the process control network from the broader IT network. These networks should be separated logically and physically by a firewall. I have found that the best definition is that the IT domain ends at the firewall and the process control domain begins at the firewall. Administration of the IT devices is then the responsibility of the respective groups within their own domains.
However, having said this, we have also found that IT processes are much more tested and rigorous when it comes to administering IT infrastructure. They have years of experience behind their teams, and process control is quite immature in this space. One possible solution to to have a dedicated IT resource for the process control domain, somebody that can understand both points of view.
Emre Eren
[email protected]
A: This has been the topic of Holy Wars between engineering and IT for as long as I can remember. We continue to move those lines within our plants as technology shifts back and forth. Recent technology involving the migration of SCADAs from PC workstations to virtual machines running on a server threatens to swing it back towards IT.
- There needs to be a separation (whether this be a physical separation or using VLAN technology) of front-office machines from process-control machines. This is required to keep the patching and software upgrade activities from disrupting manufacturing activities. This also facilitates keeping the plant-floor machines from making connections to the outside world.
- The operating system security models must allow technical users to have rights to run application software without compromising function.
- The IT community has a lot of tools at its disposal. It is useful to maintain an architecture on the control side that can take advantage of those diagnostic aids. Islands of control networks, lack of managed switches, etc., can hamper troubleshooting efforts as control system become more and more dependent on network communications.
Joseph A. Ruder, PE, CA
[email protected]
A: I do not know if there are regulations in place for such a thing, but I do know that usually there is a work around if companies have a standard operating procedure defined. IT and Automation work together to keep company asset working in good order. It is team work and has its own unique challenges. In today's complex systems, no one person can know it all.
In IT world it is referred as DMZ for demilitarized zone between business IT and SCADA IT. All SCADA/BPCS routers and switches are separate and independent.
Hiten A. Dalal, PE PMP
[email protected]
A: It's not a black-and-white issue any more, as control systems are no longer isolated islands. Each company gets to decide for itself how to handle this issue. However, people obviously need to be "qualified" for what they are going to be doing. How will you train and qualify your IT people for working on the control system? The reverse is also true; how will you train and qualify your control people to work on IT systems?
Paul Gruhn, PE
[email protected]
A: Process control (PC) doesn't belong in IT. Twenty years ago, some in management thought that computer engineers were the best fit for PC as the DCS started looking like a desktop computer. Of course this idea never really took off as PC is NOT about computer programming.
The database and algorithms running in a DCS are controlling a continuous process, there is no time for "your call is important to us, please remain in the line for our next available representative ..."
No IT group has PC expertise. How could they possibly support a plant shutdown due to a valve that is not responding after a "software or antivirus update"?
IT's place is to support PC, as far as I'm concerned.
Sigifredo Nino
[email protected]
A: There is no absolute answer. The most common structure that has evolved is a specialized group in IT that has knowledge has knowledge of process control and automation and with background has moved into the IT group.
Many Level 3 (MES, MOM) and Level 2 (Control and DCS) systems have specific Windows requirements. Often they are not using the same version as the rest of the corporate systems. There may be operational requirements on the manufacturing software that prevents it from being upgraded at the same time as normal business IT software. Therefore, if IT has responsibility for managing all Windows systems and managing all networks, then there must be the specialized group in IT that has the policies and procedures in place to ensure that the critical systems remain operational and are not handled as other business systems.
If the industrial networks are segregated from the business networks (different domains), then the automation groups may have responsibility for the networks and servers in the automation systems. This works well until the domains must be merged (for common password and account support). Once everything moved to a single corporate domain, the IT department usually takes responsibility for loading the O/S and ensuring that it meets the corporate security requirements (right version, any anti-virus software, etc …).
There are no regulations that I am aware of in the United States. There are regulations that require that the O/S configuration is known and under control, but nothing that states which group is responsible.
In my opinion, the specialized group within IT is the best solution today. The worst is to have the Level 3 and Level 2 systems handled by the general IT group. They may upgrade or shut down the systems because of a lack of knowledge of the applications and their criticality to safety and operations. The second best solution is to have the automation group responsible for loading and maintaining the Level 3 and Level 2 systems. However, in this case, there must be close coordination with the IT group to understand their policy changes and how they will impact the manufacturing systems.
Dennis Brandl
[email protected]
A: There are no standards and no hard and fast rules. Where enterprise IT leaves off and plant controls picks up varies greatly from company to company. There is always a turf war between IT and automation. "We get along great now that I run both departments," said a reader a few years ago in a poll.
This turf war is exacerbated when it comes to cybersecurity, patching and loading of software and updates. The reason for this is that IT and automation are diametrically opposed in their goals. The goal of IT is to protect the data in the servers, and they will cheerfully cut off all the peripherals to do that. The goal of automation is to keep the plant running, and they will cheerfully sacrifice the servers to be able to keep operating. See the problem? IT departments don't see why automated updating and patching can't be done at 6 p.m. every day, because it works in the business IT environment. They don't always realize that doing that risks shutting the plant down and perhaps doing it catastrophically.
If IT insists on enforcing enterprise-type upgrading and patching standards, you WILL have an accident, and it may well have fatalities. There is no reason, however, why you can't train IT personnel to use automation upgrading and patching rules instead. You will have to negotiate this. Once IT understands your needs, they should respond in the appropriate fashion.
Walt Boyes
[email protected]
A: This is, unfortunately, a controversy in several companies. It always seems to stem from the supposition that IT is responsible for all computers, regardless of application or location. That may once have made sense, but this is a controversy that should have disappeared long ago. Unfortunately, jurisdiction in a corporate environment, once established, doesn't change rapidly. The primary basis for assigning responsibility was once based on the use of a complex and expensive tool – the computer. However, jurisdiction needs to evolve because computers are now ubiquitous. They are everywhere, and almost all functions within a company usually rely on or involve a computer, no matter how tiny, in some way. The primary basis for assigning responsibility should now be based on the function of a device or system, whether it contains a computer or not.
Clearly, a control system involves many technologies and responsibilities requiring the knowledge and understanding of professionals educated in the many complexities and details of industrial control. Those people need to be held responsible for the behavior – all of the behavior – of the control system. The fact that it contains a computer is of no real importance – almost everything contains a computer.
IT should be responsible for centralized business systems, as well as the networking components that impact corporate and business-oriented systems. Also they should obviously be jointly responsible with control organizations for networking interfaces involving control systems, since networking impacts both. So the jurisdiction should be based on functional responsibility and interactions – not on whether a system or device contains a computer.
Lynn W. Craig
[email protected]
A: In our company, it is the controls group that takes care of the DCS, but the regulatory responsibility is not clearly defined. It is logical to have the control group be responsible for DCS, as it is connected to field instrumentation. Also P&IDs, cause and effects need to be built in DCS. Also, the instrument engineers are knowledgeable about measurement and control. In my view the same separation should continue.
H. S. Gambhir
[email protected]
A: In the beginning (1970s), this question did not exist since microprocessor-based and computer-based process control systems were not common. PID control loops and hard-wired and electrical relays were used. In 1980s, microprocessor-based control systems began to be common, while in 1990s DeltaV and other PLC control systems entered process control.
Instrument engineers did not consider DeltaVs computers, but IT people treated them as software, since control logic, configuration, etc. were used to make the device (DeltaV) work. Other computer-controlled systems, such as vibration monitors, anti-surge control for compressors, etc., were added in the heated debate between instrument engineers and IT people. IT people tended to view the whole picture as a function of Internet security, software validation, system integrity, etc., and often had been authorized to approve the system before a computer installation was to be procured.
In 2000s, model-based control, system identification, statistical analysis, SIS, etc. became popular, and IT people became more aggressive in taking control of the approval process. During an emergency or when a process was down because control signal failure, loss of measurement signals, valve failures, etc., instrument engineers were sent to fix the problem. When the office computers were down due to hacking, viruses, software compatibility, etc., IT people were sent to fix the problem.
I am not qualified to judge what IT people know. However, anyone who wishes to decide what instrument engineers should use in process control had better be equipped with the ever-evolving knowledge in instrumentation.
Gerald Liu
[email protected]
A: Emerson—as all DCS vendors—takes great care in testing DeltaV as a system with each software update. This includes the software for the HMI that happens to run on a Windows platform. If your IT department were to install any updates, there is a distinct possibility that those updates may break some DeltaV function. You will not be covered for any DeltaV system malfunction resulting from using a Windows version different than that supported by the DeltaV version you have installed. I believe that Emerson does offer a service contract that includes updating Windows software, but only after they have validated that it does not affect any DeltaV functionality.
Allowing your IT department to update Windows on your HMI or engineering workstations is a bad idea. We always tell our end-user clients to tell IT that the HMI and engineering stations are parts of a manufacturing machine and are not computers.
Richard H. Caro,
[email protected]
