Who should provide your network support?

            Many networks fail because of basic plant environmental conditions. The right physical connection choices such as those shown here can eliminate much of the worry.
Source: Woodhead

---

Rockwell Automation sees a trend toward customers with service contracts choosing long-term maintenance contracts, rather than point service, which yields limited benefits when technicians tend to come in cold on a problem. Slivka points out that many network failures occur because of good old-fashioned plant environmental problems, i.e., intermittent connections due to dirt, nasty chemicals, vibration or outright broken ones from accidents such as forklifts cutting cables or smashing into equipment (See Figure 1). So before you call in the specialist, make sure you’ve checked the obvious things first such as device LEDs at both ends or confirm that you at least have signal.Connie Chick, controller & I/O business manager for GE Fanuc Automation, says its customers often enlist outside services to design and build network infrastructure, but once the network is up and running, they generally like to take care of their own networks.Remotely Helpful
Another source of help are the remote monitoring services many vendors offer. It still, however, has its detractors. “We’ve offered remote monitoring for some time on our machines, but it has yet to take off in the U.S.,” says Luis De La Mora, marketing, international sales, for Rovema Packaging. “Remote diagnostics are very popular in Europe where there are more solutions in the field, and more customers have reached a comfort level with remote diagnostics.”When does remote monitoring work well? For G. James Australia, an integrated glass and aluminum manufacturer and contractor, remote monitoring can be used safely from thousands of miles away to fix a PLC problem. Headquartered in Brisbane, Australia, G. James uses Cisco networking equipment and software, and recently integrated its factory floor systems with its business systems, linking factories across five major sites, manufacturing in Australia and Malaysia, and outlets in four countries. “When I was in the U.S., I was notified of a problem in one of our Sydney plants,” says David Moy, technical services manager. “Instead of explaining the nature of the problem over the phone, I was able to VPN connect back to Brisbane and use the internal network to access the relevant PLC in Sydney and fix it.”Why Not Insource?
Many manufacturers realize their business systems must be in sync with plant systems, and this means getting plant floor engineers to work with IT, the primary answer for insourcing. G. James wanted to integrate administration and manufacturing systems to reduce costs. Administrative and machine control networks were autonomous and did not communicate. Orders on the manufacturing floor were completed using paper and pen causing long delays in collating information. “Orders were commonly filled before all the paperwork was collated,” says Moy. “The process was complex and time-consuming, and customers could not be kept up-to-date. So we began looking into extending our network to the manufacturing floor. We knew that by connecting manufacturing processes to order management systems and automating them, we could streamline our operations, improve visibility, increase efficiencies, and enable our customers to remotely inquire as to the status of their orders in real time.”The most significant challenges were integrating administration and manufacturing systems so staff could gain access to real-time information, and enabling the automated equipment to be managed by the network. This meant crossing boundaries between administration and operational control on the factory floor and redesigning process control systems so they could be interrogated remotely. “This vision has been from the top down, with systems being built from the ground up,” explains Moy, “to ensure communications of end devices with upper applications, because it’s easier to build the upper applications knowing the boundaries of system end points.”How can successful insourcing work? Manufacturers working on this issue have achieved varying results depending on their internal resources. A controls engineer for a global coatings and materials company notes that he hopes to get to the point where corporate IT and the control group understand each other’s needs. He says that they’re not there yet, and the biggest issue is on-time reliability of the network—where the business systems can afford blips, but the control equipment can’t. Right now, he says, his company maintains distance between the plant floor systems and the business systems, although they are investigating pulling data upward into their SAP system. According to him, many security issues still have to be worked out.Mark Buettner, director, electrical & controls engineering, and Terry Ledoux, director, information systems—product supply, Nestle Purina PetCare—North America, presented “Plant I.S. & Controls: Shared Responsibilities Operations Guide” at the recent ARC Performance Driven Manufacturing Forum. They concluded that such “turf wars” had arisen for several reasons including open network topologies and PCs becoming a shared resource with different needs between controls and the IT department, and that there was no defined ownership or cooperation between the two groups.Neither organization took the time to understand the complexities of its counterparts’ operating environment and end goals. There was too much focus on controlling and maintaining the “kingdom” of knowledge.In addition, misunderstanding and lack of cooperation led to reluctance between organizations to provide timely and reliable services to one another.These issues and more were covered in “A Delicate Balance,” (Fall, ‘04 p12), which offers a thorough discussion of how the controls group and IT department can find common purpose.

Security: A Problem Regardless
Security issues affect networks whether work is outsourced or insourced. “We outsource, and that’s why I know outsourcing is a bad, dirty, untouchable subject,” a senior plant engineer from a major vitamin company adamantly proclaims. He cites examples of contractors who indiscriminately lock down systems, making them difficult for operators to use. He accuses contractors of having a double standard. “While they preach about how to avoid viruses and worms by staying away from certain web sites and not opening e-mail attachments from unknown senders, the same contractors bring in infected notebook computers, tie them into the control system and infect machines on the plant network,” he laments.

His company now takes care of all its DCS equipment, the process information computer, etc. “Before, we had e-mails show up on the root directory of our process information system,” he states. “And that’s because we let these boys maintain it.” He says he’s seen it all—including contractors who wrongly think they can back up open files. “They can’t maintain files that are open. They can’t back up an open file,” he fumes. “So I ask myself what the hell are they doing in there?”

Of course, not all contractors fall into the category just described. But before selecting a third party to maintain your network, consider an incident that occurred September 2, 2003, during an outage at the Davis-Besse nuclear power plant in Ohio. The U.S. Nuclear Regulatory Commission (NRC) alerted plant operators to a potential network vulnerability caused by the Microsoft SQL Server worm. This warning described a worm infection that increased data traffic to the site’s network, resulting in the plant’s safety parameter display system and process computer being unavailable for several hours. Though safe operation of the plant was not affected, NRC regulations require that safety-related systems be isolated or have send-only communication with other systems. The cause of the infection was a contractor that made an unprotected computer connection to its corporate network, through which the worm reached the plant network. NRC also found that plant computer engineering personnel were unaware of a security patch that prevented the worm from working.

Insourcing: Security Assured
With insourcing and the merging of groups and networks, security issues become highly relevant responsibilities. “My world changed dramatically when the blaster worm hit,” says Dave Jones, automation engineer at Ash Grove Cement, Overland Park, Kan. who has sole responsibility for this company’s networks. “That was when we started taking these issues very seriously.” Jones notes that Ash Grove has been fortunate so far, and with the right precautions, it’s possible to head off disaster. Concerns remain, however; for example, numerous OS patches need to be tested with control software before they’re placed on a computer. One of Jones’ pet peeves is cleaning up spyware and keeping it off computers in the first place. “Educating users is not enough,” he says. “Putting strict policies on computers, clamping them down, and providing specific Internet-only machines can help clean up problems.”

When IT groups and control engineers don’t communicate, bad assumptions from each group can spell security problems. Holly Beum, president and consultant, Interface Technologies, works with both Fortune 100 and smaller companies alike. “In the majority of cases, nobody is doing much of anything, which is very scary,” she says. “Some control engineers think that the IT department is protecting them through an upstream firewall, and they don’t realize that the firewall is not sufficiently secure—primarily because the access control rules are set wrong (See Figure 2 below). The IT group thinks that if the control engineers needed security, they’d let IT know. So security falls between the cracks. It’s not a well-defined responsibility. The biggest problem is the lack of security awareness on the part of control engineers. If they realized their systems were insecure, they’d do something about it.”

FIGURE 2: FIREWALLS LEAK

A properly installed and configure firewall protects and separates. The control engineers might think the IT department is protecting them through that upstream firewall, but who makes sure the access control rules are set properly? Source: Cisco

Companies that insource may still need to outsource their security needs—at least until they get familiar with all the nuances of security. In this instance, security experts, SIs, and vendor companies stand ready to help. Says Invensys’ Eva, “We remind users that while they’re learning how to protect their systems, others are learning how to attack their systems. What you put in is obsolete the day it’s installed. The unfortunate part of security is that the weaknesses are always changing.”Work Together
No matter the size of the manufacturer, insourcing has many benefits including quick response time and keeping the knowledge of the network in house, which many companies perceive to be an important security benefit in itself. A potential downside, namely giving engineers more challenge and responsibility, making them more marketable to others, can be a plus as well. Engineers who feel challenged and rewarded in their jobs will stick around as long as they are adequately compensated. If you insource, you may still need to outsource education and training. Outsourcing education on networks may be free from some vendors, or it may cost money, but in the long run, it will be worth every penny.ALL TRAFIC IS NOT CREATED EQUALLY

	Control	Video	Data (Best Effort)	Voice
Bandwidth	Low to Moderate	Moderate to High	Moderate to High	Low
Random Drop Sensitivity	High	Low	High	Moderate
Delay Sensitivity	High	High	Low	Moderate to High
Jitter Sensitivity	High	High	Low	High

The care and handling needs of control network data, although relatively low in bandwidth, are very sensitive to random drops, delays and jitter. Source: Cisco