Second-tier systems replace legacy controls

By Rich Merritt, Senior Technical EditorPERIODICALLY, we cover the Second Tier of process control system vendors. As you will see in the accompanying "The Industry's Top 50 Suppliers" article, the hierarchy of control system vendors consists of a clumping of eight “Big Boys” at the top (Emerson, Honeywell, Invensys, ABB, Siemens, Rockwell, Schneider and Danaher) followed by a huge gap to the Second Tier companies gathered below. Last year, the gap was $400 million in annual sales.The Second Tier companies are not separated from the Big Boys just by sales figures: Their philosophies are different, their approach to marketing is vastly different (because of their often amateurish attempt at marketing, you probably never heard of some of the lower 42 in the Top 50), and their product offerings tend to be narrower and more niche-oriented. The Big Boys tend to spread out across the product continuum, offering everything from field instruments to process controls to asset management software, while the Second Tiers appear to concentrate on doing a few things very well. This can consist of serving specific industries such as power or paper, providing specific kinds of HMI/SCADA software, “adding on” capabilities to other systems, or providing general-purpose, lower-cost solutions.  It also may consist of exploiting chinks in the armor of the Big Boys. For example, when it comes to supporting legacy systems, the Big Boys are vulnerable. Some big companies are notorious for dropping support of certain products, charging a fortune for parts and service of legacy systems, and otherwise making it very difficult for end user companies to continue using the same brand for control. In contrast, the Second Tiers offer interesting, useful and often less expensive approaches to legacy system replacement. This article describes case histories where products and services from Second Tier vendors replaced legacy control systems. Our purpose here is to demonstrate that elegant, useful and cost-effective systems are available for process control from second-tier vendors. Be sure to consider the Second Tiers when shopping for a control system.A Waterfall of Savings
When it costs less to replace a legacy system than it does to maintain it, that legacy system is history. So it was in Upstate New York. The Water Resource Department in the town of Tonawanda, N.Y., just north of Buffalo, operates and maintains hundreds of miles of water and sewer lines. Each year millions of gallons of water drawn from the Niagara River are treated, distributed, collected and cleaned to serve the town’s 80,000 residents.       FIGURE 1: WATER CONTROLS

Software from Iconics replaced software from Intellution when the cost of supporting the legacy system got too high.

The wastewater treatment facilities were controlled by PLCs and an Intellution FIX32 HMI/SCADA system, but the software was getting too expensive to keep. According to Ed Rick, system engineer for the town, the price for support of the FIX system and other third-party applications, such as reporting solutions, became too high. The department began looking for a replacement system that could work with the existing PLC hardware.The town installed Genesis32 Enterprise edition and Web HMI software from Iconics at both the Water Department and the wastewater treatment facilities. The two locations have a total of five Genesis32 servers and five Web HMI servers. Iconics software is also deployed at 13 remote pumping stations connected over leased lines back to the control room. Workers in all locations can view displays on standard Internet browsers. Other Iconics software being used includes GraphWorX32 for operator screens, TrendWorx32 for real-time trending, and AlarmWorX32 for distributing alarms and events to the appropriate people.The system was installed by town employees, with sales and technical support from BCS, a systems integrator in Buffalo, N.Y.  OPC technology and redundancy both play a significant role in the application. Kepware OPC drivers connect Genesis32 to Modicon PLCs and various industrial control devices. The system has approximately 4,000 I/O points, and redundancy is built in at several levels. There are redundant servers in each location and each location works as a backup for the other. A wireless Ethernet between the locations runs over a Cisco backbone, while DataWorX32 software keeps the plants in sync. Redundant OPC drivers and redundant Ethernet are used, with a recovery time after a failure of less than 300 msec.Switching to Iconics paid for itself in the first year, says Rick. “For the price of one year of support on the old system, we installed Genesis32 and WebHMI on 32 bit platforms, and gained more technology, such as the ability to have true thin client viewing via standard Internet Explorers.” Why didn’t the town simply upgrade to the latest version of FIX? Rick says the Intellution solution would have cost 25% more than the Iconics system. “Iconics was able to reduce our operational cost while providing us with technology we can grow with,” says Rick. “In addition to lowering the support cost for maintenance contracts, supporting the system internally is now minimal. With WebHMI, all the administration and licensing is done on the server.”

Windows Woes Cured at Canner
The legacy control system at McCall Farms in Effingham, S.C., was based on “whitebox” PCs and an older Microsoft Windows 95 operating system that was no longer supported by Microsoft or the control system vendor. It tended to “crash” when operators entered production line changes, causing process data to be lost and production to stop. Legacy systems that crash usually get replaced.

McCall Farms produces a variety of canned fruits and vegetables under the Margaret Holmes brand, and distributes them throughout the southern U.S. in major national retail stores and major supermarket chains. Its canning process is continually moving with minimal allowances for equipment downtime and wasted product. Plus, according to a McCall Farms mandate, all products must be canned within ten hours of harvest.

The canning process starts with fresh fruits and vegetables bought from farmers across the southeastern U.S. Each type of produce has its own unloading area where it goes through a series of wash tanks for thorough cleaning before being sent to a blancher.

FIGURE 2: CANNED TOUCHSCREEN

Touchscreens allow operators all over the plant to request data, see trends, and change parameters.

Non-edible material and bad product are removed in the inspection area. The good product is sent to a filler, which puts the correct amount of food in each can. The cans are sealed and sent to a continuous feed rotary cooker where they undergo cooking and cooling according to FDA standards. In the final stage, cans are brought to the label line where they’re labeled, wrapped, palletized and readied for direct shipment to purchasers. This entire process is automated. Without automation, everything would have to be manually set with rheostats and toggle switches. The assorted can and pouch types McCall Farms uses require up to four line changes a day. “An on-line change would sometimes cause computers to lock up, stopping the process and generally causing us to lose any food in production at that time,” Jason Durant, McCall Farms’ electrical engineer said. “Obviously, this was not a good situation and made us reluctant to make changes unless absolutely necessary.”Fortunately, part of the solution was already on hand. Several years ago, as part of a project to fully automate its 500,000 square foot plant, McCall Farms installed 4,000 feet of Profibus I/O wiring, numerous touchscreen HMIs, and racks with 70 nodes of Beckhoff IP 20-rated I/O. Unfortunately, the canning process was still being run by the clunky legacy system, and software had become a source of nagging problems. "We were programming with our previous software, but it was continuing to cause process errors that we could no longer tolerate," he said. "Complicating things even more, the program had been discontinued four years earlier and would only run on an outdated Windows operating system." The company bouight Beckhoff TwinCAT control software and Beckhoff Industrial PCs to replace the legacy system. “The ‘buy’ part of the equation wasn’t very difficult,” says Durant. “TwinCAT licensing was about 25 percent less expensive than other offerings we looked at and, even better, there aren’t yearly upgrade fees.”Durant was able to expand to as many Profibus drops as required in an area and could easily add or retrofit drops as needed. There are currently three Profibus trunks –each using a separate Beckhoff C3640 PC running TwinCAT for plant-wide process control. At present, the Profibus trunks have a combined total of just over 100 drops.The software changeover was not difficult. "Our process has programming with a lot of sub-routines. Beckhoff worked with me for a couple months to figure out the best route for making a painless conversion to TwinCAT." The User Defined Function Block (UDFB) capability of TwinCAT turned out to be a major timesaver. "There was a lot of repetitiveness in our programming and the rungs were exactly the same with just different bits on them," he explained. "I made a UDFB, copied the block and changed the bit rather than having to create entire rungs over and over again. There were several operations that previously took three rungs to get one function done. Instead of using three rungs, I used the UDFB. It cut hundreds of rungs worth of logic down to 70 rungs, and amounted to about a 30 to 40 percent savings in programming time.” TwinCAT adheres to the IEC-1131 environment. “We used three of the programming languages and tailored them to the area of the facility we were controlling. These are Sequential Function Chart, Structured Text and Ladder Logic programming. As a person with no ‘formal’ programming experience, I found TwinCAT to be very easy to use.” The change-over has been so seamless that McCall Farms machine operators don’t know that they’ve switched to TwinCAT. “It has been ‘business as usual’ for all involved without any major problems,” Durant said. “We upgraded our existing HMI program, which directly links into TwinCAT via an OPC Server connection, but the operators didn’t notice any difference with the possible exception of some improved graphics.”Recently, McCall Farms avoided considerable expense from potentially wasted product because TwinCAT runs in the Microsoft Windows kernel, and is independent of other operating system processes. A whitebox computer in their cook room locked up and Durant was unable to make an immediate corrective action. But, because TwinCAT runs in the kernel mode, the process was able to complete without problem, resulting in zero loss of product. With the old system, all five of the rotary cookers and all four of the pot cookers would have went down, Durant said. “This saved us several thousands of dollars that we definitely would have lost had we still been using our previous software.” Looking back at how much progress McCall Farms has made, Durant sleeps a bit easier at night, "We don't have crashes any more and we feel a lot more confident that the equipment will be running when we come to work in the morning." New System Ups Power Plant Performance
St Johns River Power Park (See Figure 3 below), located on a 1900-acre site in northeast Jacksonville, Florida, is a coal-fired generating station owned jointly by Jacksonville Electric Authority (JEA) and Florida Power & Light Co (FPL). It has two 640MW units, each with a Foster Wheeler drum boiler and a General Electric tandem compound steam turbine-generator. The plant uses a variety of domestic and foreign coals, as well as 20% petroleum coke.      FIGURE 3: BETTER COAL BURNING

Legacy control systems at Units 1 and 2 at St Johns River Power were both upgraded.

St Johns needed to respond faster to automatic dispatch commands. In addition, they wanted to startup faster and burn myriad fuels without having to re-tune the control system. Finally, it wanted to provide its operators and engineers with better tools to improve day-to-day operations. This required replacing legacy control systems with something more modern. Replacing two complete control systems is a challenge at any plant of this size. Further, it required that planning, contract award, engineering, testing, installation and commissioning be done in about ten months.A modernization of this scope usually requires extensive up-front planning to develop project guidelines, requirements and vendor specifications. St Johns management initially considered hiring an architect engineer for this work and to oversee the project, but realized the time involved would preclude startup according to their schedule.They chose instead to have Metso Automation do a thorough study of the existing systems and propose a solution to meet schedule and performance requirements. Based upon the study, Metso Automation and St Johns personnel decided to modernize the plant with two Metso maxDNA automation systems. Metso Automation provided a turn-key solution, taking prime responsibility for design, installation and commissioning of the new control systems and field equipment. It was agreed to emulate “hard” control schemes that were familiar to plant operators. Further, St Johns wanted Metso Automation to implement the latest generation of Leeds & Northrup’s original D-E-B coordinated control, which assures parallel control of the boiler and steam turbine generator. Plant operators were included on the project team. They provided extensive input to the design of operating graphics. The team worked together to determine standards, display layouts and functions. Another goal was to make the transition to the new system as smooth as possible. To that end Metso Automation designed the control strategy configuration to match the existing system wherever possible. Plant Manager Paul Smith said, “for the most part, the control sequences, indicators, alarms, operator response sequences, etc. of the new system were designed to mirror what our unit operators have been familiar with for the last ten or twenty years.”Data base parameters from the existing system were imported into the new maxDNA system and tuning constants were imported to speed automatic operation on startup. Unit 2 was upgraded first, during the annual Spring outage. The new maxDNA system, which had already arrived, was installed as the older cabinets, cables and consoles were removed. Within four days of startup, Unit 2 was operating in automatic mode and within two weeks the unit was responding to automatic dispatch commands.While the new system was being installed, plant operators were being trained four hours per day on a maxDNA-based simulator (See Figure 4 below). The plant-specific simulator included all of the startup and shutdown sequences, plus a variety of situations that could lead to a unit trip, from auxiliary failures to process upsets. “The simulator was one of our cornerstones of success with the project and I would strongly recommend including a simulator with any planned upgrade,” says Smith.FIGURE 4: ON-SCREEN TRAINING     

Operators at St. Johns were trained on a simulator supplied by Metso Automation.

The plant burns coal from Western Kentucky, Columbia, other South American countries and several other sources at times. “We blend in 20% petroleum coke,” explains Smith. “Since the moisture content and BTUs are variable, it is important that our control system detect them and initiate smooth proportional corrections without triggering process upsets. We are definitely sold on Metso Automation’s Heat Release algorithm. Our old D-E-B control system used this algorithm and we wanted to duplicate the success we had with it. It allows us to burn a litany of different fuels without having to make constant tuning adjustments to our critical process loops. As a result we have been able to lower costs by purchasing attractively priced random lots of coal and pet-coke on the spot market.”  “This industry is all about reliability today,” Smith said. “So we don’t put any system on automatic until completely tested. Debugging a new control system can take months and I am aware of some plants that had new control systems and still didn’t have all bugs out after two years. During our startup we performed extensive testing. Our system dispatchers were pleasantly surprised we were able to control in less then two weeks.”After the successful startup of Unit 2, St Johns upgraded Unit 1 in the spring of 2005 with another Metso maxDNA system.Software Replaces PLCs
Even the biggest PLCs can run out of steam, and need to be replaced or upgraded. At Bonlac Foods Ltd., a food processor in Tasmania, Australia, a PLC-5 was running at its capacity, and could not expanded to handle more equipment. Bonlac processes milk arriving by tanker from dairies and produces a range of powdered milk products. The plant was controlled by a network of Allen-Bradley PLCs communicating with each other via a DH+ link. The plant also has a number of large Profibus valve networks.The wet process area was originally controlled by a PLC-5/80E that was out of memory and I/O capacity. The ladder program consisted of over 12K rungs of ladder code, comprising control of drives, valves, 17 PID control loops and 200 control sequences. All these were controlling routing and cleaning of milk lines and silos, where each line must be rinsed, caustic washed, rinsed, acid washed and rinsed again before use.The plant required the addition of 30 more Profibus-linked valves and associated equipment. Since the existing PLC-5/80 --the largest Allen-Bradley PLC-5 model-- was completely full, no simple expansion was possible. After reviewing the possible upgrade/replacement options, including adding another Allen-Bradley PLC and dividing the logic between two controllers, SoftPLC was chosen. The SoftPLC offered the choice of a drop-in large capacity single PLC replacement that would easily interface with all the existing equipment, networks, PLC controllers and Citect SCADA system. Another huge advantage was that the configuration of all the other equipment did not need to be changed in any way. Using tools supplied by SoftPLC, the Allen Bradley ladder logic was imported to SoftPLC with minimal user intervention. Then, the new 30 Profibus valve controls were added to the system, along with the existing seven racks of SLC-500 I/O. The system was upgraded with minimum disruption to the process; in fact, the plant continued production throughout the upgrade. From Homebuilt Controls to PLCs
Exopack is a manufacturer of plastic bags located in Tomah, Wisc. Its production line evolved over the years, using a variety of different equipment and controls. The setup to run a new product was very time consuming and wasted a lot of bag material. If the line was stopped, it was difficult to restart and wasted more material in scrap. Industrial Automation, a systems integrator in LaCrosse, Wisc., was asked to come up with a solution to consolidate the varied controls and make the line easy to setup and run.The bag making process starts with large pre-printed rolls of plastic film. The web is threaded from the stock roll over a pressure sensing roll and then to a web guide and folding device. After leaving a folder, the folded material enters a fin seal system where the open side of the web is heat and pressure bonded. The seal is inspected as it leaves the fin sealer and defects are marked with an ink spray unit. The web then travels through the first nip roll and dancer system, and is directed to the bag gusset forming sections of the line. Here the bag web is rotated 90 deg. as it proceeds to the gusset forming section. To facilitate the twist, the bag web is filled with air that is trapped between two web nip points. Near the end of the bag twist section the bag sides are heated and then the bag pleats (gussets) are formed on either side of the bag tube. Upon leaving the gusset section final nip the bag web is again collapsed with gusset folds formed on both sides of the web. Next in line is the pull roll nip and drive system that is used to pull the bag stock tube to this mid point in the production line. The auxiliary equipment section includes a mixture of electromechanical devices that create the gusset tacks, corner tacks, and vent holes in the bags.  This system requires eight temperature controlling PID loops. The last section of the production line is the bag bottom seal and cutoff machine. This machine serves as the master speed and timing reference for all upstream devices. Freshly cut bags exit the machine by a conveyor that empties to a second perpendicular conveyor for bag stacking. By manipulating the exit conveyor speeds, good exiting bags are stacked in groups of 10. Any bags identified as defective earlier in the process are stacked together with different conveyor spacing.“Because the production line is very long, and required many PID loops, we chose to divide the control system into two control enclosures,” says Herb Burfield, manager of Industrial Automation. “We used two Automation Direct D2 PLC systems to control the line, a 10-in. EZ-Touch touch screen as the primary point of operator interface, and a 6-in. EZ-Touch touch screen as a secondary point of operation.”The primary HMI includes operation prompts, alarms, setup screens, maintenance screens, device test screens, and graphical history display screens. The secondary HMI contains normal run operation screens and limited device setup screens. The end result of the above applied automation is a production line that can be setup and run from a cold start in minutes with very little scrap material. “Previously, startup could occupy most of a shift and waste rolls of material,” notes Burfield. “Once started, the line can now be stopped and restarted at will without creating any scrap. Depending upon product the production speed is 30% to 50% faster than possible prior to the automation and bag quality and consistency is improved.”Legacy DCS to Redundant Controls
Crawfordsville Electric Light & Power in Crawfordsville, Ind., wanted to replace its legacy Bailey Network90 DCS because it was old, parts were hard to come by, and it could not support CEL&P’s increased need for operation and performance data. CEL&P called upon Advanced Boiler Control Services in Orland Park, Ill., to help them put together an integrated system to control two 21 MW turbines. They decided to install a fully redundant Matrix control system from MTL-Most.The Matrix system has a backup controller for redundancy, connects to third-party devices via Modbus Ethernet TCP/IP, and uses a Wonderware system for the HMI and plant historian. An integrated, centralized development environment in Matrix coordinated control engineering development for process control and IEC 61131-3 programming languages, including design implementation, documentation and maintenance. Project documentation is automatically maintained in an as-built state from the initial engineering phases through subsequent system modifications.