In an effort to minimize shutdown duration for the remaining I/O points, we built a replica of the I/O room, known as the mock-up room, in an adjacent basement — this level of physical simulation of an I/O room was new to Lilly DCS upgrade projects. The mock-up room (Figure 2) enabled electricians to complete some work in advance of the shutdown (e.g., cutting wires to length, pre-labeling terminations, and fabricating controller and termination board panels) while getting a feel for working conditions during the shutdown. The mock-up room also allowed the hardware team to predict with a great deal of certainty the amount of time needed by electricians during the actual shutdown. This led to a shorter shutdown — the money saved by this more than paid for the cost of creating the mock-up room.
New operator workstations, controllers, servers and two fiber optic networks also were needed. The operator workstations essentially were replaced on a 1:1 basis. We determined the number of new controllers and servers based on past experiences with DeltaV at other similar facilities.
Operator training. We used two primary methods to get operators up-to-speed on the new DCS. The first involved a short computer-based training course developed in house to cover the basics of the new control system (navigation, alarming, graphics, logging in, etc.). Operators completed this training in advance of the shutdown. Once the shutdown started and removal of the old control system began, operators attended roughly 40 hours of training courses that used a process simulator that exactly replicated both the hardware and software of the production DeltaV DCS. The latter courses comprised an initial walkthrough of the new software followed by simulation of production batches using manufacturing tickets and SOP. These simulations included non-routine events, to give operators experience in this type of event handling.
Commissioning and qualification. Upon full wiring termination and power-up of the new controllers, loop checking began. The C&Q team developed a refined loop-check strategy to minimize cost and schedule impact. The nature of the conversion project meant the majority of the field wiring remained intact. So, the loop-check procedures allowed for testing only wiring impacted by the upgrade. We tested analog instruments just for signal continuity rather than the traditional multipoint current simulation at the field device. Loop checks for automated valves were completed automatically — a custom program made each valve stroke open and close. Any valve state failures received additional attention using more-standard loop troubleshooting techniques. Thanks to this approach, we verified proper wiring on 200 of the 250 automated valves in 20 minutes without any manual labor . With the loop checks completed, C&Q commenced. The C&Q strategy typically involved running a water (or process buffer) batch to test code functionality, tune control loops and check batch totalizers. The water batching also allowed for in-process leak checking, minimizing health and safety issues associated with restarting the equipment. Operations personnel executed the water batches, with process engineering, automation and contract C&Q personnel on shift. The familiarity the process team and operations personnel already had developed with the software led to a relatively uneventful qualification.
We met the primary goal of the project —safely upgrading the DCS in a timely and cost-effective manner. The shutdown duration was shorter and the total cost was lower than for similar projects within the company. We kept total cost well under budget and also achieved benefits in alarm rate, cycle time and number of manual actions required. Alarm rate was cut by 95% , while cycle time of the bottleneck step in the control room was reduced by 6%. In addition, we automated numerous manual activities. For instance, passing data automatically from one unit operation to the next removed calculations from manufacturing tickets. In effect, tanks determine their own addition/dilution volumes, columns ascertain their own charge volumes, etc. Buffer operations were automated to the extent that SOP no longer were needed — representing a 40% cut in the total number required. Also, we automated many non-routine operations (special cleanings, purified water sanitizations, etc.). The conversion included equipment state tracking (hold clean, run counts, etc.), as well.
Given the success with the first control room, Lilly decided to upgrade two more control rooms in late 2008. Using the methods developed during the first upgrade, this follow-up project provided even better results. It achieved a 36% reduction in cost per I/O, while attaining similar benefits to those observed in the first upgrade. Figure 3 shows the cost breakdown for both the initial and follow-up project. As the second upgrade was completed, another Lilly API facility started a similar Measurex-to-DeltaV project, with implementation scheduled for mid-2010.
The author thanks Ben Paterson, Kevin Duffy, Kevin Wilhelm, Brian Hrankowsky, David Chan, James Cox, Eric Hubert, Brett Conaway (ACE Technologies, LLC), Graham Smith (Brillig Systems Inc.), Doina Morusca (Invensys Process Systems), and the Cornerstone Controls Inc. team for their contributions to this article.
1. "Batch Control Part 1: Models and Terminology," ANSI/ISA-S88.01-1995, The International Society of Automation, Research Triangle Park, N.C. (1995).
2. "GAMP Guide for Validation of Automated Systems: GAMP 4," International Society for Pharmaceutical Engineering, Tampa, Fla. (2001).
3. Chan, D. and B. Hrankowsky, "Migration of a Biotech Insulin Manufacturing Plant from Measurex to DeltaV," presented at Emerson Global Users Exchange, Washington D.C. (Oct. 2008).
4. Iannucci, R., "A Practical Approach to Alarm Management," presented at ISA Expo, Houston, Texas (Oct. 2008).
Ryan M. Iannucci is a senior process control engineer for Eli Lilly and Co., Indianapolis, Ind. E-mail him at Iannucci_Ryan_M@Lilly.com.