A Reader Writes:
The technicians and operators in our brewery sometimes change the PLC programs to accommodate maintenance and production requirements. We're concerned that the FDA would take issue with this practice. What should we do to be sure our PLC program management would be accepted by FDA inspectors?
-From November 2002 CONTROL
Solutions:
You Decide
PLC and DCS control system configurations are a topic with many different opinions in the industry with regard to 21 CFR Part 11 compliance. Approximately half the companies we have spoken with believe the control configuration is an electronic record and requires the same level of revision control and change tracking as, for example, an electronic batch production record. The other half agree that modifications to the control configuration require change control, but do not believe the configuration falls under 21 CFR Part 11 jurisdiction.
...Certainly the PLC or DCS programs are not submitted to the FDA, but they do require validation to ensure proper functionality. One question to ask is, does the PLC or DCS configuration constitute an electronic record that satisfies any record or signature requirement set forth by the FDA? Many control companies, including Invensys, have taken the conservative approach and developed configuration change tracking applications that monitor the changes made to the control system configurations. In addition, there are third-party tools such as that provided by MDT Software.
...At a minimum, access to the PLC or DCS programming tools should be restricted by user ID, passwords, and/or physical barriers such as placing computers in rooms with locked doors to which access is controlled. Of course, the final decision comes from the FDA inspectors whose job it is to enforce the regulations. The FDA routinely updates its compliance guidelines. These guidelines, in conjunction with the ruling itself, should be used by the manufacturer to develop its own compliance opinion and implementation policy.
Daren Moffatt, Pharmaceutical Industry Business Manager
Invensys Process Systems, www.invensys.com/Archestra.html
Manage Changes
There are multiple issues to consider when PLCs are included in the mix of process control strategy and design. One can think of validation of processes and hardware as a snapshot of the design of the plant or process, as opposed to compliance, which is actually a process that maintains the design and the equipment in a state that is consistent with cGMPs and with the CFRs that apply to the pharmaceutical industry.
...Initial design of the PLCs includes both a hardware and a software (ladder logic) design that is used to control the process. The initial validation of the design would have verified the PLC hardware installation and also should have verified that the ladder logic was installed and controlling the process as designed.
...Once the validation of the design is accomplished, the fundamental principles of design control and compliance should be in place to allow minor corrections to PLC ladder logic as sensors change or as the process is upgraded.
...The key element in maintaining the PLC and also maintaining the design baseline is a SOP to determine what constitutes a minor and a major change. Here's what the process should be:
- A work authorization to make minor corrections should be documented and reviewed by engineering and validation prior to implementation. Emergency changes can be accommodated by allowing minor changes within an SOP that is written specifically for that purpose.
- If a work authorization is reviewed and found to substantially change or modify process parameters, whether software, hardware, or firmware, a formal design change process is necessary to accomplish and document the change. This is outside of the control and authorization associated with the SOP.
- Completed work authorizations should be reviewed by both engineering and validation groups.
- If the design change is determined to be substantial and affects the process in a way that can affect product quality, then a validation study should be performed to verify that the design change and subsequent process changes are within acceptable limits. This study should include an engineering review that examines effects on adjacent processes.
- The design baseline is updated based on the implemented design, and appropriate quality and validation organizations review and sign off on the design change and the resultant testing from validation studies.
Michael Burgin, Senior Consultant
Energy Science Applications, Powder Springs, Ga.
Its All About Security
The situation described is a direct violation of 21 CFR Part 11 although probably not sufficient to warrant a warning by the FDA. The FDA has given opinions that PLC programs are considered electronic records and as such fall under the requirements of Regulation 21 CFR Part 11. The issue is that changes to electronic records (such as PLC programs) need to be controlled and tracked. A few key things to look for are:
- Security: Make sure that only authorized individuals can make modifications to control programs. This needs to be user ID and password-based or biometric.
- Version control and history: Be able to go back and see what the record or program looked like in the past. Changes to version should also be approved by individuals other than the person changing it.
- Audit trails: A detailed log of all changes to the records and the system itself. If someone changes a program, it needs to log the date, time, user name, event, and any other details. The audit trail system needs to be secure to ensure that records cannot be falsified.
(Microbrewery Grows Big With Micro PLC)
...A product like Cimplicity Manager has the functionality required for managing PLC programs to meet FDA requirements. Manager helps companies meet 21 CFR Part 11 by requiring any access to programs to be under username and password control, by archiving any access/opening/downloading of PLC programs and by providing a means of recording exactly what changed by keeping track of versions of the PLC application program. Manager also works with any PLC manufacturer's equipment as well as any file-based software, and therefore can address from a central location a complete plant or company's PLC equipment regardless of model or manufacturer.
Gimmi Filice, Product Manager
GE Fanuc Automation, www.gefanuc.com
Programs Are Data
While many manufacturers automatically associate electronic records with information produced by humans, most do not realize that the programming codes and data produced by PLCs also are forms of data that can and must meet compliance regulations. In fact, customers who have inquired the FDA have found that programs within a PLC are considered electronic records, and as such, must comply with 21 CFR Part 11.
...All manufacturers modify PLC code on a regular basis, so eliminating changes to the controller is not an option. But auditing and archiving the changes is possible and does meet the requirements set out by 21 CFR Part 11.
...Software such as the RSMACC enabled change management system from Rockwell Automation tracks automation system and PLC changes, allowing users to detect and record altered electronic files, effectively auditing and archiving the data that falls under federal regulations. Users can restore previously used programs if needed, control program access, and manage data backup and recovery services.
(Oil & Gas Producers Innovate to Tap Hard-to-Reach Resources)
...Auditing online PLC changes (which may not be archived) is also key to maintaining compliance. Currently, Rockwell Automation is the only supplier offering software that can audit changes made online and not saved to an archive file.
Jeff Hamilton, Director, Rockwell Software Maintenance Solutions
Rockwell Automation, www.software.rockwell.com
Documentation Is Key
Managing changes on the factory floor to PLC programs is a serious concern in the overall validation process. Security, audit trail, and version control are the most critical aspects in developing an effective strategy to ensure validation compliance.
...Security can be accomplished in a number of ways, including physical locks on the PLC controller or panel. Embedded security in the PLC executive can be used to limit both local and network access only to approved system administrators. Any changes made on the factory floor need to be captured with a complete audit log that documents the scope of the change made, the rationale for the change, who made it, and the time and date. The audit log also needs to be encrypted to prevent changes or alterations.
...By using the IEC 61131-3 compiled code feature, which requires the original source code to download any changes, a user can ensure that the original source code reflects any local changes made to the program. We recommend you follow or develop quality standards for securing the source code on a protected server to provide version control. Version control can also be addressed through software-based utilities available from MDT, Taylor, and Stelex.
...The key to compliance is documentation. It is usually not the lack of procedural control, but rather the lack of documented evidence that those procedures were followed, that creates compliance problems for companies.
Mark Liston, Global Sales Director, Pharmaceutical Industry
Schneider Electric, [email protected]
Manage Recipes, Not PLCs
Managing PLC programs to meet FDA requirements has two approaches. One is to not manage the PLC program at all, but instead craft a solution using a validated flexible execution recipe capability linked to the PLC. To achieve this solution, an S88 execution recipe will contain "generic" phases both in the PLC and the corresponding recipe execution software. The phases (such as "metered add," "heat to x degree F," "agitate," etc.) will contain parameters appropriate for the phase such as setpoints, limits, timeouts, setup, or completion criteria.
...During recipe creation, the values are configured in the recipe by the recipe author; during recipe execution the phase instruction and parameters are sent to the PLC as a transaction. The PLC will execute the phase using the recipe parameters, updating the recipe of phase completion, alarms or output parameters. Upon receiving phase completion notification from the PLC, the next phase instruction in the recipe will initiate.
...If the recipe process order or a phase parameter requires modification, the recipe (under validated change management) is modified, approved, and released. The PLC software (phase) is not modified, but the process will react differently due to modification of the recipe parameters or the order of the phases in the recipe. The strength of this solution is in crafting and validating the generic phases in the PLC. If a process change is warranted, the recipe is restructured, but the PLC validation remains intact. The Honeywell POMS product contains this functionality, linking to a PLC or DCS using an OPC connection.
(Control System Buying Check List)
...The second approach is to include change control, change management, and 21 CFR Part 11 compliance within the PLC/DCS environment. This approach is difficult and requires a large commitment by the vendor to craft the functionality within the PLC/DCS environment. Honeywell is embedding change management, change control, and 21 CFR Part 11 compliance within Honeywell's Experion PKS for use in regulated environments.
Mike Power, Sr. Consultant
Honeywell Life Sciences, www.acs.honeywell.com
Make Sure Its Transparent
In operating a brewery, it is essential that the technicians responsible for production be able to use the PLC program to adapt to the changing needs of the brewery. The ideal solution should incorporate the existing workflow methods without changing work habits on the plant floor.
...Our solution is to use secure Internet technology to enable a remote data center to have access to the live PLC programs. This access is achieved using known VPN and WAN technology, and is available from a number of companies, including Rockwell Automation and ei3 Corp. Once the PLCs are connected to a remote data center, complete access is enabled. Online programming, data archiving, and most importantly, program uploads can be performed from the data center.
...Using an automated script handled by the data center computers, a copy of the program is uploaded on a scheduled basis; e.g., daily, weekly. The programs are stored with incremental version identification so that changes can be compared to the validated original program. Using a file archiving program, such as Microsoft's "Source Safe," enables many program versions to be tightly compressed and allows version notes to be attached to the files. By adding a web front end to Source Safe, programs can be made available to authorized personnel, and check-in/check-out procedures can be established.
...To provide independent verification that the PLC programs are properly managed, a consultant is retained by the brewery. This consultant is granted access to the various versions of the PLC program by securely accessing the data warehouse over the Internet. The consultant would compare the program versions (on a scheduled basis e.g. weekly or monthly), and write a technical assessment report on the nature of the changes and the potential impact.
...These reports could be made available to the FDA for routine spot-checking to see that there is compliance with this process. If the consultant finds a change that may have a significant or detrimental impact on the process, the brewery can be notified immediately. This solution is possible today using Rockwell Automation's In.Site products or ei3 Corp.'s suite of Internet tools.
Spencer Cramer, President and Chief Technology Officer
ei3, www.ei3.com
Work Around the Code
We don't have a solution for this particular problem, however it does hint of a conceptually related problem that some of our customers have encountered, and for which we have a solution. Many of our customers use PLCs in critically controlled processes that, once released, cannot be modified without a complete revalidation process due to potential liability. These types of applications include pharmaceutical product manufacturing, specialty food processing, and amusement park ride control.
...When a control problem occurs on these lines, the maintenance staff has their hands tied somewhat in that they are not free to modify the executing code in attempts to isolate/troubleshoot the problem. We have been providing a product for years called the Crakker Logic/Timing Analyzer, which has proven to be a very valuable tool for troubleshooting these problematic (especially intermittent problem) control systems without interfering with the control system operation.
(Where does the control system integrator go?)
...The Crakker instrument can be connected directly to various I/O points on a control system via isolated, quick-connect test leads and left on the production floor. The Crakker then benignly monitors and records all of the logic state changes and inter-channel as well as real-world timing associated with these changes. When a control problem occurs, the timing and logic associated with the I/O (before and after the fault) has been logged to memory. The data can then be downloaded from the Crakker memory to a PC where it can be viewed in a powerful timing chart-type display as an aid in troubleshooting the problem.
...The Crakker allows a view into the control system logic and timing, and trapping of problems without interfering with the operating control system.
Dave Parks, President
Logic Beach, www.crakker.com
March's Problem:
Can We Use Control Valves for Safety Shutdown?
We would like to avoid the cost of separate valves by using existing modulating valves for safety shutdown. Is this an acceptable practice under any circumstances? If so, where and how can it be done?
