Pharmaceutical Automation Roundtable (PAR) - Part 2 - Virtualization & Software Configuration Management

Pharmaceutical Automation Roundtable (PAR) - Part 2 - Virtualization & Software Configuration Management

 
By Bill Lydon - Editor, January 2011
 
This is the second article in a series that are the result of the annual Pharmaceutical Automation Roundtable (PAR). The individual PAR group members have a wealth of practical knowledge and knowhow to share with other participants, truly learning from each other.
 
I had the privilege of attending the Pharmaceutical Automation Roundtable as an observer this last November at Pfizer's Andover, Massachusetts biotech facility. The PAR was co-hosted by Jim LaBonty and James Galloway, both from Pfizer’s central engineering group. Over 30 automation lead engineers from various parts of the world attended the invitation-only, two-day event. In my opinion, this was likely the most knowledgeable group of automation professionals gathered in one place at any one time focused on discussing automation issues. Life Science companies represented in the PAR include Abbott, Alcon Labs, Allerga, Amgen, Bayer, Biogen Idec, BMS, Boehringer Ingelheim, Centocor, Cook Pharmica, Dow, Genentech, GenMab, Genzyme, GSK, Imclone, J&J, Eli Lilly, Lonza, Merck, NNE Pharmaplan, Novartis, Novo Nordisk, Pfizer, Roche, Sanofi-Aventis, Schering Plough, Talecris, and UniLife.
 
The PAR was founded about 15 years ago by Dave Adler and John Krenzke, both with Eli Lilly and Company at the time, as a means of benchmarking and sharing best practices for automation groups among peer pharmaceutical companies. The group specifically does not discuss confidential or proprietary information, cost or price of products, price or other terms of supply contracts, plans to do business or not do business with specific suppliers, contractors, or other companies.   
 
This year’s PAR topics covered the following items:
 
  • Manufacturing Execution System Projects - Benchmarking
  • Control System Virtualization Approaches
  • Governance Organizational Structures
  • Software Development Environment & Configuration Management
  • Control System – Automation Lifecycle Management and Long Range Planning
  • Electronic Testing Tools & Validation
  • Wireless Networks
  • Disposable Technology and Automation Implications
  • Control Loop Operating Modes – Operator use of Auto & Manual modes relative to Safety
 
Virtualization Experience
 
A PAR member started the discussion by relating their experience deploying Virtual Machine (VM) technology. The systems were deployed using blade servers with 2 CPUs per blade with 6 cores per CPU. Their IT department has standards for VM deployment and recommended approximately 3 applications per core and 30-36 applications per blade. The DCS vendor recommended 2 cores per application (minimum) which the automation group has adopted. A key feature is the ability to dedicate CPU cores to specific applications. The system configuration is two blade enclosures with 16 blade server capacity each. Each rack has 8 servers with 10 terabyte storage (at this time). The retrofit system has consolidated 65 servers and they are using 50% less Ethernet switches. The system is dedicated to automation and the automation group will be supporting the VM environment. There are a number of reasons that the system is “owned” by the automation group. For one, IT has weaker standards for process validation and change control than automation requires. The automation group is working to secure service level support agreements with the help of their internal IT group.
 
Comments from the PAR group
 
Some old control system software cannot be migrated to VM technology particularly UNIX based systems.
 
The VM systems should be near manufacturing plants due to communications reliability and security issues.
 
Many HMI software packages do not support redundancy which is desirable when configuring VM systems.
 
Some companies are leveraging established IT processes such as server backup.
 
IT provides patches a week in advance so the automation group can test them before the application and operating system are updated.
 
IT has unique validation methods, change control that are not directly suitable for process automation and this is a growing issue that needs to be harmonized.
 
My Observations
 
The adoption rate of virtualization is high and it is another tool that can have great long term benefits.
 
There can be a big savings and performance increase by eliminating Ethernet connections and switches by running application software in one virtual machine rack.
 
Management of the virtual machine by the automation group seems logical since process validation and operations requirements are more stringent and more focused on the process than compliance required by business IT.
 
Software Development Environment & Configuration Management
 
Another PAR member started the discussion on this topic by relating their experience with Software Development Environment & Configuration Management. They have cut startup and commissioning time by using controls in the lab to build, test and verify controls and HMI, etc. before going into the field. This lab environment is also used for in-house training rather than sending people out for training.
 
The configuration management goal is consistent code delivery for maintainability. They created a configuration management standard and guidelines around the IEC 61131 control programming and S88 batch standard. This includes the adopted PDFS (Process Description Functional Specification) incorporating ISA 5.6 (ANSI/ISA-5.06.01-2007 Functional Requirements Documentation for Control Software Applications) and S88 (ANSI/ISA-88 Batch Standard) standards.
 
They are now using more skid based systems that lower startup time and insure reliable operation. In their plants they specify that the controls supplied with the skids to be from the same vendor as their primary DCS.
 
Comments from the PAR group
 
The idea of a complete off-the-shelf, vendor-delivered skid system with controls “is fiction that only exists in a project manager’s mind, it doesn’t really happen.”
 
Consistency between skid vendors is a problem. The vendor’s standard is never as good as you were led to believe. Skid vendors propose that their supplied control schemes carry less risk because they are supposedly proven already in the pieces of equipment that they have sold to multiple companies. “The reality is the control is never as mature as they try to pass it off to be.”
 
Consistency between skid vendors is a big issue. It is a question of customizing their standard offering and taking the risk that modifying to site requirements will result in a complex and difficult to maintain control system. 
 
Do we struggle to change the skid vendor’s supplied controls to meet our requirements, taking on more risk or take the deduction for controls and do them ourselves to meet our control and safety requirements?
 
Rely on high level coding specifications and provide block representation of functions that skid vendors need to adhere in their programs.
 
Enforce consistency on interfaces not on code by demanding that vendors provide historic data consistency using naming conventions and global libraries.
 
When the skid is delivered we own the source code.
 
My Observations
 
Skid delivered systems offer great advantages, but they create problems with control systems and automation. This was generally a sore point with participants. The solution may be better specifications for the integration, control, operations, and network interfaces which requires a more refined procurement process.
 
The controls and automation on a skid become part of the plant just as much as site installed controls and automation. Ideally the manufacturing company needs to be able to efficiently maintain, troubleshoot, and do continuous improvement on the control and automation embedded in the skid. This cost should be factored into the lifecycle cost analysis of purchasing skids to expose the real cost tradeoffs.
 
Demanding that controls adhere to open control standards such as OPC, IEC 61131, and PLCopen may become more important.
 
Summary
 
The Pharmaceutical Automation Roundtable is a terrific event and I am sure attendees gained a number of ideas. The next article covers the group’s thoughts on Executive Governance, Electronic Testing Tools, and Quality by Design (QbD).
 
Links to the other articles in this series: