By Bill Lydon, Editor
This article on Automation Usage of IT Network Technology is the third article in a series covering the recent annual Pharmaceutical Automation Roundtable (PAR).
I had the privilege of attending the Pharmaceutical Automation Roundtable as an observer in November 2011. This PAR was hosted by Johnson & Johnson in Spring House, PA, with Dave Stauffer, Terry Murphy, and Joel Hanson of Johnson & Johnson participating.
Lead automation engineers from various parts of the world attended the invitation-only, two-day event. This is the most knowledgeable group of automation professionals gathered in one place at any one time focused on discussing automation issues. A range of companies participated including Abbott, Amgen, Biogen Idec, BMS, Genentech, Genzyme, Glaxo, Imclone, Johnson & Johnson, Eli Lilly, Lonza, NNE Pharmaplan, Novo Nordisk, Pfizer, and Sanofi-Aventis.
The PAR was founded about 15 years ago by Dave Adler and John Krenzke, both with Eli Lilly and Company at the time, as a means of benchmarking and sharing best practices for automation groups among peer pharmaceutical companies. The group specifically does not discuss confidential or proprietary information, cost or price of products, price or other terms of supply contracts, or plans to do business or not do business with specific suppliers, contractors, or other companies.
The individual PAR group members have a wealth of practical knowledge and knowhow to share with other participants, truly learning from each other.
Topics are agreed upon prior to the meeting and a member with make a presentation on their organizations views and approach to the topic. After this presentation others comment on their organizations situation.
This topic was about networking approaches and considered the differences between manufacturing automation control system networks and business IT networks.
The presenter described their company’s past configuration and goals for a unified approach and business results. In the past each facility had its own “manufacturing network” - some connected via firewalls to the corporate level. Each site designed their own system and governance policies, resulting in uncoordinated and/or conflicting network maintenance and no application sharing. There was a huge variance in the quality level of network performance, security, and validation methods across the plants. Site differences included unique services, back up/recovery, and even wire tagging, etc.
A couple of years ago the company set out a vision for a global network infrastructure to achieve:
- Improved Quality
- Improved Security
- Knowledge & Resource Sharing
- A Platform for Global Manufacturing Applications
- Centralized Shared Services
- Area Back Bones For Common MES Applications
- Reduced Risk for Unplanned Plant Stoppages
The mission was to build a coherent global network across all manufacturing sites (which are worldwide) consisting of process networks and site backbones all based on common IT standards to ensure a globally consistent quality level of the IT infrastructure. Accomplishing this included virtual tunneling between sites with local services such as antivirus and Active Directory maintained from a central location, all accomplished on corporate network lines as virtual networks. Many sites also have Gigabit fiber networks for high performance. The rules and minimum requirements for being part of the network were defined with specifications:
- Hardware components (Wires, Switches, Racks, Servers etc.)
- Configuration (Security, Firewalls, OS, WINS, DNS, DHCP)
- Centrally managed Antivirus and User Administration
- Central Operations & Maintenance Organization
- Operational Procedures and User Guides
This is a validated network that went operational April, 2011 and includes KPI dashboards that provide information including system availability, system changes status, incident status, and other system information all on one screen.
The next steps include central governance of peripheral devices (bar code readers, label printers, etc.) and applications that are to be allowed on the network, and central back-up / restore services. Another task is documentation to the point that network administration could be outsourced to another service supplier.
These are issues noted in the presentation based on their first-hand experience.
- IT Quality level varies greatly by site location and from the pool people available for hire to do these services.
- In-house IT team lack of experience and competencies in technical skills and understanding for automation validation and overall program controls.
- Company viewpoint was that this was an expensive program for something that was already working.
Now that this is in place the real cost savings are now being made visible and tangible.
It was noted that they expected to get resistance from local sites, but so far the sites have been cooperative. They all want convergence with central services and the harmonized service level. Central administration takes a lot of burden off of the local site resources.
These are comments from other participating PAR companies discussing their experiences. This is valuable interaction and a key business value of the PAR meetings.
Based on numerous comments, it seems clear that there is a need for a support level that bridges automation and IT, which is a hybrid skillset. At this time, the combination of IT skills, process knowledge and automation is hard to outsource without risk.
Outsourced IT that created problems for sites seems to be a large area of potential risk (downtime, production disruption, etc.).
Successful programs have a close collaboration between IT and site automation people, which is at best difficult to achieve using outside or outsourced IT contractors.
Your thoughts and comments are welcomed.
Part 3: Automation Usage of Network Technology (You are currently reading this article)
