A subsidiary of the International Society of Automation
Cancel
Search Sponsored By Sponsor

Automation Usage of IT Network Technology

  • May 07, 2012
  • Feature

Summary

Automation.com Exclusive - Part 3 from Pharmaceutical Automation Roundtable (PAR) 2011
 
May 2012
 
By Bill Lydon, Editor
 
This article on Automation Usage of IT Network Technology is the third article in a series covering the recent annual Pharmaceutical Automation Roundtable (PAR).
 
About PAR
 
I had the privilege of attending the Pharmaceutical Automation Roundtable as an observer in November 2011. This PAR was hosted by Johnson & Johnson in Spring House, PA, with Dave Stauffer, Terry Murphy, and Joel Hanson of Johnson & Johnson participating.
 
Lead automation engineers from various parts of the world attended the invitation-only, two-day event. This is the most knowledgeable group of automation professionals gathered in one place at any one time focused on discussing automation issues.  A range of companies participated including Abbott, Amgen, Biogen Idec, BMS, Genentech, Genzyme, Glaxo, Imclone, Johnson & Johnson, Eli Lilly, Lonza, NNE Pharmaplan, Novo Nordisk, Pfizer, and Sanofi-Aventis.
 
The PAR was founded about 15 years ago by Dave Adler and John Krenzke, both with Eli Lilly and Company at the time, as a means of benchmarking and sharing best practices for automation groups among peer pharmaceutical companies. The group specifically does not discuss confidential or proprietary information, cost or price of products, price or other terms of supply contracts, or plans to do business or not do business with specific suppliers, contractors, or other companies.
 
The individual PAR group members have a wealth of practical knowledge and knowhow to share with other participants, truly learning from each other.
 
Topics are agreed upon prior to the meeting and a member with make a presentation on their organizations views and approach to the topic. After this presentation others comment on their organizations situation.
 
Automation Usage of Network Technology
 
This topic was about networking approaches and considered the differences between manufacturing automation control system networks and business IT networks. 
 
The presenter described their company’s past configuration and goals for a unified approach and business results.   In the past each facility had its own “manufacturing network” - some connected via firewalls to the corporate level. Each site designed their own system and governance policies, resulting in uncoordinated and/or conflicting network maintenance and no application sharing. There was a huge variance in the quality level of network performance, security, and validation methods across the plants. Site differences included unique services, back up/recovery, and even wire tagging, etc. A couple of years ago the company set out a vision for a global network infrastructure to achieve:
 
  • Improved Quality
  • Improved Security
  • Knowledge & Resource Sharing
  • A Platform for Global Manufacturing Applications
  • Centralized Shared Services
  • Area Back Bones For Common MES Applications
  • Reduced Risk for Unplanned Plant Stoppages
The mission was to build a coherent global network across all manufacturing sites (which are worldwide) consisting of process networks and site backbones all based on common IT standards to ensure a globally consistent quality level of the IT infrastructure. Accomplishing this included virtual tunneling between sites with local services such as antivirus and Active Directory maintained from a central location, all accomplished on corporate network lines as virtual networks. Many sites also have Gigabit fiber networks for high performance. The rules and minimum requirements for being part of the network were defined with specifications:
 
  • Hardware components (Wires, Switches, Racks, Servers etc.)
  • Configuration (Security, Firewalls, OS, WINS, DNS, DHCP)
  • Centrally managed Antivirus and User Administration
  • Central Operations & Maintenance Organization
  • Operational Procedures and User Guides
This is a validated network that went operational April, 2011 and includes KPI dashboards that provide information including system availability, system changes status, incident status, and other system information all on one screen.
 
The next steps include central governance of peripheral devices (bar code readers, label printers, etc.) and applications that are to be allowed on the network, and central back-up / restore services. Another task is documentation to the point that network administration could be outsourced to another service supplier.  
 
Issues Experienced
 
These are issues noted in the presentation based on their first-hand experience.
 
  • IT Quality level varies greatly by site location and from the pool people available for hire to do these services.
  • In-house IT team lack of experience and competencies in technical skills and understanding for automation validation and overall program controls.
  • Company viewpoint was that this was an expensive program for something that was already working. Now that this is in place the real cost savings are now being made visible and tangible.
It was noted that they expected to get resistance from local sites, but so far the sites have been cooperative. They all want convergence with central services and the harmonized service level. Central administration takes a lot of burden off of the local site resources.
 
PAR Roundtable Comments
 
These are comments from other participating PAR companies discussing their experiences. This is valuable interaction and a key business value of the PAR meetings.
 
“You need skilled people on local sites to deal with problems and retain site experience.”
 
“We cannot get IT people to support automation systems due to the organizational split between IT and automation. Anyone who has IT in their title has to work in an IT organization in our company. That skillset is now being required to maintain automation.”
 
“We have not seen a need to centralize plant control networks onto the corporate network. If anything we are doing more control system segmentation.”
 
“Cyber security initiatives have brought into management focus the need to keep IT involved with automation. There is a specific skill set that is not automation engineering but system management that has to be embedded and co-located with automation groups for efficiency. This can be ‘federated’ and shared with other sites.” (Editor’s note: There was a discussion about this comment by someone that tried this approach and the people were so committed to the local site they could not put time in as a shared federated resource pool.)
 
“We have two IT organizations - corporate IT and manufacturing IT - but with manufacturing initiatives we need a little different knowledge and skillset in the beginning to ‘kick start’ the program.   Initially you want a local IT person supporting site startups.”
“We have recently centralized moving from local to one global standard. This was a two year effort to accomplish this. Enterprise LAN comes into the site through an IPS (Intrusion Protection System) device." (IPS reference: http://en.wikipedia.org/wiki/Intrusion_prevention_system)  "Manufacturing network environment, IT applications and device support is done centrally.  Control system network layers are not directly connected to the manufacturing LAN. Using dual NIC servers for linkages between control systems and IT managed networks. This is working very well.”
“Trying to leverage IT group’s expertise as much as possible to support switches, storage, firewalls, and hardware servers. A problem is their lack of skills in control systems and validation.”
“We are taking HMI to thin clients on the corporate networks which remove an administration burden from automation.”
“We have a hybrid model for central governance and architecture with site support and operations more federated. Local site manufacturing automation IT is dedicated for automation and manufacturing.  A couple of years ago they went on ‘an IT outsourcing binge’ and purposely shielded manufacturing sites from outsourcing manufacturing IT support keeping  in-house people. This saved them from having major problems. The business systems that were outsourced experienced all sorts of outages, …from ERP to anything you can imagine…, but the manufacturing network stayed solid because they had dedicated full time local employees looking after it, who knew the details.” 
 
“There are centers of excellence that define peripherals (label printers, bar code scanners, etc.) so there are standards for MES and production.”
 
“We have a mix of sites with some that have totally separate automation networks maintained by the site automation people and those sites that leverage the IT department to work together to cover support. A lot of IT resources are going away and outside contractors are being used more which is starting to create problems." (Editor’s Note: An example was given of outside contractors working in their corporate office that did not follow SOP (Standard Operating Procedures) who patched a domain controller and shut down all production systems that relied on network authentication. Pretty much anything critical was knocked out.)
 
Thoughts & Observations
 
The integration of enterprise IT systems with production to improve operations and efficiency is becoming a requirement to compete. This group of end-users is taking a logical approach to working with IT groups to leverage their expertise and knowhow of computing and networking technology. One comment by a PAR member is a good definition of the goal, “Working with IT, while safeguarding the automation systems.”
 
Based on numerous comments, it seems clear that there is a need for a support level that bridges automation and IT, which is a hybrid skillset. At this time, the combination of IT skills, process knowledge and automation is hard to outsource without risk.
 
Outsourced IT that created problems for sites seems to be a large area of potential risk (downtime, production disruption, etc.).
 
Successful programs have a close collaboration between IT and site automation people, which is at best difficult to achieve using outside or outsourced IT contractors.
 
Your thoughts and comments are welcomed.
 
Links to other articles in this series:
 
Part 3: Automation Usage of Network Technology (You are currently reading this article)

Did you enjoy this great article?

Check out our free e-newsletters to read more great articles..

Subscribe