- May 06, 2013
By Bill Lydon, Editor
A pharmaceutical company discussed their strategy and tips for system obsolescence planning and presented a specific case study of a DCS replacement at a specific site. Other pharmaceutical companies also shared their perspectives on the topic.
Pharmaceutical Automation Roundtable (PAR) 2012 – Part 6
By Bill Lydon, Editor
This is the sixth article of a series about the annual Pharmaceutical Automation Roundtable (PAR). I attended the Roundtable in November 2012, hosted by AbbVie, which at the time was part of Abbott in North Chicago, Illinois. Lead automation engineers from pharmaceutical companies all around the world participated. The PAR group members have a wealth of practical knowledge and knowhow to share with other participants, truly learning from each other.
PAR was founded about 15 years ago by Dave Adler and John Krenzke, both with Eli Lilly and Company at the time, as a means of benchmarking and sharing best practices for automation groups among peer pharmaceutical companies. The group specifically does not discuss confidential or proprietary information, cost or price of products, price or other terms of supply contracts, plans to do business or not do business with specific suppliers, contractors, or other companies. This is the most knowledgeable group of pharmaceutical automation professionals gathered in one place discussing automation issues. A range of companies participated including AbbVie, Amgen, Biogen Idec, BMS, Genentech, Genzyme, GSK, Eli Lilly, NNE Pharmaplan, Novo Nordisk, and Pfizer.
Topic: System Obsolescence Planning
The presenter described his company’s lifecycle management program for automation systems along with a case study. He started with a review of his company’s overall initiative for lifecycle management of automation systems recognizing the need for a process to deal with obsolescence of components or complete systems in order to do better planning and forecasting of system TCO (Total Cost of Ownership). The overall goal is to have a system life cycle plan for each site. Different lifecycle times were defined for each layer of the system:
Layer 3 – MES, Data Management Systems: 3-5 years (Hardware Only), MES Software (10+ years)
Layer 2 – SCADA, DCS, Batch, Historian, HMI: 5-10 years
Layer 1 – PLC, DCS-Controllers: 10-15 years
Layer 0 – Devices, Field Bus; Wiring: 15-30 years
The presenter discussed an existing site that has been reviewed for DCS replacement. In 1996 this site purchased a DCS and in 1999 began commercial production. A year later the DCS vendor was purchased by another vendor which raised additional concerns. In 2007 the presenter performed an assessment of the system’s longevity based on concern about future support for the system. Risk assessment was used as a primary way to evaluate the situation. In 2010 it was decided to do a system replacement in layers rather than replacement all at one time. The first phase started with replacement of infrastructure including servers, network, and batch manager, HMI, historian, and data storage. Phase 2 (2014 to 2016) is MES replacement and Phase 3 (2014 – 201X) is control layer replacement.
The risk assessment was done by gathering together a total of eight site people from engineering and systems support that were familiar with the system. The perceived drivers for replacement were unavailability of replacement parts, lack of vendor software/hardware support, and observed and anticipated system failures.
The goal was to determine if the system, sub-system and/or component replacement was required within the next 5 years (2012) and then forecast required replacement time. The group ranked and quantified production risk on a system, sub-system and component basis for Severity of Impact (S), Likelihood of Occurrence (O), Probability of Detection (D), and created risk probability numbers (RPNs) = S x O x D. They created a large spreadsheet for major system component types to quantify and estimate how many spares would be required to keep the system running for five years. A financial analysis of failure mode impacts on production loss was also performed to quantify product loss based on severity to better estimate a total financial loss per year.
The conclusion was system and all sub-system components were capable of short term remediation with life extension beyond 2012 (5 YRS) outward to 2015 (8 YRS). Lack of vendor support for downtime/incident root cause identification and correction, compounded by limited site expertise remaining, is the key factor for replacement action beyond 2015.
The presenter noted that this was first time everyone was brought together to discuss and sort through the issues to determine short term remediation to defer total system replacement. There were good technical discussions and risk analysis. The discussions also yielded practices and procedures to minimize software and hardware failures. They also used the development system to force some failure modes they had not seen before to be ready to respond to them. In addition, small projects were created to scope out the pull in of non-vendor approved communications modules to replace some components that could no longer be purchased.
It was decided to do an incremental system replacement to work within operating constraints starting with system software. The production planning change out window was limited to 10 days (2 day cutover; 5 day startup; 3 day cutback if needed). This requires a great deal of pre-deployment system testing that was accomplished by creating a simulator which allows recipes to run and has been used to identify about two hundred issues.
System Obsolescence Planning Pearls of Wisdom
The presenter offered thoughts based on his experience.
System Replacement Drivers
Drivers for system replacement include unavailability of replacement parts, observed failures, lack of vendor software/hardware support for applications and/or operating system, and lack of internal resource software/hardware support due to attrition of resources. The system is incapable of future expansion requirements. System software was the biggest driver for system replacement and controller hardware was the least critical driver.
Limited Upgrade Time
Normally it takes six 6 months to install a new DCS and commission it but typically there are limited and compressed windows of opportunity to upgrade an in-use system since it impacts production. Warm plant cutovers are a reality requiring incremental replacement or parallel replacement strategies to limit downtime.
Suggest future proof system infrastructure designs for ease of obsolescence remediation with more horizontal system redundancy to ease doing vertical replacement slices.
Develop a standard methodology for system obsolescence ranking / replacement planning that is risk based. Keep a database of systems, components, component availability windows, required quantity of spares, etc.
PAR Group Discussion Comments
Individual PAR members from other biopharmaceutical companies injected the following comments on this topic:
“For all assets we have a resource assessment for each site annually that provides a good view of capital requirement over the next 5-10 years. Two years ago automation started adding an assessment to the overall analysis. The analysis is mainly based on new functionality required to get the most out of systems rather than when we cannot run anymore. The software operating systems have become the most frequent driver for installed base change.”
“We have been tracking and upgrading systems and in the last month I was asked to lead an initiative to apply that same level of planning to all systems including building management and LIMS. We now have an upgrade plan.”
“We don’t have a company-wide strategy rather each site manages it separately. It is difficult at the site level because there are other higher priorities for staff to handle.”
“Next year this is in our objectives to have a method and process.”
“Each site develops a risk-based plan every year that is rolled into a corporate program and managed appropriately."
"The automation assets are tracked in a central maintenance management system from high level to components.”
“We seem to justify upgrades based on fear of failure but the actual data does not support that. We have upgraded systems and the process runs much better which is a better reason for an upgrade. If you reengineer processes with lessons learned over the last 10 years they will run better.”
“A big barrier to upgrades is the amount of production downtime - we need to figure out how to replace the tires on the car while it is still moving…”
“The biggest impact is on operations which can take 4-6 months to get proficient with the new system after a replacement.”
“The bottom line…No one is going to pay to get you to where you were yesterday…”
Other Articles in this PAR Series:
- Part 1 - Management’s View of Automation - Impressions of Automation Performance
- Part 2 - Managements View of Automation - Investment Value
- Part 3 - Management's View of Automation – Focus of Improvements
- Part 4 - Integrated Manufacturing Systems
- Part 5 - Real Time Data Integration and Dashboarding
Did you enjoy this great article?
Check out our free e-newsletters to read more great articles..Subscribe