- March 17, 2014
By Bill Lydon, Editor
Pharmaceutical automation leaders from around the world gathered to discuss a number of automation challenges, including how their companies are dealing with reduced personnel and the need to support multiple technologies at multiple sites. This article highlights those conversations and addresses the automation support models and the use of managed services.
Perspectives from Pharmaceutical Automation Leaders – PAR Article Series, Part 1
By Bill Lydon, Editor
Pharmaceutical automation leaders from around the world gathered for the annual Pharmaceutical Automation Roundtable (PAR) in Copenhagen, Denmark to discuss a number of automation challenges facing their companies. While the context was specific to the pharmaceutical industry, these challenges are applicable to all industry segments. The topic of this article, based on presentations and follow-up conversations among participants, is automation support models and the use of managed services.
One PAR participant presented his challenge by first describing the situation at his company. The company has a small group of people with a select skill set that is expected to support multiple technologies. Those headcount constraints coupled with cost presents a challenge in the current business climate. The company needs to provide complete 24x7x365 operations support. The support resources must be flexible to deal with new products and a multi-product environment that is continually adding more products to the production mix. In addition, they constantly need to be ready to implement new technology to improve efficiency and standardization in the business.
The presentation explored managed services, a concept that has been around for many years in the IT industry and is not really used in industrial automation. Managed services are a way to transfer day-to-day management activities to a service provider in an effort to improve and create an effective operations environment. The service provider remains accountable for the functionality and performance of managed services but does not assume overall management responsibility of the organization or systems. The goal is to reduce the administrative time spent on the existing automation and IT, therefore allowing employees to concentrate on the core business.
The company recognized that personnel were being consumed by support responsibilities and were not available to implement more strategic projects. This situation created a vicious circle where they ended up hiring a contract resource to do the new investment project or strategic project. The result was the contractor had the application knowledge, and internal staff later had to pick up the pieces on the site and learn the application. Their goal was to change the model so that day-to-day tasks were offloaded from internal staff, and managed services was the solution. Under a managed service agreement, the vendor takes on full responsibility for assigned activities. These services are fixed-fee based on defined scope and agreed norms.
Managed services provide the flexibility to scale up and down personnel resources to meet business requirements with a known contracted resource. The personnel risk is transferred to the service provider, and the goal of this partnership is maintaining shared system knowledge. The contracted company generally assigns a site leader to manage the scope of service and align activities. Valued, strong partners are given a preference on capital projects, and they offer agreed rates for tasks.
Scope of Service
The presenter’s company started this effort about 5 years ago and broke out services into various areas. System administration is a key outsourced function. Based on his experience, the presenter noted that system administration gets done as long as there is nothing else to do. And yet, when it’s not done, this is the one that is going to come back and bite you.
They define the scope of outsourced tasks by listing all the activities required along with frequency. The tasks and requirements are agreed upon with the service provider. For all activities, the service provider submits detailed reports confirming the work was performed. “People think that we are just buying heads with managed services, but we equated it to the cleaning contract in the building,” said the presenter. “If you want all the windows in the downstairs offices cleaned once a month, it doesn’t matter if they bring one guy on a Monday, and it takes him all month, or if they bring 20 guys on the last day of the month and do all windows at once. This is how we present it to management.”
Complex changes are kept in-house using internal subject matter experts and engineers. How much support the plant needs is hard to actually measure, so they rely on historic information to help plot out the support tasks. They also specify what type of talent is required for tasks, ex. junior engineer, senior engineer, etc.
Over the last few years, they added systems including data historians, MES, LIMS and DMS, that required a great deal of support. When issues and problems occur, automation engineers must decide from where to get support, ex. IT, quality systems, infrastructure, or a vendor.
Troubleshooting interfaces is a new skill set that we have developed over the last 2-3 years.
Here are some of the PAR participant answers and comments to questions posed at the roundtable. Each paragraph below the questions represents a response from an individual.
Do you leverage outsourcing for your site support model, and if yes, what percentage split between direct/indirect labor?
Support is done locally below the MES level. The trend is to centrally manage systems at the MES level and above. If possible, we outsource to vendors.
We typically outsource capital projects and assign a staff person to be in charge of that activity. Typically, we outsource 10-20% of our administrative and production support. Level 1 and 2 front-line support is done with contractors. If there is a production issue, contract resources diagnose the issue, and 9 times out of 10 they can resolve it. If there is a change required to improve system reliability, that activity is performed by staff resources.
Capital projects are all in-sourced. Before we started outsourcing, our 24x7 support was accomplished by internal staff, which negatively impacted projects. Also, no one enjoys performing 24x7 support, so when we forced our internal staff to do it, we took the risk of losing those people.
Managed services can deplete the internal knowledge.
We performed major outsourcing of site IT infrastructure and support. No one has technical knowledge at sites. The entire model was justified based on IT parameters. Automation is now under engineering for the most part.
Our sites are now comparing the cost of outsourced and in-house support. They are finding lower cost using in-house resources. By outsourcing, we are giving all the knowledge-gaining opportunities to someone else; we are not learning. There is no motivation for the outsourced partner to reduce cost. In-house support provides value because smart engineers are learning to understand the data and can make process improvements. The other issue we have is a Vice President will react to a problem and say, "We can’t wait; we have to do this right away," and that shifts priorities. You can’t say no, but we need a culture where you can say, "Good idea, but we have to put it into the planning process." We are trying to accomplish quarterly reviews of priorities to ask ourselves, “Did we meet plan?” If not, why and what can we learn that will improve planning?
We outsource some engineering and maintenance - about 50%, and up to 80% for engineering when we have large capital projects.
We use agreements that have a fixed-cost component and a variable-cost component.
We are heavily outsourced. At larger sites we have a small team to coordinate projects.
How is Level 1 and 2 support handled on your sites?
The first level support is local, and we use some pre-identified “super users.” MES and above is owned centrally and support is typically outsourced.
It is important that the people who are supporting the lower levels hand off problems identified as level 1 and 2.
Some sites are staffed 24/7, and others are supported with on-call internal staff. Other sites have automation people embed in the operations group.
Level 1 and 2 systems are locally owned and operated. Plants in some geographic areas have no capable outside resource in the geographic area, so we employ talent locally. In other areas, we have good partnerships with capable system integrators.
MES and data historians are global systems that are locally implemented.
We are now building some plants in China with standalone automation - no MES, all paper, and practically no IT.
We are going to total outsourcing of our site’s IT based on a CIO cost saving analysis. The service level agreement defines 4-hour response to problems. With this outsourcing model, there is no longer IT knowledge at sites.
Do other departments have a troubleshooting or (Subject Matter Experts) SME capability from an automation perspective within their functions?
It is very important to have local SMEs to be the first to identify where is the problem resides. We need local people that can get the problem fixed as quickly as possible.
This varies by site. At our company, automation is part of the IS group, not part of engineering. Any IT systems also fall under the IS group.
For the first two months after a new project is up and running, we have “hyper care” SMEs from the project team, then they back off.
We have maybe one SME per site.
Are your business priorities aligned with the automation department objectives? Is there a defined process in place for daily, weekly, monthly, etc. priorities?
We did an analysis of why some of our automation projects are delayed significantly, and we found a shifting of resources between operations and automation projects.
We typically take our corporate goals and objectives and cascade them down to departmental objectives and site objectives. In general, that gets to about 75% aligned with business goals.
At a site, priorities change on a daily basis. For example, at a production meeting, a senior manager makes a comment that they want higher reliability on a process, and they just created two years of work for the automation group.
If we don’t show where we add value, our budgets go down.
We have built a three-year automation look-ahead process at each site for our business leaders.
We are trying to move to a more business-led, manufacturing automation strategy based on 3-5 year business plans.
Over the last couple of years, we have implemented technology maps to align with the business, and they cover automation-specific projects and installing new process equipment. We’ve seen a mixed level of success. The problem is the technology maps are not “apparent enough.” People look at them once a year and don’t refer to them as often as they should. We want these to become part of the culture.
How do you justify the cost of automation support on your sites (per IO, per number of changes, etc.) and does the cost of this support sit within automation budgets or operations budgets?
The site is responsible for the costs since they have ownership, and it is hidden in other budgets. Because of this approach, it appears support never costs anything when compared with doing it centrally.
The support budget varies by site - about 50% covered by site and 50% covered by operations.
A few years ago, we did a benchmarking on maintenance to determine the correct staffing.
We have a central engineering group, and management felt that the costs were not apparent, so we have to compete with outside resources.
About the PAR Meetings and this Article Series
Every year, I have the opportunity to attend the Pharmaceutical Automation Roundtable (PAR) meetings as the only outside observer. Last year’s meetings were held in September of 2013 at Novo Nordisk A/S at their facilities in Copenhagen, Denmark. Lead automation engineers from around the world attended this invitation-only, two-day event. This group of engineers has a wealth of practical knowledge and knowhow and is willing to share with other participants - truly learning from each other. The PAR meetings represent a knowledgeable group of automation professionals gathered in one place at one time to discuss automation issues. This year, the participating companies included Amgen, Biogen, Idec, J&J, Eli Lilly, NNE, Novartis, Novo Nordisk, Pfizer, Sanofi-Aventis. The PAR meetings consist of various presentations given by PAR members on specific automation topics. Other members then provide comments about their experience, ideas, and challenges relating to the topics. This article series presents a summary of those conversations, with each article highlighting one or more of the topics covered by the PAR meetings. Comments by specific PAR members are reported anonymously.
PAR was founded about 15 years ago by Dave Adler and John Krenzke, both with Eli Lilly and Company. At the time, the purpose of the roundtable was to provide a means of benchmarking and sharing best practices for automation groups among peer pharmaceutical companies. The group specifically does not discuss confidential or proprietary information, cost or price of products, price or other terms of supply contracts, or plans to do business or not do business with specific suppliers, contractors, or other companies.
Did you enjoy this great article?
Check out our free e-newsletters to read more great articles..Subscribe