Results from Automation Benchmarking Survey

  • April 28, 2014
  • Feature

Perspectives from Pharmaceutical Automation Leaders – PAR Article Series, Part 2

By Bill Lydon, Editor

Pharmaceutical automation leaders from around the world gathered for the annual Pharmaceutical Automation Roundtable (PAR) in Copenhagen, Denmark, to discuss a number of automation challenges facing their companies. While the context was specific to the pharmaceutical industry, these challenges are certainly applicable to all industry segments. The topic of this article, based on presentations and follow-up conversations among participants, is the results of the annual PAR Benchmarking Survey.

PAR Benchmarking Survey

Before the annual meeting, Dave Adler ( conducted a PAR Benchmarking Survey. The survey requested feedback from members on a number of automation-related topics. He presented the results of that survey during the roundtable meeting. Seventeen companies participated in the survey. Sixty percent of respondents were from North America, 32% from Europe, and 8% from other parts of the world. Respondents held various responsibilities within their companies. Twenty percent were in packaging, 18% in pharmaceutical aseptic, 15% in pharmaceutical solids, 21% in bulk API, and 26% in biotech API.

Big Data, Data Capture & Utilization

The survey asked about utilization of data technologies in production facilities:

  • Data historian (automation system storage)
  • Information integration (linking systems)
  • Data warehousing (central storage from several systems)
  • Data aggregation (summarizing)
  • Data visualization (using schematics)
  • Data mining (finding patterns)
  • Big data (increased velocity and volume)

The two most adopted technologies are data historians followed by information integration and linking systems. All of the others have some adoption or are in pilot testing. The exception is big data, where respondents answered undecided, unsure, or don’t know.

Comments during the PAR meeting discussion:

“We have an informatics group that is more likely to apply this type of technology, but they are not getting automation involved. They only focus on quality, so it is not a true aggregation or full analysis.”

“Our people in the supply chain are looking at a lot of information, mainly financial and ERP. They are probably missing the big picture of all the flexibility we could accomplish. We could increase and decrease production to become more demand-driven rather than reactive.”

“Some of this is about the health of your process, lean activities and continuous improvement.”

The survey then asked how the data is used. The top three frequent uses are: review process and batch performance, resolve quality issues, and as part of batch acceptance. Least used are: production and management reports, lab sample results, environmental lab results, data on raw material, information to optimize manufacturing processes, and input for production planning.

The survey then asked about major issues with the use of these emerging data technologies. The most frequently cited issues are:

  1. The operations staff doesn’t understand the benefits of these technologies, so they do not include them in requirements.
  2. These technologies have a lack of skilled professionals to design, install, and maintain the applications.
  3. The purchase price and installation costs for these technologies are still too high to justify for new and existing facilities.
  4. These technologies are not needed to meet the operational requirements of the facility.

Comments during the PAR meeting discussion:

“Operations base their needs on experience and what has worked well for them in the past. They don’t want new toys. They just want to do it the way they have always done it.”

“There are situations where people want systems such as OEE, visualization and other. If it is not required for the machine and process, it is not viewed as business-critical and therefore is not added.”

“We had an area where the packaging group wanted to calculate and report OEE. The group told us they were capable of using a system. We installed one to track downtime and other parameters to calculate OEE. After the system was installed, they did not know how to use the system. They hired a full-time contractor to sit with them in their morning meetings and walk them through the data. In contrast, the identical system was installed at another plant and used to increase throughput and decrease downtime. It is all a matter of what specific groups are capable of using.”

“The change only occurs when the fear of change is less than the fear of survival. When your site is in trouble and you have been told to cut your costs or you are gone, then you will get a cultural change.”

“It’s not painful for the senior managers or the supply VPs. It is not enough to motivate them to invest in these systems, because they are time-consuming and risky.”

Data Utilization

The survey asked if the review of data is a formal process. Fifty-nine percent answered yes, 36% answered infrequently when required by procedures, and 5% answered no. The survey asked who looks at the information generated the most. The most likely groups are operations/manufacturing, followed by automation, technical services, engineering, quality, maintenance, and IT.

Comments during the PAR meeting discussion:

“I would hazard a guess that the automation people are looking at the data because operations is asking them to do it for them.”

“There is a key difference between what we view as the automation group’s responsibility and what others view as automation group’s responsibility. We feel that automation has a number one goal of process optimization. I’m not sure other groups view it is that way.”

Return on Investment

The survey asked about approximate return on investment estimates for data technologies used in an automation system. The average responses were between 2 to 3 years. The systems with the highest return on investment were virtualization and data historians. Vision systems were much lower.

Comments during the PAR meeting discussion:

“It would be interesting to keep track of the vision system’s return on investment over the next few years. Our most recent projects have one- to three-month payback. The cost of vision equipment and the associated computer equipment has dropped significantly.”

“You can virtualize a full development system in a computer system for about $2,000. Quotes from our vendors for similar systems cost $75,000-$100,000. We’ve implemented 10 systems in the last six months and we do them ourselves.”

“Virtualization is a surprisingly stable and easy to apply technology.”

Last year’s survey asked how often automation projects achieved ROI goals. Eighty-four percent of automation leaders believed the return on investment goals were achieved. Operations leaders believed they were achieved 63% of the time.

Major Innovations

Respondents were asked about their opinion on what will be the one major innovation in data technologies to impact automation in the next five years. The most frequent answer was mobile.

Comments during the PAR meeting discussion:

“One of our sites purchased iPhones for all of their maintenance staff. With the iPhones, they have full access to all SOP's (Standard Operating Procedures), drawings, and other documentation. While performing preventive maintenance, they take pictures and attach them to information sent to the CMMS (Computer Maintenance Management System). Calibrations can also be accomplished with one person rather than two.”

“We have a plant that went paperless and gave everyone iPads.”

“Cyber security is a concern. Management does not want to use wireless because of the cyber security threat.”

“It is trendy and a bit of hype, but I am not sure how useful it will it be 5 years from now.”

“In our plant, the operations group drives us. We installed MES systems and removed as much paper as possible from the floor. Now that we use mobile, they are starting to look at the data. It might be that this technology enabled people to look at the data more than they have in the past.”

Automation Organizational Structure and Staffing Levels

The survey asked if there is a dedicated automation function that manages process automation personnel. Sixty percent answered, “Yes, at global headquarters and locally.” Twenty-eight percent responded at the local level only. With respect to projects, 50% of the respondents indicated the local site group had responsibility. Twenty-three percent indicated central corporate group responsibility. Seventy-seven percent responded that long-term support was the responsibility of the local site group. Nine percent said that responsibility lie with the central corporate group.

When asked how automation staff is justified, 55% responded that justification is based on day-to-day support for local needs. Twenty percent said it is based on unit operations; 17% on business return on investment from process improvements; and 8% on capital projects.

Comments during the PAR meeting discussion:

“The biggest management frustration is that we are not making things better. What they really want from us are process improvements, but we are justifying our staff on day-to-day support needs. So, how do they expect us to deliver improvements if they are only paying for support?”

In-house & Contract Automation Professionals

The survey asked about the percentage of in-house and contract automation professionals. Sixty-three percent of respondents indicated they use in-house staff, and 37% use contract staff. On projects, more contractors are used and in-house staff is primarily used for long-term support.

Dedicated IT

When asked if there is a dedicated IT function as part of manufacturing systems, 33% indicated seldom, 28% most, 28% frequent, and 12% never. The most frequent use of centralized IT support staff is for automation infrastructure in production facilities for networks, storage, and servers. One major issue with using centralized IT infrastructure support is that the central IT department doesn’t have enough skilled personnel. A second issue is that the automation professionals do not trust the personnel in the IT department. A third issue frequently cited was that the conversion to centralized IT support doesn’t have positive business case.

Comments during the PAR meeting discussion:

“The fear of operations is the response time of IT support. If they contact the typical IT support group, they have to wait for hours. In contrast, the automation person will work on the problem immediately. However, using an automation person to do IT support may not be the best use of resources.”

Staff Utilization

The survey asked where the automation staff applies responses. Forty-one percent said day-to-day support, 32% said project implementation, 15% said system administration & infrastructure structures support, and 12% said optimization.

Comments during the PAR meeting discussion:

“How much time should be spent on optimization? Daily support is very important; maybe 25% should be spent on optimization.”

“Project implementation could include optimization.”

“It is staggering how inefficient or immature some of the equipment is compared to buying a car that can run for 30,000 miles without an oil change. Packaging lines or filling lines stop all the time.”

“Because the processes are automated, the operators do not need to understand the process as much. When issues occur, they really don’t know how to diagnose it, and they are reliant on the automation people to handle it. It is the same with process engineers and anyone else on the process lines - they do not understand the automation component of the system.”

“We shouldn’t have to support investigations, pull trends, and batch data.”

Automation Management

Thirty-nine percent of respondents said they are reorganizing the management structure of automation. Looking further into the data, 47% of the companies are reorganizing. When asked about the top characteristic of an ideal organizational structure, the most frequent response was being aligned with operations and focused on process improvements.

Comments during the PAR meeting discussion:

“Many of our sites have decided to bring in the operation support of LIMS, SAP, and MES into a centralized group that reports to operations. Then they put automation in there as well. The idea is that they can educate across functional groups.”

“At our company, corporate IT has changed their name four times in three years.”

Level of Automation

Sixty-seven percent of senior operations leaders believe the level of automation in their facilities is too low. The majority (70%) of operations managers believe the value of automation is high value. Senior management respondents generally see a high value in automation and a little better than moderate value for automation personnel. Operations management’s biggest concern about automation is that it is too expensive. When asked what the automation discipline should focus on in the next 5 to 10 years, they answered reducing automation costs.

Comments during PAR meeting discussion:

“Because we are regulated, we often use the excuse that we don’t need to be concerned with return on investment.”

Operations management responded that the major values provided by automation are to improve product quality (70%) and to improve safety (70%).

Automation opinion leaders asked specifically what automation has done to improve operations quality. The largest response was it reduced operator error and process variation from batch control. The survey asked about what types of automation are implemented to prove operation safety. The top two responses were interlocks and automatic control of abnormal situations.

Automation Changes

The survey asked what is the relative time and effort for automation changes versus other changes in manufacturing. Respondents believe that changes to automation were comparable to other changes. When automation changes take longer, the leading reason is the heightened QC focus that requires more documentation for software changes. The second reason is availability of equipment, and third is the associated time required for testing and qualification.

Comments during the PAR meeting discussion:

“I had a conversation with a QC person a couple years ago. They were looking for completely outrageous stuff (documentation) regarding the automation system. My point to them was you have to rely on the subject matter experts. You are not going to ask an electrician to explain to you how panel wiring all works. Why you do that for automation? I partially won that battle.”

“I was working on a project with class-based control modules and 1,000 alarms. QA asked for a document that showed all 1,000 alarms that occurred. I told them I wasn’t going to give them a list so they could compare the setpoints and determine what is in the system. The QA guy called an eCompliance guy who challenged me for two hours about the benefits of class-based and how it works. Finally, he accepted a comparison list as being fine. If you don’t defend class-based design, you are opening the doors.”

When asked about unforeseen startup problems, 68% indicated they seldom have them. Thirty-two percent indicated that problems are frequent or most often occur.

Comments during the PAR meeting discussion:

“Our problems are seen in the performance of the systems, because we tend to forget that in our design.”

“A purely class-based DCS site is very solid. Another option is custom code everywhere with indirect addressing, loops, and everything resulting in a lot of errors.”

“Over the last 2-3 years, when we started an automationonly change, it wasn’t thoroughly tested or implemented correctly. When you started things up, it was in the wrong mode or other problems.”

“We are seeing more emphasis on checking the data between the systems. When making a change on one system, the data is sent out differently and stops a third system. Default values are always a problem.”

Changes to sites have an impact on automation 66% of the time. The majority (75%) of changes were not initiated by automation.

Comments during the PAR meeting discussion:

“Automation can be impacted by process changes owned by the process engineers. You end up getting lumped in with the change control and chasing it. I would suggest you shoot it off to the process guy and tell them this is their change. This is my part in it and you deal with the others.”

“A good number of those changes are infrastructure, version control, and patches.”

The top reasons for change are to increase production capacity, improve product quality, reduce manufacturing costs, and improve regulatory compliance.

Survey Impact

All of the respondents received a report on the results of the survey. The question was posed, “Did you make any changes in automation based on the results from last year’s survey?” Major changes were made by 2% of the respondents and minor changes by 19%. Below are a few changes based on the results from last year’s survey.

  • An automation representative was added to the site’s Process Data Informatics team.
  • Better on-boarding approach.
  • Survey results were discussed with management and automation associates. The awareness of survey information impacted job rotations within engineering, IT, and automation.
  • Express benefits of automation changes in business terms.

Improving Automation Brand

The question was asked, “What does automation leadership need to do to improve the discipline's brand?” These are the responses listed from most frequent to least frequent.

  1. Lead efforts to improve, optimize, and innovate. Be a change agent.
  2. Communicate the value and benefits of automation.
  3. Align, partner and connect with other disciplines and leadership.
  4. Retain automation professionals and develop their technical skills.
  5. Establish a clear strategy.
  6. Become process, business, and operations oriented.
  7. Increased focus on data availability and analysis.

Comments during the PAR meeting discussion:

“We are not getting enough cross pollination among automation, process engineering, tech service, and leadership.”

“Everything on that list is what we need to do.”

About the PAR Meetings and this Article Series

Every year, I have the opportunity to attend the Pharmaceutical Automation Roundtable (PAR) meetings as the only outside observer. Last year’s meetings were held in September of 2013 at Novo Nordisk A/S at their facilities in Copenhagen, Denmark. Lead automation engineers from around the world attended this invitation-only, two-day event. This group of engineers has a wealth of practical knowledge and knowhow and is willing to share with other participants - truly learning from each other. The PAR meetings represent a very knowledgeable group of automation professionals gathered in one place at any one time to discuss automation issues. This year, the participating companies included Amgen, Biogen, Idec, J&J, Eli Lilly, NNE, Novartis, Novo Nordisk, Pfizer, Sanofi-Aventis. The PAR meetings consist of various presentations given by PAR members on specific automation topics. Other members then provide comments about their experience, ideas, and challenges relating to the topics. This article series presents a summary of those conversations, with each article highlighting one or more of the topics covered by the PAR meetings. Comments by specific PAR members are reported anonymously.

About PAR

PAR was founded about 15 years ago by Dave Adler and John Krenzke, both with Eli Lilly and Company. At the time, the purpose of the roundtable was to provide a means of benchmarking and sharing best practices for automation groups among peer pharmaceutical companies. The group specifically does not discuss confidential or proprietary information, cost or price of products, price or other terms of supply contracts, plans to do business or not do business with specific suppliers, contractors, or other companies.

Did you enjoy this great article?

Check out our free e-newsletters to read more great articles..