Lifecycle Maintenance of Documention for Automation Systems |

Lifecycle Maintenance of Documention for Automation Systems

May 222012
Lifecycle Maintenance of Documention for Automation Systems
What to maintain and how long to keep it? Exclusive – Part 4 from Pharmaceutical Automation Roundtable (PAR) 2011
May 2012
By Bill Lydon – Editor
This article on maintaining documentation for the life cycle of installed automation and control systems in pharmaceutical plants is the fourth article in a series covering the recent annual Pharmaceutical Automation Roundtable (PAR).
About PAR
I had the privilege of attending the Pharmaceutical Automation Roundtable as an observer in November 2011. This PAR was hosted by Johnson & Johnson in Spring House, PA, with Dave Stauffer, Terry Murphy, and Joel Hanson of Johnson & Johnson participating.
Lead automation engineers from various parts of the world attended the invitation-only, two-day event. This is the most knowledgeable group of automation professionals gathered in one place at any one time focused on discussing automation issues.  A range of companies participated including Abbott, Amgen, Biogen Idec, BMS, Genentech, Genzyme, Glaxo, Imclone, Johnson & Johnson, Eli Lilly, Lonza, NNE Pharmaplan, Novo Nordisk, Pfizer, and Sanofi-Aventis.
The PAR was founded about 15 years ago by Dave Adler and John Krenzke, both with Eli Lilly and Company at the time, as a means of benchmarking and sharing best practices for automation groups among peer pharmaceutical companies. The group specifically does not discuss confidential or proprietary information, cost or price of products, price or other terms of supply contracts, or plans to do business or not do business with specific suppliers, contractors, or other companies.
The individual PAR group members have a wealth of practical knowledge and knowhow to share with other participants, truly learning from each other.
Topics are agreed upon prior to the meeting and a member with make a presentation on their organizations views and approach to the topic. After this presentation others comment on their organizations situation.
Lifecycle Maintenance of Control Systems
The PAR presenter discussed the significant amount of details created, defined, and communicated during the construction of a facility resulting in a large amount of information and documentation. Some of this information should be kept current throughout the facility’s lifecycle which could be a long time (i.e.: 30+ years). The question is how much of the information created should be maintained as accurate during the life cycle of the automation and control system?
Reasons for maintaining the information include product safety, personnel safety, facility safety, and regulatory requirements (FDA, EPA, OSHA, etc.) Having the appropriate documentation can also save time and money when doing upgrades and changing products produced in a facility which is happening ever more frequently.
Various groups create and maintain information including process flow diagrams, equipment flow diagrams, piping and instrument diagrams, and facility design. Information from the automation and instrumentation group includes automation system hardware design, logical design, systems architecture, network design details, instrument specifications, instrument installation details ( i.e. mounting), instrument wiring details (including packaged equipment), hardwired interlock and safety circuit details, power wiring details, and cabinet layouts. The automation software design is another major amount of information including functional requirements/specifications, software interlocks, control loops, equipment and control modules, batch recipes & phase designs, software design details, and controller application code. 
Information can be presented in many ways including manual drawings, CAD drawings, electronic models, commercial engineering software formats, un-contextualized text (Word documents), contextualized text (e.g. custom databases).
Issues & Experience Discussion
The following items are based on the discussion by PAR members on this topic. Defining the value of investments in creating and maintaining documentation over the lifecycle is becoming clearer. Accurate data can save money on audits, maintenance, renovations, modernizations, multi-product production conversions, optimizations, troubleshooting, problem investigations, and disaster recovery.
Document Management
The experience of most participants illustrates that efficient commercial software offerings do not exist to meet their documentation requirements.  The documentation of legacy operations is difficult since much of the information is not in digital form and in many companies the documentation over time on these sites is not kept up to date. However, when upgrades and retrofits are done on existing sites, that documentation is done. 
Participants had various methods for document management as illustrated by these participant comments:
We are decentralized and there are a variety of documentation methods including Word documents, spreadsheets, and CAD drawings.”
“Maintaining the information is costly with drafting people, technical writers, plus the associated training and support of document systems (i.e. CAD; databases).”
“We hand over the project package and leave it to site operations to maintain documentation.”
“Historically information has been in drawings and specification documents. In the mid 1980’s we decided keeping so much information such as instrumentation and wiring data on drawings was cumbersome so it was then stored in a database along with drawings of typical configurations and this has worked very well. After a change in our corporate computing systems to PCs our engineers lost easy access to this database and for about 10 years we really did not have access to a central database. All of the engineering facilities developed their own solutions using spreadsheets, drawings, and other ad hoc approaches. Recently for loop drawings in particular we are trying to organize the information using data models of wiring inside CAD models. Unexpectedly we are finding these different types of presentation methodologies have a huge bearing in the cost of using and maintaining these systems. For example, Word documents can be put together quickly but are hard for finding information. A custom database is good but is expensive to develop and maintain. How you capture and present the data can be done in many ways but each has its own cost/benefit ratio.”
“Tried one of the commercial packages that has a proprietary instrument database and CAD engine to produce drawings plus the wiring details and that became very cumbersome and costly.   The company was difficult to work with especially with upgrades, and licensing so we did exports of the instrument information to maintain a spreadsheet database and exported the CAD drawings.”
“Some use VISIO rather than full CAD for simplicity that are checked into the document management system. If you rely on CAD drawings you need a responsive CAD system and department.”
Editor’s note:It was noted that most engineering firms have created their own software and some users have done the same.   One user commented they are finding custom created databases are much more efficient for documentation.
“Interesting that one company uses the EPR system for change control and the system generates tasks that engineers work through, that guide the documentation for automation system changes.”
“Drawings from all global sites have been collected into one database and it is possible to view the drawings over the Intranet. Existing paper drawings have been scanned so they are available electronically.”
Packaged/Skid Equipment
Packaged and skid equipment continues to be a challenge, including documentation. It seems a major issue is getting this equipment specified properly and then enforcing the specifications in the purchase process. 
PAR Participant comments:
“Some packaged equipment we treat as a black box, there are others we require complete documentation. We do a vendor assessment, rating how well they engineer their products. If we don’t feel the vendor has good documentation and processes in place, we then need to retest it.”
“The decision to buy packaged equipment is not generally driven by automation so we generally need to “fix” the automation after we have the equipment on site and create appropriate documentation.”
“The capital equipment purchasers don’t factor in the lifecycle cost of automation, just initial cost. Many times the automation and controls on the packaged equipment and skids need to be redone to integrate and fit into plant operations.”
“It may be smarter to buy just packaged equipment with minimal or dumb controls and then automate later rather than reworking the skid/package controls and creating new documentation. These packaged and skid equipment vendors are not out to help make you or your process run more efficiently… but to sell their equipment.”
“At our company this has swung both ways and now the capital delivery groups now ask us to quantify the life cycle cost impact of automation from different equipment providers. The best situation is when they ask us what our preference is and how much is that worth to the project and to the site.”
"Another game experienced is when the packaged and skid equipment vendor tells purchasing,'The controls are free.' The problem with equipment purchased this way is you need to add and document controls to make it work properly in your application. It might be better to throw the “free” software away and just start from scratch.”
“We did a project with 20 pieces of packaged equipment and we did an analysis for each one. There were a couple that were fine, others needed a complete controls retrofit. It depended on how integrated the equipment needs to be with the plant, how many future changes, how much of the data needed to move and integrate with the overall system, and how comfortable we were with the engineers and their capability from the vendor.”
“We have created specifications for packaged equipment including interfaces, instrumentation, HMI, device networks, and data requirements. The data interfaces are really important so that we get the right information for OEE and other purposes.”
“Validate all of our projects including vendor provided skids and subsystem equipment including components and wiring. We have picked up problems with skids through this rigorous process.”
Standards & Policies
These comments from participants will give an idea of current operating standards and policies. It makes sense that using plants for multiple products, flexible manufacturing, requires accurate up to date documentation. 
PAR participant comments:
“We now have a wealth of global engineering standards coming down in addition to governance. We are going to more multiproduct production facilities, so it is becoming important to be able to determine if equipment, automation, controls, and safety systems will work and be safe for new products to be produced.”
“Lifecycle management is done at the sites using the tools provided with common standards and governance.”
“We have system owners at each site and that person needs to know what drawings are associated with that site so if there is a change proposed this person needs to sign off on them. There is no way to replace human knowledge."
"Keeping documentation synchronized with the maintenance management systems takes a great deal of work.”
The discussion on documentation reminded me of the BIM (Building Information Modeling) trend that started in the 1980s in the commercial buildings industry.  BIM is a process involving the generation and management of a digital representation of physical and functional characteristics of a facility. The resulting building information model becomes a shared knowledge resource to support decision-making about a facility from earliest conceptual stages, through design and construction, then through its operational life before its eventual demolition. BIM has taken years to evolve and only recently are information standards starting to come into focus, allowing BIM to be practical. BIM requires vendors to provide detailed information in open digital form, typically XML. Biopharmaceutical plants are far more complex so an effort similar to BIM might be monumental and it is unclear what body or organization would be the focal point for this holistic effort.
It is interesting that the experience of most participants illustrates that efficient commercial software offerings do not exist to meet their documentation requirements.
Your thoughts and comments are welcomed.
Links to other articles in this series:
Part 4: Lifecycle Maintenance of Control Systems (You are currently reading this article)
Did you Enjoy this Article?

Check out our free e-newsletters
to read more great articles.

Subscribe Now
Back to top
Posted in: