The following discussion is part of an ongoing column called Ask the Automation Pros, authored by Greg McMillan, 2010 ISA Life Achievement Award recipient, industry consultant and author of numerous process control books and articles. Submit your questions to [email protected] with “Ask the Automation Pros” in the subject line. Browse previous questions and answers here. Past Q&A videos are also available on the ISA YouTube channel; you can view the playlist here.
Peter Morgan’s Question
What strategies, practices, hardware and software can be successful in enhancing control system reliability during normal operation and start-ups when the application does not merit or does not have a safety instrumented system (SIS)? For nonsafety systems, where the economic cost of a system failure can be high (but not hazardous), the benefits of improving reliability can be overlooked, or the addition of features to improve reliability could be discouraged to reduce project cost. Please suggest some strategies, including instrumentation, input/output (I/O) and controller design, installation and testing methods, to improve reliability and the process for justifying the investment.
By reliability, we are focused on the avoidance of a malfunction in a continuous or fed-batch control system (measurements, final control elements, I/O, controller hardware and software, configuration, tuning and control scheme) that would cause a significant decrease in process performance (decrease in process efficiency or process capacity). We are not trying to address problems in process or equipment design or issues in batch sequences. I think we could include the challenges during start-ups or transitions.
Hunter Vegas’s Response
I personally find it difficult to answer open-ended questions like this, as I find my answer very much depends on the specific details of a particular application. That said, I’ll try to provide some general guidance on various design decisions that ultimately impact control system reliability and the information I use to drive my selections. (As an aside, some clients have specific design requirements, such as for redundancy. If that is the case, their requirements will typically override any of the options below.)
Controller redundancy: A redundant controller certainly can improve overall system reliability, but is it worth the price of the controller, extra power supply and possible redundancy license? For some systems, a redundant controller might approach or even exceed $50,0000. Key data points I use to drive this decision are:
- Process type: A continuous process that runs months or even years without a shutdown or a process with very long batch cycles (such as that found in pharmaceuticals) will pretty much always justify a redundant controller. Short batch cycles or continuous processes that come down often may not require a second controller, particularly if the process can be interrupted without losing product. However, if a controller loss will likely force the site to throw away an expensive batch, a redundant controller might be justified.
- Online downloads: Some control systems need a redundant controller to make certain online changes without a shutdown. If on-the-fly changes are often required and the process cannot be interrupted, that may justify a redundant controller.
- Hardware reliability: How often have controllers failed? If hardware failures are likely, a redundant system may obviously be a wise choice.
Power redundancy: There are many ways to power a control system. One might have a single AC (non-UPS) source, a single UPS source, dual independent UPS sources, or dual AC sources, one UPS and one non-UPS or another configuration. One might also have single or redundant 24 VDC power supplies. How does one choose? Similar to the controller redundancy question above, certain process types that run very long batches or run a continuous process for months and years will almost always require redundant AC and DC power supplies. Decision drivers for other systems include:
- Redundant 24 VDC power: Given the low cost of 24 VDC power supplies and their tendency to fail, I will almost always go with redundant 24 VDC power supplies running through a diode. I also wire the status contacts in series and connect them to the system. There is no point in having redundant 24 VDC power supplies if you don’t have an alarm when one fails.
- UPS or no UPS? If power is lost or dips with some frequency, a UPS is usually warranted. Continuous processes or long cycle batches will almost always have a UPS.
- Redundant AC power (UPS/non-UPS versus dual UPS): In my experience, UPS systems seem to become IPS systems (interruptible power supplies) more often than not, especially if the UPS is an older design. For that reason, it is better to have a combination of UPS and non-UPS if possible. If 24 VDC is largely used to power the system, a non-UPS can feed one 24 VDC power supply and a UPS can feed the other, allowing the system to continuously operate despite the failure of either AC source. However, if AC is used to power the I/O or motor control center (MCC) motors, using redundant AC feeds can be more difficult to implement.
- Fuses: Regardless of the system, I am a firm believer in individual fuses using blown indicator blocks. This helps to immediately isolate faults and limit the damage. The blown fuse indicator also makes it easy to find, troubleshoot and fix the problem. There is nothing worse than a control system with a single circuit breaker and nothing else. A fault can take hours to locate.
Redundant I/O: Very few systems can justify redundant I/O except for higher integrity safety systems. However, having redundant transmitters for critical applications (such as pH, level, pressure and temperature) can be justified, particularly if the loss of a reading can have significant impacts on quality. For instance, it could be well worth having two and even three pH measurements for critical pH control applications. In cases where redundant transmitters are utilized, always separate the inputs into different cards to keep an I/O card failure from taking out multiple readings.
Process interlocks: The assumption for this question is that the process isn’t hazardous and a safety integrity level (SIL) system is not installed. However, if a failure can generate significant product loss or create extended outages, adding process interlocks to eliminate and/or reduce the probability is easy to financially justify. The resulting interlock may not be a true SIL interlock, but it can be treated like one as far as reliability design and/or testing interval.
Network design: Like the redundant controller, redundant networks may be required and can be justified for many of the same reasons. Some systems have redundant networks built in and present as a matter of course. Unfortunately, many programmable logic controller (PLC)-based systems do not, and adding network redundancy can be difficult and significantly complicate the control system. If that is the case, one has to carefully weigh the pros and cons. In some cases, a PLC may not be the right choice for the control system if you need redundant networks and the PLC cannot implement that without incurring significant expense.
Gateways: Be careful implementing any control system design that relies on a gateway to move data from one part of the system to another. This represents a single source of failure and may not reliably transfer key information like diagnostics or low-level alarms. It can also simply freeze, locking in the data but not throwing an alarm. Be particularly wary of smart MCC applications that rely on a single communication module to move motor and/or drive controls. Depending on how it is implemented, a gateway failure might make it impossible to control or trip the motors at all.
My response really only begins to cover the topic. There are hundreds of design decisions associated with any control system, but the ones I discussed are some of the more common issues that have a significant impact on system reliability.
Erik Cornelsen’s Response
Here are some considerations from a system integration perspective, some of which have already been highlighted by Hunter. They broadly fall into three areas: hardware, software and reliability/maintenance.
Hardware
- Redundant power supply: Use redundant power supply terminals where available to improve system availability.
- Ring topology: Consider implementing a ring topology between the PLC/distributed control system (DCS) and remote I/Os, variable frequency drives (VFDs) and other networked devices, using Rapid Spanning Tree Protocol (RSTP) or similar functionality supported by modern controllers or industrial managed switches.
- Hot-standby PLC/DCS: Consider CPU redundancy for critical processes where the required availability justifies the additional complexity and cost.
Software
- Fail-safe I/O values: Define appropriate fail-safe values in the event of I/O or communication failures.
- Communication heartbeat: Monitor communications with third-party controllers and detect communication loss to prevent stale data from being used.
- Channel failure handling: Configure the channel failure pin, particularly for analog input blocks, and define the appropriate bad value. Depending on the process, this would typically be set to either the minimum or maximum limit.
- Device interlocking: Implement device-level interlocks on actuators based on relevant sensor conditions.
- Sequence interlocks: Implement prestart and hold interlocks to prevent sequences from starting or continuing under unhealthy conditions.
- ISA-88 sequencers: Where possible, use a sequencer aligned with ISA-88 states and transitions (e.g., Idle, Running, Holding, Aborting, Stopping).
Reliability and maintenance
- Preventive maintenance: Maintain regular maintenance schedules to keep equipment in good condition and minimize failures.
- Overall equipment effectiveness (OEE) monitoring: Use OEE to quickly identify the main sources of downtime and prioritize root-cause investigations.
Peter Morgan’s Response
While I posed the question on this topic, I have to own up to implementing more control loops than I can remember with redundant transmitters, redundant I/O and redundant controllers, all for the sake of reducing the impact of a single device failure on the operation of the controlled process.
Corporate and plant-specific reliability requirements should be considered when choosing the control system hardware; this includes process measurements. Adding redundancy increases the cost of an installation, which may be resisted; however, when the impact/cost of a control system component failure is identified, the added cost is often easily justified. Speaking of control systems not requiring the rigors of an SIL determination, the justification can be based on input from operators, process designers and control specialists, using failure statistics from vendors and process impacts such as process upset, reduced product quality and controllability (in manual mode, for example). For those interested in SIL calculation for SISs, my InTech article “Safety Instrumented System SIL Calculation” (January/February 2016) may be of some use.
For process measurements, there are three implementations in common use:
1. Nonredundant transmitter. Single measurement is used for control. Transmitter failure directly impacts the operation of the control loop and has the potential to cause the controlled process to shut down. If the transmitter failure is detected by the operator and the operator successfully intervenes to control the process in manual, automatic control remains unavailable until the transmitter is replaced. While automatic control is not available, any process disturbance could compromise operation and may result in a trip. When available, transmitter status can be utilized to trip the control loop to manual.
2. Dual redundant transmitters. Implementing dual measurements enables the operator to select either transmitter for control, typically tripping the loop to manual on deviation between instruments (10%-20%). The implementation enables the operator to bypass the deviation trip, select the transmitter judged to be correct and re-engage automatic mode. In bypass mode, successive trips to manual are avoided when the failed transmitter is replaced and is being tested. With this implementation, individual transmitters can be serviced without taking the loop out of service, noting that the Bypass/Normal selector should be returned to Normal when both transmitters are confirmed to be operating normally. The trip of the controller to manual mode limits any plant upset as a consequence of a sudden failure.
3. Triple redundant transmitters. When three measurements are implemented, the median value is normally used for control. Deviation between the values is monitored and alarmed when it exceeds 10%-20%, but it does not trip the loop to manual. The implementation enables the operator to select either the median value or an individual transmitter signal for control so that a failed transmitter can be serviced without it being included in the selection logic.
In this implementation, a single transmitter failure does not impact process operation except when it is initially the median value, when any upset would be small. This is the favored implementation in that it reduces the direct impact of transmitter failure on the process while allowing the loop to ride through the failure without any loss of control.
On the subject of redundancy at the physical controller, while we have grown accustomed to the high reliability of this programmable component of the control system, failures do occur. Basing the design on limiting the impact of a single failure can have interesting results. For instance, selecting a controller that has the capacity to implement the controls for the entire plant and using it for that purpose (always a tempting option) would, in most impact assessments, require redundant controllers. On the other hand, if the control is functionally distributed, for example, grouping common controls in a redundant controller and assigning unit controls to individual nonredundant controllers, the implementation may offer reduced impact on single device failure as well as lower risk in maintenance activities, such as controller downloads and upgrades.
For controller I/O, proven vendor statistics should be used in assessing whether or not redundant I/O is required, but keep in mind that a single card failure can affect the functioning of all connected channels and that the signals from redundant transmitters should not be wired to the same card.
Greg McMillan’s Response
Middle signal selection ignores a single failure of any type. It also helps to reduce measurement noise and ignore the slowest measurement response. This was a standard practice for pH measurement at Monsanto and its spin-off, Solutia Inc. The frequent failure point was 7 pH, which was often the pH setpoint. Also, slow electrodes were common due to normal and accelerated aging or imperceptible coatings. Noise was common due to extreme sensitivity to hydrogen ion fluctuations from imperfect mixing. Middle signal selection was extended to all measurements in a plant with extensive recycle pushed way beyond the original design capacity. Middle signal selection eliminated four or more trips each year, saving a hundred million dollars or more each year in lost production.
Reliability can be greatly increased by utilizing the best instrumentation, such as spring-loaded sheathed sensors in tapered thermowells with head-mounted transmitters; eliminating impulse lines; using magnetic flowmeters and Coriolis meters whenever possible; and installing throttle valves that meet ISA-TR75.25.02 Annex A requirements as to resolution, lost motion and T86 response time.
Dynamic simulation using first principle models is essential for increasing reliability by testing the ability of the control system to handle problems, especially unmeasured disturbances and abnormal operation. It is essential for procedure automation to reliably handle start-ups and transitions.
Russ Rhinehart’s Response
An excellent question! It is a difficult solution.
Nominally, industry would use a rational economic justification based on the time value of the cost of enhancement relative to the future savings. But such is grounded in the probability of events happening and the probable savings from a more reliable device, procedure or maintenance program. Although the increased cost associated with a better device or maintenance might be easy to estimate, I suspect that both the economic cost of a malfunction and the probability of a malfunction would be very difficult to predict.
There are other context issues to be considered. 1) How critical is maintaining production? The loss of production due to a malfunction might have scant impact if the company has excess production capacity, but there could be major repercussions if running at full capacity with low product inventory and impatient customers. 2) If the product or process is speculative, a semiworks process might buy inexpensive devices. Alternately, if the process is expected to run for a long time, then more reliable devices, strategies and procedures could be justified. 3) The economic impact could include waste generation, equipment cleanup and risks associated with repairs, which would be difficult to estimate. 4) The impact could include labor relations (improved reliability may justify reduced operational or maintenance staff) or the ambitious plant manager’s image with corporate. Either of which might override any rational economic analysis.
My first inclination for an answer was to simulate the future many times with independent realizations for the probability and penalty for all the events that might impact the economic benefit/cost factor. It is termed a Monte Carlo analysis. My investment managers do this to forecast what my portfolio might do in the future to let me choose a balance of safe-to-speculative investments. While this might be impressive, it is grounded in assigning probabilities to all the many future events, and the final decision is an intuitive feel of my comfort. For designing control system reliability, I don’t see how these choices could be defensible.
However, every time I make a major purchase (car, refrigerator, washer-dryer, table saw, etc.), I intuitively balance the reliability-to-cost economics or the purchase tempered by auxiliary issues such as aggravation of malfunctions or associated inconveniences, my current financial situation, my projection of years of use and my wife’s opinion. I’ve been happy with this intuitive approach guiding my personal decisions.
So, I would offer that the right way to select “strategies, practices, hardware and software [to] … enhance control system reliability” is to let an expert, with their understanding of all the issues, intuitively choose.
If so, this means that we need to train control engineers to be able to understand all the issues, to be able to balance the desirables and undesirables, to be able to explain the recommended choice to management and to be flexible to accept a management override to their control action without winding up.
Bob Heider’s Response
I recommend the technology in the “Two-Dimensional Contribution Map for Fault Identification” article in the October 2014 issue of IEEE Control Systems Magazine. The following is the introductory paragraph:
“All control engineers should be able to detect and identify faults (that is, abnormal conditions in a system) from the analysis of large heterogeneous time-series data sets. This “Focus on Education” column provides an introduction to multivariable data-based methods for fault detection and fault identification, with the latter being the determination of system variables that contribute the most to a detected fault. For fault identification in statistical process monitoring, the contribution plot is the most commonly used tool for quickly identifying the most affected variables. Contribution calculations are revisited in the context of principal component analysis (PCA) and T2 statistics, and a two-dimensional (2-D) contribution map is illustrated for the examination of time-series data under faulty conditions. The 2-D contribution map is compared to the traditional one-dimensional (1-D) contribution plot using simulated data from a realistic chemical process. The 2-D contribution map demonstrates the potential to enable a greater understanding of the fault and how its effects are propagated through the system.”
Ed Farmer’s Response
Long ago, back in the ‘70s, there was a sewage treating facility with a huge pond and floating aerators. It was critical in meeting objectives, but a long way from the people who serviced it. Performance depended on floating aerators, so some sort of monitoring was important. One of the staff had an idea. He positioned signal lights and detectors such that a missing water fountain over an aerator would trigger an alarm, motivating a drive out to the pond to determine why. All in all, it worked great for the intended purpose.
Of course, its success encouraged even more effort. Knowing what was wrong “out there” enabled maintenance staff to show up with what was needed to “fix it.” Eventually, technology was developed and implemented so that the entire plant could not just be monitored but also completely controlled remotely.
When monitoring capability is commensurate with current technology, it becomes possible to know lots of things about system functionality. This enables assessment and focused intervention.
“Details” increase the capability for appropriate and timely response plans. Collections of “details” can enable a single operation center to support a large number of operations over a large region. I remember a morning at a California refinery when an east-coast research and development facility took “remote control” of a refinery process in order to test a new control algorithm.
In general, it’s a good idea to carefully assess risks and promptly implement appropriate management. That should be a thought in most process control and management projects.
The opinions and views expressed are solely those of the authors and do not necessarily reflect any official policy, position or views of the International Society of Automation (ISA), Automation.com or the ISA Global Cybersecurity Alliance (ISAGCA).
