Auto and Remote Manual versus Manual Control | Automation.com

Auto and Remote Manual versus Manual Control

July 102015
Auto and Remote Manual versus Manual Control

In Water & Waste Water Applications

By Gene E. Keyser, Ph.D., Key Solutions, Inc.

Maintenance Superintendent in response to Electrical Engineer after Instrumentation and Controls Designer advised that there would be no control panels and no manual controls in the new facility:

“Let me see if I’ve got this right.  You’re taking away my electrical control panel that’s outside, overheated, corroding in a bad atmosphere sitting next to raw wastewater, that I have to work on in the rain on top of the deck two stories off the ground and giving me a portable control panel that I can take anywhere I’m working and have full control of every motor or switch with a complete record of everything I touch.  Just why am I supposed to be upset?”

Why does the “Auto v. Manual Control” or more correctly, “Auto and Remote Manual v. Manual” come into a discussion of reliable process information and process integrity?  The chemical, pharmaceutical, and a host of manufacturing industries long ago abandoned the concept and implementation of hard-wired manual operation.  Bypassing the control system and the traceability of every action it offers simply cannot be tolerated in today’s safety conscious, litigious, efficiency conscious workplace.  Further, for most real world processes, there are already more unknowns than there are equations to determine them; why add more unknowns into the mix?  Why spend more to get a system that performs less well?

Discussion

Five significant elements must be considered when upgrading a facility or when building a new one:

  • Liability
  • Operations
  • Maintenance
  • Capital Cost
  • Reliability

Consideration of these same elements comes to the forefront when looking for opportunities to reduce operating costs through “automation” or while applying automation to improve compliance and product quality.  It is unfortunate that “bad” automation exists, but it does, and it strikes fear in most operators and maintenance folks and is used as an historical bludgeon in the middle of the night by opponents of the practice. 

Liability

In a litigious world, the last thing a manager wants is to have a workplace accident where the protective equipment is not being used, or worse, where it has been circumvented.  With a manual bypass (Local-Remote) or hand switch (HOA, Hand-Off-Auto), what’s normally being circumvented are the device controls such as motor temperature, ground fault, phase monitoring, device speed, bearing temperature, lubrication monitor, torque sensor, oil level, motor overload, protective fuses, control power, etc.  All of those devices should be in place for safe operation of rotating equipment, regardless of personnel qualifications.  If troubleshooting, all of those same devices are suspect and should be investigated individually BEFORE attempting to operate the equipment.  Especially if all of the above are hardwired in series, add to the inspection list the wiring and connections themselves.  If not all, which should be bypassed for “manual” operation?  Why not bypass the feed breaker?  It makes just as much sense as any of the others.  With all of the above reporting individually to a PLC, including the feed breaker and starter, AND controlling the motor directly rather than through a switch, the culprit would be known when the problem occurred.  Operations could then make a decision to bypass the individual control, not the entire collection of safeties.

So when is “manual” no longer really manual control?  Is partial manual akin to being “a little pregnant”?  The liability of bypassing the safeties is real.

Operations

The most frequent demand for manual operation of a device comes from operating staff that doesn’t trust “automation”, doesn’t agree with the choices available in “automatic”, or most often, doesn’t have a properly automated system.  The apparent shortcomings of poor process control are revealed when variables or indicators of process management are monitored continuously and plotted to show the effects of process changes and the variability of treatment versus throughput versus the time of day.  An example is warranted.

Figure 1. Nearly Identical Daily Totals but Different Flow Rates

A splitter box with two different gate sizes and elevations was “automated” with gate actuators to balance the flows going to mirror image water treatment trains on one site to correct differences in power consumption, efficiency, and treatment efficacy.  The simplest of programming, there having been no performance specified, had the daily totalized flow leaving the process trains and the gate positions adjusted daily to balance the splitter box output.  The daily total to each side was reproducibly within 1%, but the plants remained as different as if they were in different zip codes.  Actions valid for the West train were categorically wrong for the East and vice versa.  When flow meters were eventually placed on the direct outfalls of the splitter box, the data shown in Figure 1 was typical of the results and the problem became apparent to all – equal daily totals are not the same as equal flow rates on a minute to minute basis.  While in this case the control timing “problem” was made obvious by the physical distance between the original flow monitoring point at the end of the process after three overflow weirs and the technically correct monitoring point just after the control gates, it is not always so obvious.  Adjusting the flow from the control box on a minute to minute time frame not only balanced the flows but made the plants relatively equal in their treatment process.  Using a control scheme based upon 15 s geometric averaged flow with 5 sample points/sec, the difference with real time feedback versus without is about 120 kw average daily savings, (420 kw vs. 300 kw), ~3000 kwh daily, ~$109,500 annually, ~$0.011 per thousand gallons treated for <$10k gate actuation and programming (installed cost) – an ROI of about 5 weeks.

The splitter box control example is one of the simpler examples where preconceived “facts” and design control parameters are not borne out by the evidence.  Everyone “knew” if the daily flows were the same that the treatment trains would perform equally – after all say most engineers, designers, and owners alike, “it’s just a wastewater plant.”

Another large objection to automation from operators is the concept that it can eliminate the need for operators.  Staffing requirements in the water and wastewater industry are generally set by the regulating authorities on the basis of long standing practices and formulae and those requirements are rarely supplemented by local management to address real problems. According to a broad sampling of owners, regulators, and licensing authorities, no states have an excess of licensed, skilled, wastewater operators.  In some, like Florida and Georgia, the average age of licensed operators is now over 50 – an ill portent of things to come – and none are more keenly aware of the shortage than the regulatory agencies.  To diminish the licensing and staffing requirements puts the very fabric of a self-reporting industry at risk, and offers liability of the kind that appears in the headlines – automation is not a “designated jailee” that can be held responsible for performance as can a “licensed” operator or an elected official.

Without understanding, there is no trust; without trust there is no confidence; without confidence there is no implementation; without implementation, there is no understanding.

The key to breaking this “chicken and egg” conundrum is to understand what automation can do.  Automation can handle the recordkeeping, reporting, and eliminate tedious or repetitive tasks without lunch breaks or interruption; automation can make the same good decisions based upon good data in every case and advise when the data is “bad” for whatever reason; automation does not create the action plan nor reduce it to finite details and choices.  Automation can provide the record for improvement and the justification for expenditure, or lack thereof.  And when properly done, it does not “forget” and it does not get distracted by lesser or more important events.

Maintenance

When troubleshooting a problem or starting up a new piece of equipment, one of the first and more frequent complaints is that “the program changed”.  Literally countless times the programs of Programmable Logic Controllers (PLC’s) have been checked to verify in fact that the program didn’t change;[1] it generally doesn’t.  In traditionally wired systems with relay logic in place, the most frequent problem is a blown control power fuse caused by a failed relay.[2]  In hybrid systems having relay logic and a PLC, the most frequent problem is a blown control power fuse caused by a failed relay followed closely by a failed timing relay. In hardwired PLC systems with “hardwired” digital I/O and 4-20 mA analog I/O, control power transformers and their blown fuses compete with loose terminal block connections, even when those systems also include a communications backbone of fiber, copper, or fiber and copper.  Entering the communications based era of automation with systems having reduced wired I/O[3], residential quality fiber to copper converters and unmanaged, non-industrial grade hubs and switches begin to compete with failed fuses and so-called “isolation” relays as the primary sources of failure, the latter being a poor “spec/bid/buy” substitute for surge protection.  Note that in none of the above typical scenarios does PLC failure become a maintenance issue.  What is the primary maintenance issue, in new systems and old, are the wired connections.  The lesson to be learned is reduce the connections, reduce the maintenance.  And the system is better and simpler if the connections and the wires are monitored for system integrity – then you know where to look.  If you must still use wired I/O3, one simplifying rule is to have all signals be dedicated wiring between the sensor and the PLC or between the switch or the breaker and the PLC; no mechanical relays, no mechanical timers, no unmonitored fuses.

The most significant contribution to life cycle cost of rotating equipment after the electrical bill itself is the cost of maintenance, both in manpower and parts.  A distinct advantage of well automated, remote systems is the diagnostic capability.  The network itself and the individual parts and pieces right to the motor and device level can be continuously monitored with reporting in the event of loss of data integrity.  The maintenance records in the system referenced above tell the greater story.  When fully automated, maintenance callouts and overtime was reduced to less than 1% versus the hardwired, manual systems, primarily because the problem was identified before “maintenance” even entered the picture.  Knowing the “problem”, operations either remotely bypassed it, took the unit out of service pending actual repair during normal hours, or logged in a specific override to ignore the alarm pending further investigation and correction.  The typical solution – a programming change to allow a particular event or set of circumstances not originally contemplated – once done a single time at the programming level, that unknown situation never occurred again.

Capital Cost

Perhaps the loudest objection to automation in any form is its presumed higher capital cost.  There is a shift in the budget items from mechanical equipment that comes without controls at a lesser price with all of the controls moving to the electrical and instrumentation line items.  Consider that local control panels can be replaced with a portable control panel in all situations – there is a significant reduction in the original cost of the equipment.  Consider that an indicator “RUN” light is not just the $39 for the fixture and $9 for the light but wiring to have it driven, by the PLC or separately, the power supply, and output from the source of the signal – all installed, about $370.  The HOA switch safely interlocked with just basic motor overload and breaker auxiliary contacts, about $540.  Wiring to and from PLC for the local control panel in parallel with the “auto” circuits – more copper, more conduit, more design, more underground conflicts.  To what gain and to what end – so that the safeties can be bypassed and the control logic circumvented without permanent record?  In even the most basic of control systems in industrial settings the installation and use of “remote manual” is a requirement.  The strings attached to remote manual are all positive and the benefits are real information with accountability and integrity – and lower capital cost, not more.  In a recently bid project, the added cost of automation, just eliminating the HOA switches, indicator lights, and start/stop buttons from the motor control centers and replacing them with an extra PLC, communications cabling,  and programming was a savings of about $400,000 out of a total of $2.07 million actually bid and $2.4 million budgeted a year earlier.  More information and more control for lesser cost.  At the individual motor or instrument level, hardwired intelligence and automation is slightly less or equal to standard, local manual wiring and configuration; wiring both intelligence and local, manual controls is about 30 – 40% more than either alone; intelligent communications based equipment can be as much as 15 – 25% less with no hardwired switches, lights, bells, or whistles – with more information and more control available.  From a fractional perspective, the costs of controls percentage will likely increase and the percentage cost of electrical gear as part of the whole will likely not go down, but space and building requirements, mechanical equipment packages, installation, and start-up will drop in real amounts, in real dollars.

Reliability – the Core Requirement

Perhaps the most frequent argument heard about automation is from either operations or maintenance staff demanding that there be manual back-up for everything because the automation system “cannot be trusted”, or “it’s not reliable enough”, or “it always fails”, or “we need to take it down for maintenance”.  These claims are not necessarily true because more correctly, the statements are self-fulfilling prophecy.   When manual switches and overrides are added to an automation system, it is less reliable than without the manual overrides. The extra parts and pieces and their installation are not just costly but contribute significantly to the poor reliability of crossbred installations. 

Mean Time Between Failure (MTBF) is the de facto common denominator for comparison of electrical and electronic hardware service life and is the average time between failures of components or devices.  Initially it is a statistically calculated number; reputable manufacturers then adjust the reported value on the basis of real world experience.  MTBF is a measure of reliability, not a measure of life expectancy.

To determine the MTBF of an assembly of parts with known MTBF data, you take the inverse of the sum of the inverses of the component parts.

Component

MTBF Data[4]

(years)

Softstart

50

HOA Switch

10

Indicator Light

25

Terminal Block

30

Power Supply

25

Single Wire

300

in the example shown, the same softstart is to be compared with and without an HOA switch as well as with and without Run, Stop, and Fault lights for the face of the cabinet.  Both installations require a control power supply and premium components are used. 

Component

 

 

with

without

  Softstart

1/50

1/50

  HOA Switch

1/10

n/a

  LED Run, Stop, & Fault Lights

3 x (1/25)

n/a

  Terminal Block

5 x (1/30)

n/a

  Power Supply

1/50

1/50

  Wiring

20 x (1/300)

2 x (1/300)

 

Total Assembly

 

148/300

 

14/300

 

Total Assembly MTBF, years

 

(300/148)

 

2

 

(300/14)

 

21

When multiples of the same component are used, the effect on the total assembly is most readily apparent as in the case of 20 wires.  Even though the MTBF of a properly sized and connected wire is 300 years, the additive effect is a large one, just as with terminal blocks or LED indicator lighting.  Without the wiring and components of “manual” bypass arrangements, the MTBF is significantly increased in every case; the addition of the extra components is what shortens the average time between failures, even though each component may be long-lived and “reliable”.  Eliminating the manual and visualization components and moving them to a single composite, remote control system is the primary mechanism to increase both the real and perceived MTBF.

A further extension of the reliable assembly argument comes to the forefront when comparing assemblies of competitive components, softstarts as an example.  In the example given below, two softstarts, one of the best and one of the worst in the marketplace, are made near equal in performance, or lack thereof, by surrounding them with clutter and extra components.  The differences and the values are apparent, when as shown, the assemblies require identical installations.  The real case is that the installation, wiring, terminal blocks, cabinets, etc., are not identical and often ignored – and the differences between Brand A and Brand B suffer their reputation instead caused by lack of design and lack of forethought.

 

MTBF Summation

 

              Brand A            

Brand B

Softstart

1/50

1/50

1/20

1/20

HOA

1/10

n/a

1/10

n/a

Run, Stop, and Fault Lights

3/25

n/a

3/25

n/a

Terminal Block

5/30

n/a

5/30

n/a

Power Supply

1/50

1/50

1/50

1/50

Wiring

20/300

2/300

20/300

2/300

 

Total Assembly

 

148/300

14/300

157/300

23/300

Total Assembly MTBF, years

2.0

 

 

21

1.9

 

13

Another key question for debate in the midst of this comparison is the use of local displays, and/or local control panels versus portable controls without local displays and without local panels.  Wires, switches, programming, wireless hardware all come into play.  From an MTBF perspective, portable wireless control devices, no displays and no local panels are the unequivocal choice – it’s simply math, significantly fewer components, and a lower life cycle cost.  From the perspective of information and control, the accountability of portable control devices without the ability to manually bypass security and safeties is the clear choice.

Local displays are sometimes a point of contention in the automation debate, like indicator lights, best reduced to a series of pointed questions: a) who’s there to see them; b) are they real numbers (as in calibrated, true, and correct); and c) if they’re so important to know, they should be included in the visual control system.  Florida summers and Minnesota winters do wonders for liquid crystal displays, and of those remaining, UV and condensation will do them no good in short order.  A careful look at many pipe galleries or pump stations should be the most convincing; whether oil-filled high end gauges or low bid plastic cased ones, pressure gauges installed as some unknown convenience are rarely in agreement and even less often accurate.  If the pressure is so critical that it needs to be continuously monitored, do just that; if not, install a pressure tap so that it can be checked, precisely and accurately, when it needs to be known.

Reliability, or in economic terms, availability, is directly addressed by a combination of MTBF and Mean Time To Repair (MTTR) – “how long until we’re running again?”

In a failure of a component in an assembly, MTTR becomes the sum of the times to find the offending component, get the component, replace it, and return the assembly to its running state.  By example, for a 1 year MTBF assembly, a once per year four hour troubleshooting hunt to find the loose connection, plus seconds to tighten it, plus two hours to start up and return to normal production, a total of 6 hours, is:

With 20 of those 1 year MTBF sub-assemblies in a process unit:

Five days per year of down-time!  Improving the Process Unit MTBF by 25%, either by improving the individual components or reducing the number of components in the Process Unit, offers comparable increases in availability.

Redundancy – hot or standby – back-up systems, and alternate processes immediately spring to mind, and they have means and methods to determine MTBF and Availability as well:

For a redundant PLC controller without manual in parallel, having an individual MTBF of 37 years:

For the same PLC controller wired with a single HOA switch, MTBF 2 yrs., in parallel with a manual operator, MTBF 1 year in the best case:

When MTTR is added into consideration to determine availability, the failover or transition time is the effective MTTR to restore availability; let’s consider a response time of the operator to change to manual of 15 minutes, 0.25 hr., versus the PLC redundant pair at 0.3 seconds.  At 0.3 seconds, the process does not require restart, but at 15 minutes, the same 2 hour restart as above is required.

 

In a simple process with as few as 20 HOA switches capable of bypassing PLC control, the anticipated downtime is 22.5 hours! – and that using a gracious allocation of a single operator error per year.

To retreat to a more distant perspective, the addition of manual bypasses to a properly automated process is a self fulfilling prophecy; the “automation doesn’t work” premise is actually a result of the manual back-up plan being imposed on the automation system.

Conclusion

To restate the less than obvious, beyond the traceability and liability issues, even remote manual operation should be an alarm condition for any device or process.  It indicates that the current condition of the process is not recognized as it is in need of correction, or worse, that the current condition is not recognized at all.  Essentially, either the process controller does not know what to do (a bad thing), or the process is sending false information in the opinion of the operator (it’s lying, a worse thing), or the operator is taking a different action than has been planned and programmed (questionable, but maybe the operator knows better?); any of these call for corrective action – if it happened once it will happen again.

As capital, liability, operating, and maintenance expenses come under scrutiny from an increasingly sharp budget knife, today’s automation platforms can provide for lower total costs and reduced liability while saving both operating and maintenance costs.  At ground zero where the individual motor is operating, a new approach and an open mind are essential to take advantage of the opportunity for savings and improved availability.  From normally divergent perspectives, all signs are pointing toward change for the better.

Acknowledgements

Every composition requires more than just words to print, in this case including the efforts of Wes Maffett of GE, Dean Breaux of MWH Constructors, Brian Hinkle of Schneider, Andreas Pirsing and Anna-Maria Preta of Siemens, and Mike Stoup of HT/DCR, among the many who provided both information and served as foils.  The quote in the abstract is real, but at his request it shall remain unattributed.

Sidebar

Almost right...can be very expensive.

As grandson of the paper mill powerhouse superintendent, son of the maintenance superintendent, and brother to the senior (electrical) project engineer I grew up knowing as a fundamental law that nothing was faster than copper, and nothing would get you killed faster than copper in the wrong place.  These were tenets spoken at the dinner table, and unfortunately over the years at the wake of more than one family friend, wannabe do-it-yourself electricians.

Common in the wastewater industry, and seldom elsewhere seen, is the practice of using the same ground grid for lightning, load, and instrumentation.  Some are so bold as to have them common in the cabinet but most use multiple connections to the same grid underground; out-of-sight, out-of-mind, until lightning actually does strike.  With a common grid copper is fastest and one strike successfully destroys power supplies, motors, instruments, controls, and careers.

A recent project took the approach described in this article and is the poster child for its efficacy and cost effectiveness.  However, somewhere along the way someone decided that another large savings could be had by using a common ground grid for load, lightning, and instrumentation.  Some difficulties encountered during installation of the Profibus DP networks in the project were ascribed to local wiring mistakes in the grounding configuration. The near equivalency of the grounding tests were ascribed to chance.  After a direct lightning strike on site the three weeks to recover motors, instruments, and drives, just to name a few casualties, can be ascribed to greed, or ignorance, or both.  Proper execution of any approach does not include “almost right” as a possible path to success.

 

___________

[1]Occasionally, less than thorough programmers have been known to change the runtime version of PLC code without changing the code to be used on a “cold restart” of power.  In this case, it’s either a mistake or an unknowing error or a failed backup power supply, but rarely does it happen twice to the same programmer.

[2]65% of the more than 2000 work order actions based upon callout, operations, and maintenance records for three facilities (7.5, 25, and 40 mgd) and their three collection systems totaling 215 lift stations over the course of 5 years prior to the start of automation.

[3]Wired I/O in this context is wired circuits to and from individual inputs and outputs versus communications based transfers of blocks of information and control signals.

[4]Manufacturer’s data rounded to whole years for the purposes of this example, courtesy of Siemens, available on their web site.

Did you Enjoy this Article?

Check out our free e-newsletters
to read more great articles.

Subscribe Now

MORE ARTICLES

VIEW ALL

RELATED