System Migration Attacks Skills Crisis

 

By Bill Lydon - Editor

 

 

Daniel Tadie, Manager of Remote Energy Plants, Colorado Springs Utilities (retired) gave an impressive and thought provoking presentation at the ARC World Industry Forum February, 2010.  Tadie described a successful migration from a legacy DCS to a new system at Colorado Springs Utilities Birdsall power plant. I was impressed with the philosophy, methodical approach, system performance results, and significant improvements in operations. This experience has established the future standard for all remote sites for Colorado Springs Utilities. Tadie worked for Colorado Springs Utilities nearly 30 years before retiring in 2008. Dan and I talked about the project and how he believes positive results were achieved by applying a well known management methodology, Kepner-Tregoe.

 

 

For the last four years, Tadie was manager over the utility’s fleet of hydroelectric generating plants, peaking power plants, emergency generating stations, and a natural gas peaking plant. Much of his career has been focused on maintaining and optimizing power plant control systems at many of the utility’s power plants. Tadie oversaw the controls modernization for several generating units, which they upgraded from pneumatic-based controls to DCS with in-house staff. Tadie also established remote site monitoring and control for the hydro plants, and guided construction of new state-of-the-art control rooms at two plants.

 

Colorado Springs Utilities (SU) provides electric service to more than 200,000 homes and businesses in the Pikes Peak region. The municipally owned utility also supplies natural gas, water and wastewater services. The utility has 7 power plants with 17 generating units consisting of Combined Cycle, Coal, Gas/Oil fired units, and 5 hydro units. It also operates three water treatment plants and two wastewater plants. The utility employs 1800 people with an annual budget of around $1 billion.

 

 

Department restructuring placed six plants into a new separate group named, Remote Energy Plants. The Birdsall plant is part of this group and was considered unreliable by the power schedulers, requiring them to purchase power from the grid. The 1950s-vintage Birdsall power plant produces 54 MW and was to become the new base for a 24/7 monitoring operation of the remote hydro plants. The Remote Energy Plants had five different obsolete control systems.

 

 

The 18 year old DCS was obsolete. Tadie said, “I got that dreaded letter…support dried up in 2008.” He further explained, “The migration path the vendor suggested may be the easy route but it was the most expensive route.” “Just to add an HMI was $80,000 each and I needed three of them.” 

 

 

Tadie had a number of problems to deal with at the plants including obsolete control systems, staff of operators without technical background, breaker problems, and boiler problems. Tadie and his staff applied Kepner-Tregoe (KT) Decision Analysis methods, designed to take emotion out of the decision process - to focus on facts and identify the real problems. The utilities general manager embraced KT a number of years ago by training a wide range of employees so they had a logical and consistent decision making process. Tadie and his people use KT methods routinely for making decisions, troubleshooting, and analyzing plant processes. The basic concept of KT is to use methodical, rational thought processes, and be very specific about documenting information to solve problems. “The goal is to make decisions that bring the best value to the company,” says Tadie.

 

 

Tadie stressed that the first step was to quantify five “strategic measurements” to understand the situation.

Applying these measurements to the Birdsall plant brought into focus that this old plant did not measure up to operating goals.

 

The company’s investment to train employees in the KT methodology provides a common process to solve problems, indentify improvement opportunities and justifying them.   Tadie noted, “Following the process made it easy to build a business case for making changes!”

 

 

Using these measurements he identified and prioritized problems. The numbers were used to develop a proposal for improvements with estimated economic value based on specific improvements in the operating measurements. Tadie described how they worked to understand the real problems and potential opportunities for improvement. Another key method he discussed was illustrating the value of replacing old unreliable items using potential problem analysis to estimate the economic consequences of failures based on lost time, production loss, and repair cost.

 

 

The team developed 13 criteria that were used to make decisions. The two top criteria were the ability to interface and display data from other sites and operations reliability. Other criteria included a common platform for all applications and locations, simulators available for training, open architecture archiving, and ease of operation. It became obvious that selecting a common control platform that would scale from small to large sites brought the greatest value.

 

As part of the selection process the team applied a KT technique of testing your choice against worst case scenarios with the best solution to expose flaws in selection.

 

The original concept was to standardize on a platform and then have a number of system integrators bid. After analysis a turn key project cost could not be justified so they decided to use in-house staff for installation and relied heavily on the selected vendor for the system design and configuration services. This had the advantage that the in-house people would understand the system.

 

Tadie wrote the technical specification that included plant equipment descriptions, functional processes, and point list for the system. He then had to explain to purchasing people the justification for using a single platform and how it could be bid by a number of system integrators. 

 

Rather than rely on old control strategies he told bidders, “You guys figure out the best control strategy for my plant; bid that…” They decided to use the Rockwell Automation platform based on their criteria. A big factor, Tadie cited, was long term support, “I loved the fact that they supported the very first equipment they ever sold.”

 

A number of system integrators bid and Rockwell Automation’s Global Solutions engineering delivery business won. They worked directly with Rockwell Automation to purchase the system and to fill in with integration services that they needed. Operators developed HMI screens from the ground up based on reviewing the best at other plants and made use of newer HMI features. 

 

Some people had factory training and the plant purchased computer based training programs. Weekly training was held for their in-house people.

 

 

Tadie emphasized, “Painstaking coordination by all contributed to success.” They mapped out every task and assigned staff members to them. At the kick-off meeting with Rockwell Automation, they discussed and refined responsibilities for all aspects of the project.

 

 

Plant documentation was outdated, so staff proceeded to verify every field device, range and scaling, and confirmed every cable. Tadie noted for the staff, “It was great training!”

 

Colorado Utilities IT group setup our networks and installed OSI PI archiving software and servers.

 

 

Tadie explained they decided to install new cabinets following the same configuration as the old cabinets, with terminations behind and the control components in front. They verified every sensor, every wire, recalibrated every field device, verified failure mode of valves with air or power loss. Field cable lengths were measured before the new cabinets were designed so terminal locations could be defined to accommodate the cable lengths and minimize splices. Months before being disconnected, all field cables had been researched, confirmed and labeled to minimize errors during the re-wiring process. Cabinet layout and termination assignments were planned so well that only 6 cables had to be spliced. No wiring errors made it through the pre-operational testing phase, and only 2 were found swapped during the testing.

 

 

Tadie contrasted this approach with migration approaches that use existing wiring and adapter connectors stating, “That will make for the worst nightmare.” He explained that some of their other plants tried this with bad results. “Our other plant went that route and went from a predicted 7 week outage to 12 weeks costing millions of dollars.”

 

 

 

 

Pre-operational testing was performed after system installation, but prior to Rockwell Automation arriving for System Acceptance Test (S.A.T.) all points from field back to system terminals were tested. During S.A.T every input point was modulated in the field & observed on the HMIs. Every output was stroked from the HMI and observed in the field. 

 

 

Dan Tadie highly recommends using the Kepner-Tregoe methods. “You can buy the book for about $20,” say Tadie. Tadie and his team took the time up front to do analysis, develop criteria, and clearly understand their objectives.

 

Involvement from the operators gave them a far better understanding of the control system and how to operate the plant.

 

Tadie is passionate about using your own staff since it creates a valuable in-house resource.   Six years ago he used in-house people to install a different system and a routine outside interview of plant people illustrated the value of doing as much in-house as possible. Tadie says, “This plant had an 80% confidence factor among the staff that they could fix 90% of what broke. Another plant that did a turnkey system had a 30% confidence factor that they could fix 50% of what broke.”

 

Tadie strongly believes if you have frustration with the current system, you can develop justification by using a methodical rational method like KT. 

 

It is easy to get caught up in simply looking at replacing an old system rather than improving operations. Using the KT decision methods allowed Tadie and his team to go beyond simply deciding to replace the existing system to explore how to improve operations to provide more economic value to the company.

 

One of Tadie’s philosophies (he mentioned more than once) is, “We can always do better.” Nice management slogan but he put it into action by understanding the problems/opportunities; then developed a justification and action plan.

 

The poor reliability of Birdsall translated directly into lost profits because schedulers couldn't count on Birdsall's units consistently so they would purchase power from the grid.  By upgrading generator breakers, re-tubing the condensers, and replacing the control system (which totaled <$1.5 Mil), they can now count on Birdsall to consistently operate as needed, reducing the need to purchased power. With Birdsall reliable & available, it reduces purchased power by over $6 million annually, mainly by operating as spinning reserve or generating during other unscheduled unit outages. 

 

I believe that Tadie’s philosophy, “We can always do better”, is a good perspective. Many times management is looking for the lowest cost migration to make the obsolete system problem go away, without looking to improve.

 

Tadie’s perspective on the skills crisis is that companies have not been developing in-house talent and are now paying the price. Earlier in his career he learned that a plant can be run more reliably and efficiently by developing and training in-house people.

 

What are your experiences with system migration? Please share your comments, thoughts, and experiences.