Why doesn’t everyone use remote access?

  • May 03, 2011
  • News
May 2011
 
By Frank Hurtte, River Heights Consulting
 
I can cut back in time like a knife to that very day. The year was 1988 and a factory PLC expert had just demonstrated remote access.  It seemed a miracle as he connected the monstrously heavy “programming terminal” to a modem and then accessed the PLC across the room. While he changed a few timer values – my mind whirled. Later on, that demonstration spurred a lot of discussion. We imagined a day when PLC’s and other controls could be troubleshot and monitored without an expensive and time consuming trip. And the trips have grown far more expensive and a lot less fun in the post-911 era. 
 
The question we would like to explore is pretty simple. Why doesn’t everyone apply remote access in their automation projects?  
 
What is the potential economic impact? 
Back 20 years ago it was significant, today it’s huge. To illustrate the point, let’s look at what might happen when a special machine builder experienced issues in the field. As they say on TV – the names have been changed to protect the innocent. There is no RHC Ltd, but the data is real and collected from machine builders across the Midwest.
 
RHC Ltd. is a special machine builder. Each of the machines produced by RHC Ltd. uses (at least) a single PLC, a number of drives or motion control systems and a Human Interface System. RHC employs qualified engineers and technicians – these are guys with 5 or more years of experience in manufacturing. The owners of RHC pride themselves on great customer service and they try to maintain a balance between great customer service and profitability – in this business reputation is everything. Their machines go into mission critical applications where downtime is costly. When an RHC machine goes down the customer wants it up and running ASAP and RHC scrambles to help.
 
RHC’s experience dictates the average machine goes down an average of three times during the first six month of operation. Listed below are the steps of a typical machine down scenario:
 
The machine stops
The beginning of our experience
Start
Customer tries to fix himself
Customer does some rudimentary troubleshooting and is unable to make the machine resume operation.
2 hours
Customer Calls RHC
The customer calls into RHC. The transaction looks like this:
  • Call to switchboard routed to the engineer
  • Engineer provides customer contact with a number of things to check – then call back.
  • Customer calls back, nothing stands out as the cause of the problem. The engineer then asks the customer to check I/O, confirm configuration and back-up files
  • Customer phones back – this time a bit agitated because the machine has been down for a five hours by now.
  • No Solution – Customer asks for a sight visit
3 hours
Decision to visit
Engineer meets with owner/manager to reach decision to travel out to the customer
30 minutes
Personal Plans are changed
Engineer calls his family, cancels plans to attend their daughter’s softball game and catches grief from spouse
timeless
Flight Reservations
Engineer arranges for transportation to customer site. This includes the following:
  • Flight to customer location
  • Rental car at customer location
  • Hotel reservations
45 minutes
Engineer travels to customer location
Travel included 2-hours lost at the airport due to airline security, time lost waiting in line for rental car and travel from airport to customer location
6 hours
Customer site
Engineer meets with local maintenance team, speaks briefly with operator, makes a few programming adjustments and waits for the machine to start up. After running for a couple of minutes everything seems fine, but the Engineer hangs around for half a day to “watch the machine” for other problems. 
8 hours
Engineer overnights
Check into hotel, dinner alone, calls family
timeless
Engineer travels back to home location
Same lines, time lost waiting in airport and issues associated with modern travel
6 hours
Total time investment RHC Ltd                                                                24.25 hours
 
First let’s look at the out of pocket costs for this short visit.
 
Item
Cost
Travel to local airport
$15
Airfare – Davenport, IA to Charlotte, NC
$538
Rental Car 3-days
$150
Hotel 2 nights
$250
Living Expenses
$200
Parking 3-days
$24
Totals
$1177
 
But the out of pocket costs pale in comparison to the real costs. First the RHC Ltd engineer was out of the office, taken away from normal work responsibilities for the equivalent of 3 man days. All this time resulted in a couple of hours of productive work. This results in a loss of productive time – which would normally be worth (conservatively) $80 per hour. This results in a loss of another $1,800 dollars in value to RHC.  
 
But the big loss comes at RHC’s customer where a down machine might cost literally thousands of dollars an hour in lost production revenue. Here we could make a conservative estimate. The customer lost two full work days on the machine – valuing this at just $500 per hour translates into an $8,000 dollar loss of productivity. 
 
So what is the point of this little exercise? For both RHC and their end customer, the costs are huge. What’s more terrifying is the statistic that most (60-70%) of these trips are unnecessary and easy to avoid. They involve things like electrical contacts, operator mistakes or elementary tuning to adjust for variation in raw material - one OEM engineering manager reported NPF (no problem found) as the leading cause of failure.  
 
Again, what’s the point? A typical down condition easily accounts for nearly $11,000 dollars in lost productivity. 
 
Modems and related technology
So, why didn’t everyone jump on the bandwagon back in 1988? A case like this screams for action. But before our conference room chairs cooled down, we ran into the harsh facts of life. Back then, the only option available was a modem – the dial up variety. These required a lot of tinkering to work and they faced three obstacles. First, they were slow – really slow. Any thoughts of actively making changes on the fly were shot. Yeah, you could update a few counters and times but anything that required lots of data was sometimes too slow to use. Secondly, there was no security. And the “coup de grace” came when you started looking for a telephone line. As you talked to the customer you discovered issues with physically getting the line down to the machine. When there were dozens of machines scattered all throughout a manufacturing facility, it was nearly impossible. 
 
What ended up happening was modems were applied on some of the absolute mission critical systems – the ones that could shut down a whole plant. Things like the main ammonia chiller inside a food processing plant might justify the line but the rest of the applications went begging and engineers continued to go on expensive unplanned trips. 
 
The modern marvel of the internet   
In the late 1990’s we experienced the internet explosion. Along with this phenomenon came a proliferation of Ethernet. In a nutshell in a couple years it was Ethernet everything. And in 2001 when companies like Rockwell began introducing Ethernet enabled Programmable Controllers (and later drives, operator interface devices and other components) it looked like our remote connectivity problems were over. 
 
A few of us were still fixated on the ability to make that remote access thing work. And, to a limited extent it happened. Using plant wide networks hooked to the internet, it would be possible to sit in my comfortable (and cluttered) office and fine tune processors wherever they may be. 
 
But there were issues
In those early days, none of us could imagine the extent of the data security issues. The plant network with all of its great abilities to facilitate data connection had to be protected. Once a person managed to maneuver their way onto the plant network, they conceivably had access to lots of important private information. Not only could they dig their way through payroll records and human resource information - a hacker could access trade secrets – proprietary processes, formulas for new products and other critical information. Remember this was in an age before malware and viruses. The IT guys were mostly interested in keeping unauthorized people from stealing stuff – but soon the internet environment got really tough. Successful IT departments now had to worry about changes to the system.
 
All of these concerned created a gigantic wall of internet-based security. Effective yes, but in a good many instances so difficult to get through its judged not worth the effort. The Virtual Private Network (VPN) is the most common tool for allowing someone off-site access to the plant network. If you have the ability to access your email or company file servers from home (or your motel room when you travel), you are most likely using a VPN. When you joined your organization, someone from your IT department created a special password that allows you entry to the company network. 
 
A VPN is defined as a network which uses a public infrastructure like the Internet to provide remote offices or individual users with secure access to their organization's home network. It aims to avoid an expensive system of private or leased lines that can be used by only one company at a time. VPN’s encapsulate data transfers between two or more networked devices that are not on the same private network. This keeps the transferred data secure from other devices on one or more intervening local or wide area networks. Access to the VPN is a highly guarded thing, because once on the VPN the user is on the home network.
 
Generally, control of the VPN lies in the hands of a corporate IT group. They are assigned work orders for setting up new users – and they have automatically generated work orders to shut down employees who quit or are terminated.
 
So why not just assign a VPN password for remote access?
Let’s put on our hard hats and investigate this intriguing question. First, there is security. If a customer grants you VPN access to their network so you can access your PLC – they are giving you access to their whole network. This is risky. Secondly, there are a number of steps to making it happen – many steps. In most company environments the VPN must be open to you for only a couple of days before or after you do your work. This shoots the chance of taking a random and proactive look at your customer’s system on the spur of the moment. Finally, once you get on the network you must remember a long string of IP address numbers just to find the right PLC.
 
Add to this the long running battle of wills between the controls engineers and corporate IT, and your ability to do remote access in all but the most critical applications is shot. 
 
But we are on the verge of a great break through                           
There are a number of promising technologies pushing into the remote access arena. Many of these come on the verge of Stuxnet and an inherent escalation of the computer-securities war. One such new technology comes from Belgium-based eWON (a systems integration company turned manufacturer). Their product takes advantage of a unique hardware and cloud computing platform platform. Let me explain how it works.
 
Even though eWON provides all of the standard full power IP router features/technologies (LAN, GPRS, EDGE, UMTS, HSDPA/HSUPA and ADSL), their real draw is the unique way they establish secure Internet connection between the user and the machine. 
 
eWON has developed Talk2M (“talk to machine”), a smart Web-based remote access method (think cloud computing). The cool point about Talk2M is the full integration of IT security standards. It enables Internet tunneling between the user and the remote machine without requiring any changes to IT network security settings at either end. This allows easy deployment while hiding the complexity of the IT network infrastructure. It also allows the cloud-based system to scan for malware and virus – like Stuxnet.
 
When we asked questions about the ease of set-up and connectivity, the eWON folks referred us to Mark Farina Security and IT Manager of Church & Dwight Products. Mark’s company received a handful of new machines from a European company. Because of the importance of uptime in his facility he immediately began the process of making his system ready for remote access. He describes the process used in the past as “complex and time consuming with a number of steps that required work orders to be submitted to the corporate network group”.
 
When Mark discovered his new machines had unexpectedly come equipped with the eWON devises, he made a couple of investigatory calls to eWON to learn how to connect them. He assumed the he would be forced to apply them the old fashioned way – holes in firewalls, VPN hassles and all the rest. Instead, in Mr. Farina’s words, “What I saw is revolutionary. With the Talk2M software we are talking first of a whole new class.” The work orders, the complicated interfacing with other departments, and all the rest were gone. After learning how it worked, he made the eWON/Talk2M Pro software the standard for his plant. Here is an insight into Mr. Farina’s thinking.
 
First the eWON automatically grabs an IP address, so there are no issues with assigning an IP address. This is time saver from the very get-go.   Then the Talk2M Pro service manages control access between users and the machine. Plus the software only allows communication with eWON devices so all of the security issues are resolved. But the greatest advantage came in the form of ability to manage access.
 
“With the Talk2M software I can control and monitor exactly who goes into each of my control devices. I know when they accessed the device, I can easily change control device assignments for systems integrators, internal people and machinery OEMs. And all of this can be done with just a few keystrokes.” For those of us in the controls business, it’s pretty easy to see why this would be important to a security minded end user. 
 
Remote Access with a few added benefits
Years ago when we stood in that conference room looking at remote access in the making, we imagined easily tweaking times, resetting counters and maybe watching some data point change state. Minus the security concerns all of that is now in our grasp. With tools like the eWON and Talk2M web-software we may have put the security bugaboo to bed. And, there is a pot of gold at the end of our remote access rainbow. 
 
Because of the nature of the eWON device we don’t get access to just the PLC. We get access to every device that’s Ethernet or serial enabled. Drives, Motion Control Systems, Operator Interface Panels are on our radar screen. And we can access IP Cameras - imagine having a couple of low cost cameras streaming video of machine movement. No more questions like, did that cause the arm to move? Instead we pan a camera over toward the arm, look for ourselves and save man-hours of pointless phone calls. Pretty cool? I think so.
 
Our imaginations are the only limitation now that we finally have a secure, high speed remote connection. There’s something here for everybody. OEMs and Systems Integrators save engineering time and thousands in out of pocket expense. End users get more up time and greater productivity because proactive maintenance is affordable. We engineers see more of our kid’s ball games. We reduce our carbon footprint by eliminating unnecessary flights. Remote access may finally be a real deal.

Did you enjoy this great article?

Check out our free e-newsletters to read more great articles..

Subscribe