A study on making best use of existing technologies
By: Daniel L. Fenton, Product Marketing Specialist, Controls & Software, Control Systems Technology, Phoenix Contact USA.
It is the author’s opinion that integration of the controls network and the IT network is inevitable. It became inevitable the moment the controls industry chose to use Ethernet as the medium with which to communicate data. The controls industry may choose to be dragged kicking and screaming into the modern communications era, or it can gracefully embrace the change. Embracing means the controls industry would be able to leverage the myriad rich, existing technologies that have been proven foolproof in the IT world. To be dragged kicking and screaming into the modern communications era would do a terrible injustice to those who have worked diligently to bring it about.
This could quite possibly add an entirely new facet to the fieldbus wars, which I hope have not been forgotten.
With that said, the controls world is going to be moving with an industry that has a definite consumer bias, with product development and release cycles of six months or less. In an industry where the average life expectancy of an automotive production line is eight years, it is impossible to expect the networks in an industrial setting to keep up with modern IT standards. Therefore, we turn our attention to the technologies that have existed the longest, with the most open standards and the very best support. These are the protocols we wish to use and keep, and this article highlights and explains some of these technologies.
Figure 1: With the inevitable fusion of the automation network with the IT network, controls engineers need a better understanding of some of the proven technologies and protocols already used in IT networks.
This article does not focus on the technical implementations of each piece of technology. Rather, it is assumed the reader will be using packaged solutions such as a function block for a PLC. These packages typically require only that the user specifies the relevant server to connect to, the data to be gathered and an activation bit. The particulars of each protocol and concept are, ideally, transparent to the user, and therefore it is not pressing that the user understands what is contained in each packet passed between the server and the client. As each protocol described in this article is openly documented and supported, a simple search on the Internet for the technical details will likely yield the relevant implementation details.
Keeping the Time
With the modern demands of industrial controls networks, it is imperative that we keep our clocks in synch across all of the devices in the network. Between the PLC, the SCADA system (PC or otherwise) and even the remote devices, many of these devices maintain real-time clocks (RTCs) on an Ethernet network for protocol support purposes. Therefore, we need to make certain that each of these devices synchs to one another lest the RTCs conflict, leading to packet losses and clashing time stamps. An extremely useful, and often implemented but forgotten, feature of modern intelligent devices is a protocol called Simple Network Time Protocol, or SNTP.
Prior to SNTP implementation on modern industrial devices, engineers were forced to utilize some form of messaging to pass integers relating to the appropriate pieces of the time and date across their respective industrial protocols. Although this worked, it was tedious and very prone to errors, operator or otherwise. Furthermore, the precision was insufficient due to the nature of the messaging protocol, since milliseconds, or sometimes even seconds, could pass between the time the message was sent and when it was received. However, without another way, this was the only method left to controls engineers.
Meanwhile, the same situation had occurred much earlier in the networking IT world. The solution for this was to develop a protocol that occurred behind the scenes of the operating system to make sure that all the computers on the network, and on the Internet itself, maintained clock synchronization. Also quickly implemented was time zone keeping, such that users could synch their clocks not only to the world time, but to the specific geographic time zone they inhabited.
Cue the new millennium and the rise of industrial Ethernet. Controls engineers, in addition to grasping the new concepts of Ethernet-based networks, are still struggling with the age-old question of synchronizing the time on everything on their network. Meanwhile, the savvier controls engineers are watching their Windows XP machines automatically grab the time from the Internet. Why can’t we do that, too?
Controls manufacturers listened. Now, almost all of the intelligent devices for industrial controls applications can utilize SNTP to update their onboard clocks. And while this is often all well and good for Internet-connected devices where we can quite simply update from a local SNTP server (Such as nist1-ny.ustiming.org for Brooklyn, NY), it is not intuitive for non-Internet connected devices.
So how is this done? Well, almost every controls network includes an industrial PC. An SNTP server is a device that functions as a provider of the time. An easy way to remember this is the device “serves” up the time. SNTP clients request the time from the server. Any industrial PC running a full operating system, be it a modern Windows (XP, 7, 8, Server) or Linux distribution (Red Hat, Fedora, Ubuntu, Debian), can function as an SNTP server.
There are also some PLC manufacturers who include the server functionality as part of their PLC package, although this particular functionality has varying degrees of ease of use. Utilizing SNTP, we can say with complete confidence that anything on this network, even if it is not Internet-connected, will have all of the clocks synchronized. In addition, the PLC will require very little programming, saving the controls engineer time, and therefore money.
Logging Data
Many applications require that controls engineers log the data they gather. Regardless of whether this is done to follow certain laws and regulations, or simply as a feature of a particular product, the fact is that data must somehow be stored.
In the past, before modern communications, there were many ways to do this. People would be paid to go to each machine and write down the numbers on a clipboard. Chart recorders were utilized where appropriate. Sometimes data wasn’t recorded at all.
Meanwhile, big companies at the beginning of the information age wanted better ways to track things such as sales, inventory and accounting statistics. The development of databases, where large sets of data could be stored securely and retrieved reliably, changed the landscape of the workplace. Soon, employees punched their data into a computer, which in turn entered the data into the appropriate place in the database. Gone were the warehouses of file cabinets, replaced by server rooms. The emergence of widespread networking allowed people to make use of this stored treasure chest of data, resulting in an entirely new era of sales analytics and marketing tools.
The sales and marketing side of this rapidly changed, but the industrial side needed to catch up. The ability to track inventory and electronic warehousing was the first result. But now, it is almost expected that we can see precisely where in the process a product under construction is and log that point, linking it to the operator responsible at that precise moment in time. Water/wastewater engineers expect to be able to wake up in the morning and make sure the system operated flawlessly overnight. Oil refinery management expects to have precise production numbers on a daily, monthly and annual basis, available at whim.
The question is, then, how is this transformation achieved? It started before the real emergence of Ethernet networks. Historians utilizing a serial connection were able to store and display trends and graphs of past data. A historian is really a database with a handy frontend (user interface). Historians were useful when Ethernet networks weren’t widespread or well enough understood to help move data into a true database. Now that Ethernet use is far more widespread, it is well worth the time investment to make sure data is stored in a more open, accessible format than previously possible.
This allows the controls engineer to worry about getting the data to one place (the database), so that if the data is needed elsewhere, the IT department can make those connections work without interfering with the rest of the controls system. It also reduces redundant storage of data, though in certain cases this can be helpful to prevent data loss in the event of a system failure.
Most modern SCADA packages allow connections and automatic use of a Sequential Query Language, or SQL, database. SQL is defined as a programming language, though its use for the controls engineer is narrowed to simply moving data back and forth between an SQL server, the device where the information is stored, and a PLC. Some PLCs allow direct connections with a SQL server, thus removing yet more middleware, allowing for a sleeker, less expensive system to be put in place. Once again, this means the modern controls engineer can save time and money. Time is saved because there is less time spent configuring different pieces of software, and money is saved because it does not take a powerful PC to run a SQL database.
Segregating the Networks
An industrial controls network is a unique application of the Ethernet medium. High uptime, deterministic data transfer and high data throughput are the hallmarks of an industrial controls network. Sacrifices in ease of connection (since connections ought to be static and unchanging), Ethernet infrastructure (extra switches, dynamic load handling) and sometimes security must be made in order to keep the industrial network as well maintained as possible. However, as the controls engineer has no doubt recently seen, the security aspect is quickly becoming part of the more important aspects of an industrial network. Plants cannot necessarily shut down to update the latest software revision, and there is a real lack of security expertise in the industry.
Some automation engineers think the best solution is simply not to connect the industrial network to the corporate network.
This is a mistake, since it is precisely this connection that will move the industrial network out of the 20th century and into the 21st. Not only that, but this solution, referred to as an “air gap” in the IT world, has been demonstrated over and over again to not be nearly as secure as one might initially believe. Therefore, the security implications of connecting to the corporate network must be taken into account.
If an industrial network has older, unrevised software, the threat of a malware infection via multiple methods of attack (referred to as “attack vectors”) skyrockets. A simple mistake on the part of a user on the corporate network, while not a big deal on that network, becomes a real problem on an unprotected industrial network. Malware can generate a large amount of traffic on a network and begin to activate watchdog timers on remote IO drops as the network infrastructure begins to lose its grasp on the traffic load.
In addition to simple security steps that can be taken to update the industrial computers (which are beyond the scope of this article), segregating the traffic between a deterministic, high-uptime, high-throughput network and an easy-to-use, user-biased and dynamic network will have the most benefit. Industrial hardware firewalls and routers are available to do precisely this. In the “Logging Data” section above, the database is described as the primary link between the corporate network and the industrial network. Putting firewalls on both sides of the database computer and the respective networks prevents uninitiated connections between the networks. The particular architecture creates what is referred to as a “demilitarized zone,” or DMZ.
A PLC or SCADA system on the industrial side of the network can easily access the database to log data, and the corporate IT infrastructure can do the same. However, an infected computer on the corporate side cannot connect to the SCADA computer. This computer could connect to the database and infect that computer, but since the firewall is in place between the database and the industrial network, the malware cannot initiate a connection to the SCADA computer. Thus, the database can be replicated and the infection wiped out without ever having to touch the industrial network.
Secure Connections over the Internet
The concept of being able to remotely connect to a network anywhere in the world is not a new one. It has long been utilized in the IT world to prevent IT administrators from having to drive into work at 3 a.m. to fix a software configuration.
The implications of a secure connection from anywhere in the world extend beyond nightmare troubleshooting scenarios, such as a remote PLC in the middle of a desert with only basic cellular coverage connecting to the primary information center (SCADA PC or database server) across the Internet. How is this accomplished in a secure, simple fashion?
The answer is the humble Virtual Private Network (VPN) connection, sometimes referred to as a VPN tunnel. This particular piece of technology allows two disparate networks separated by the cloud to connect to each other as though they are on the same local network, albeit with a substantial penalty in connection speed. This penalty is not particularly relevant when passing small pieces of data, such as a SQL command or a Modbus function call. It is now even possible to send programs to PLCs and monitor visualizations of industrial networks across such tunnels. These capabilities allow for truly unparalleled possibilities.
VPNs are typically established between two pieces of network infrastructure, such as a commercial router and an industrial hardware VPN solution. Alternatively, software VPNs are available that would allow a PC to connect to the remote network. Software VPNs are particularly useful for the traveling technician, allowing remote troubleshooting to one customer’s site while at another site for long-term support. This sort of flexibility means that a technician can, quite literally, be in two places at once.
Network Health and Awareness
Over the course of an automation project’s lifetime, it often becomes important to check on the lifeline on which the communication system lies — the network. On most Ethernet-based networks, such as Profinet, Modbus TCP and EtherNet/IP, normal network traffic is shared on the same line as the industrial traffic. Overloading one could cause the entire network to fail.
Figure 2: On most Ethernet-based networks, such as Profinet, Modbus TCP and EtherNet/IP, the normal network traffic is shared on the same line as the industrial traffic. Overloading one could cause the entire network to fail.
Once again, IT installations suffered the same problem. Starting up and properly implementing an office network is by no means an easy task, so IT workers created and implemented their own tools to help make the process as painless as possible. One of these tools monitors the “vital signs” of a network, so to speak. The protocol is called the Simple Network Management Protocol, or SNMP, and it can monitor everything from network load to the time it takes a packet to leave a switch.
There are two types of SNMP devices: agents and managers. Agents describe the software that runs on managed devices, such as switches, routers and sometimes PLCs. A manager is the software that runs the agents, communicating with and controlling them. Managers are typically run on a PC, but more and more often PLCs can be managers.
SNMP has several forms of messaging. The two most common are the traditional manager polling setup, in which the manager gets information from continuously polling network devices across a certain period of time, and an asynchronous SNMP trap, in which an agent will send a message to the manager with certain information, usually due to some issue within the network under the agent’s purview.
So what can an SNMP manager running on a PLC do for the industrial network? Well, with the proper network infrastructure and a well-programmed manager, the industrial network can be saved from some fatal network failures.
For example, a manufacturing plant is expanding its capacity. As additional devices are added, the network load begins to climb. The traffic from technicians and engineers on the plant floor, which isn’t strictly industrial messaging, might also climb. The switch running the backbone of the plant begins to become overwhelmed and drop packets, causing chaos in the manufacturing process. A proper SNMP implementation could monitor this in either of two ways:
1) the PLC manager polls the switch, looking for network load going over a certain threshold or even the count of dropped packets, or
2) the switch could send an SNMP trap after certain thresholds have been reached asynchronously.
The value of the second method would be that the SNMP information gathering would not normally be contributing to the overall network load issue. Regardless of how the information is gathered, the SNMP manager, in this case a PLC, now has a range of actions to choose from. The ideal situation is the agent supports Quality of Service (QoS) over SNMP. QoS allows network infrastructure to prioritize packets destined for certain ports. In simplest terms, the switch can prioritize industrial network traffic (Modbus TCP, Profinet, EtherNet/IP) over an employee checking their favorite social network or email.
Therefore, the manager would detect a network load issue at a certain switch and direct the switch to prioritize the industrial network traffic while simultaneously alerting the plant manager that a network load issue exists and steps should be taken to find a way to minimize the situation until a new topology or better infrastructure can be found.
Getting the Right Attention
Many automation projects operate 24 hours a day, seven days a week. Humans cannot always be around when something bad happens at a plant or remote site. Therefore, it is critical to figure out how to program a system to get the proper person’s attention in the event something bad occurs. Sometimes this is merely a simple case of blinking lights on an HMI or SCADA system. Often, however, more is needed.
For those times where the system might hiccup when no one is around to notice, many PLC suppliers have implemented SMTP, or Simple Mail Transfer Protocol.
SMTP simply allows for a PLC to send an email to a designated email server. The programmer sets up an email address with any email server. When the email is sent from the PLC, the receiving email server checks the password with the PLCs email server then allows the email to go to the addressee’s email server.
It is usually quite simple to select a list of email addresses that each contain a different title and program the PLC such that if one condition exists, the email is sent to one employee, and if a different condition exists, the email is sent to a different employee. This allows the PLC programmer to get the attention of the proper technician for the job, and with the ubiquity of smartphones, it should not be a problem to make sure the technician is notified.
Names Instead of Numbers
Much of the network architecture and topology stage of an automation project is devoted to the planning and assignment of IP addresses. Along with this stage usually come several spreadsheets of details concerning subnet addresses, gateways and other details that usually take the form of tables and tables of numbers, with a column describing what these numbers are devoted to.
People don’t particularly like working with numbers, which is why websites are no longer determined by IP address, but by domain names. Each of these names is linked to an IP address. Sometimes the IP address changes but the name does not, allowing Internet service providers (ISPs) to perform address maintenance without causing the website to lose traffic. The service providing these names is called the Domain Name Service, and it is a simple matter to set one up on a local network.
The benefits of this sort of setup are myriad. For example, instead of referring to certain remote devices by an IP address and risking a potential IP address conflict arising from statically defined IP addresses, the remote device can simply be referred to by its domain name, allowing the network to determine exactly which IP address hosts the remote device. The tables of numbers linked to each device are now obsolete, because the network handles the IP address assignment itself.
How exactly does a network assign IP addresses by itself? It does so the same way a PC or network printer is assigned an IP address in an office setting; that is, via Dynamic Host Configuration Protocol (DHCP). A router will typically come equipped with a DHCP server. Devices with DHCP configuration enabled will send a message (called a DHCP query) to the router for all of the network configuration information, including IP address and subnet. This is similar to Bootstrap Protocol (BOOTP), except BOOTP requires a Media Access Control (MAC) address to be assigned to an IP address before the device is connected. DHCP will assign a free IP address to any device that asks for it on the network.
What happens when these two protocols are combined for a controls system? It means it is no longer necessary to keep track of IP addresses on a network — merely the name of each device on the network. In fact, connecting to a running system would be made easier with more granular and descriptive maintenance messages.
Final Thoughts
Hopefully this article has clarified some particular use cases for IT standards. Perhaps it has provided some insight into common automation issues and simple solutions for them, or perhaps it merely provided some insight into what some consider to be the future. Regardless, it is clear that the further fusion of the office network and the automation network is inevitable. To this end, it is important to understand the vital link between these networks and how to quickly and easily accomplish this link.
