Open Automation Systems: Update on the State of the Art

Summary

In the world of industrial manufacturing, where operational technology (OT) systems control, automate and keep mission-critical manufacturing processes safe, there has been a strong and steady push to adopt more open automation systems by employing modern software technologies that are already mainstream in the IT industry.

Some of the characteristics of these new open process automation systems are:

• Standardized communication protocols for improved interoperability of systems.
• Automation software that is decoupled from the hardware on which it executes, meaning the software applications essentially become vendor-independent. 
• Object-oriented technologies for efficient creation and re-use of software component libraries.
• Event-driven architectures that simplify the integration of real-time automation systems and applications with other enterprise applications, such as analytics, asset management systems and resource planning systems.

Multiple benefits can be gained from these more open and modern approaches. To name just a few: 

1. Industrial manufacturers are better able to efficiently and more cost-effectively manage control system hardware end-of-life and obsolescence issues.
2. Better return-on-investment as enterprises digitize and invest more-and-more in advanced application software.
3. Using “proven-in-use” software components and standard IT infrastructure management tools increases the reliability of plant assets, as well as the cybersecurity and safety of the entire operation.
4. Using best-in-class “Plug & Produce” software components increases innovation over the lifecycle of controlled assets. 
5. Enabling modular plants, modular machines, remote operations and other flexibility makes operations more agile and the global supply chain much more resilient. 
6. Last, but not least, attracting young software engineers into the automation profession to run the plants of the future with more modern technologies and architectures.

This article summarizes the initiatives related to open automation and describes the technologies behind UniversalAutomation.org open automation architecture and the ExxonMobil Open Process Automation Forum (OPAF) test sites.

Several major ongoing industrial initiatives/organizations today promote this new world of open automation software and the portability of industrial control applications. A few examples are: The Open Process Automation Forum, NAMUR, OPC Foundation and UniversalAutomation.org

Open Process Automation Forum. In late 2016, the Open Process Automation Forum (OPAF), an industry consortium, was formed by end users, vendors, suppliers and academics within The Open Group to advocate for the definition of a highly standardized reference architecture for process automation systems, thereby allowing for the modular integration of products from multiple vendors into a single control system. This modular standardized architecture is achieved through the incorporation of existing standards and specifications wherever applicable, making the Open Process Automation Standard (O-PAS) a ‘standard of standards.’
 
The diagram (Figure 1) outlines a sample control system architecture using O-PAS components. A new OPAS system is centered around the use of Distributed Control Nodes (DCNs) which connect to a standardized and secure network backbone, the O-PAS Connectivity Framework (OCF). A DCN can provide a controller with logic-processing capabilities, an I/O module or both (similar to a traditional PLC or DCS controller with attached IO).

• Development and maintenance of specifications
• Certification and compliance testing of implementations
• Cooperation with other standards organizations.

The OPC Foundation has launched the OPC UA FX (Field eXchange), an initiative that will further enable OPC UA adoption by covering the use cases and requirements for the field level.

The goal of this initiative is to deliver an open, cohesive approach to implementing OPC UA, including time-sensitive networking (TSN) and associated application profiles. This will advance the OPC Foundation by providing vendor-independent end-to-end interoperability into field-level devices for all relevant industry automation use cases. The OPC Foundation’s vision is to become the worldwide industrial interoperability standard by integrating field devices with the shop floor.

UniversalAutomation.Org. UniversalAutomation.Org (UAO) is a not-for-profit, international association established in 2021 to drive a vision of “plug and produce” automation using vendor-independent software components (think app store) that are enabled by the IEC61499 standard. 

The association is growing rapidly: As of June 2024, it has 89 members from the user, vendor and academic communities. UAO believes “open automation” as it exists today is not open enough. It believes interoperable and portable application software is an essential enabler for Industry 4.0. In practice, UAO is a combination of two things:

1. A standardized automation layer across vendors, based on the IEC 61499 standard, in the form of royalty-free license and runtime execution engine source code, which is available to its members on GitHub.
2. An ecosystem of members from the user, vendor and academic communities, who are committed to driving the adoption of the UAO runtime.

UAO offers are available on the market and are being deployed by end users across various industry segments. For example, ExxonMobil and several other OPAF members use the UAO runtime as one of the components of their open-process automation architectures.

Universal automation will allow OEMs, integrators and end users to build automation solutions by plugging together best-of-breed apps using no-code graphical tools. In industry, we call this “plug and produce."

Much in the same way that consumers can easily access the latest mobile phone technologies and apps, industrial stakeholders, with plug-and-produce applications, will be able to experience ease of use at lower costs through much simpler, less labor-intensive behind-the-scenes integration.

Open systems performance

Proprietary systems have done a great job bringing automation to where we are today. Defenders of proprietary systems argue they are required to achieve a high level of real-time performance and determinism.  While this may have been true 15 years ago, when compute power was expensive and open-source software systems were less developed, it is no longer true today. According to Moore’s Law, computer power continues to double every 18 months, and open-source software, like Linux, is already running mission-critical enterprise systems.

Linux operating system. In the past, manufacturers typically used commercially available proprietary real-time operating systems. In parallel, the use of open-source Linux operating systems continued to expand, driven by the growth of the Internet and the associated data centers, to the point where the majority of internet traffic flows through Linux-powered systems. Linux is now being used extensively in consumer and industrial appliances that have embedded compute power.

To meet the stringent requirements of control systems and other hard-deadline computing, a real-time patch for the Linux kernel scheduler was developed, prioritizing predictable performance. This has found use in industrial automation applications, as well as other embedded applications, providing the deterministic foundation for Linux to operate in control system environments.

Today, several industrial automation companies use Linux as the OS for their real-time control systems. Some examples include PLCNext from Phoenix Contact; ctrl X OS from Bosch Rexroth; ROS 2 (Robot Operating System 2). Kernel-RT patches are available from vendors such as Canonical and Red Hat.

Because the real-time scheduling patches simply change scheduling behavior, all the other benefits brought by Linux are available to industrial workloads, such as portability, customization and process isolation.

In addition, Linux is being used as the foundation for next-generation offerings to meet or exceed existing embedded solutions. For example, Linux-based certified functionally safe systems are currently in development. Certification in both the industrial and automotive markets will be available for general consumption before the end of the year.

Some vendors have even built high-availability (HA) solutions on top of a Linux platform, which when combined with network-based I/O, allows for failover to happen exclusively at the application level, without proprietary, expensive hardware.

Automation platforms like PLCNext from Phoenix Contact and crtl X OS from Bosh Rexroth also promote the use of other apps programmed in multiple languages. And they even provide an app store with the possibility to download and run other apps alongside the real-time control apps. 

These systems typically have a common data layer through which different apps can access the system IO. Apps can be traditional control programs e.g. IEC 61131, or more modern programming languages such as C++ and Python. Mechanisms are in place to segregate real-time control applications from “right-time” applications, allowing the programmer to achieve the desired level of performance and determinism. 

The Bosch Rexroth ctrlX OS platform combines Linux with EtherCAT to address the very demanding multi-axis motion control domain which requires a very high level of performance and micro-second determinism.

The argument that Linux is not a “real-time OS” is clearly no longer true, as Linux has proven its deterministic capabilities, and is trusted in mission-critical applications by many different organizations, with more choosing it as the foundation of the next iteration of their solutions.

Event-driven protocols and programming. Typical automation systems in the past used scan-based execution models and client/server or request/response communication models and protocols to ensure that the behavior of the control system was predictable and deterministic. 

Enterprise IT applications on the other hand had long ago switched to change/event-driven execution models and change/event-driven communications and protocols to leverage and monetize the benefits offered by the change/event-driven approach.

In the recent past, IT-OT integration has received significant traction among end users seeking to integrate more and more their control systems with their enterprise applications. In these examples, they need to embrace and use the change/event-driven approach both in control execution and communications. A good number of control systems in the market today already support and use change/event-driven communication models and protocols in their products.

Additionally, a good number of modern protocols, like OPC UA and MQTT, Ethernet IP and Profinet, inherently support event-driven communication models. All or most of these protocols are already in use and supported in control systems in today’s market. In line with the above developments, UniversalAutomation.Org (UAO) takes this one step further by applying the event-driven architecture to real-time control execution as defined by the IEC 61499 standard.

To guarantee performance/determinism for demanding applications, resource-based prioritization is part of the UAO runtime. This allows a high-priority resource to interrupt lower-priority resources, ensuring that tasks requiring a rapid response time are executed in a predictable and deterministic manner. 

This is nothing new. Even scan-based systems have similar mechanisms to ensure that high-priority tasks can interrupt the general scan as and when required. The introduction of function blocks programmed in high-level languages such as C++ adds to the challenge of guaranteeing determinism, even in scan-based systems. 

Schneider Electric, one of the founding members of UAO recently demonstrated the deterministic performance of event-driven systems well below the millisecond level using the UAO resource prioritization mechanism on a Raspberry Pi running Linux. Despite running event chains that required more than 50ms to complete (matrix convolution calculations), higher-priority resources were able to interrupt the longer-running resource to execute the higher-priority tasks. A deterministic response in the range of a few hundred microseconds was achieved (additional technical details can be found here). 

In addition, event-based systems can implement “cyclic” tasks just like scan-based systems can. Reading of physical inputs is typically programmed as a high-priority cyclic task. 

Automation engineers with a background in traditional PLC programming indeed find it easier to understand scan-based systems compared to event-driven systems. However, the converse is also true: Young software engineers with experience in high-level languages struggle with scan-based systems.

At the end of the day, at a low level, all systems are event-driven. Users must take care to ensure that their performance/determinism requirements are met. 

From theory to practice: ExxonMobil OPAF testing

In 2016, ExxonMobil (an OPAF founding member) entered into a development agreement with Lockheed Martin to begin a proof of concept for an open automation system architecture. In early 2017 they successfully used the IEC 61499 standard as a component for achieving that goal. Since 2018, several end users, with vendor support, have demonstrated their proof of concepts, prototypes and test beds using IEC 61499-based control algorithms. 

ExxonMobil has extensively used IEC 61499-based control logic in their Open Process Automation Program since early 2017. The UAO Runtime has been tested extensively to prove that it has the capabilities to create reliable and predictable control logic applications.
 
[optional pullquote] This demonstrates that an open, standards-based, secure and interoperable automation architecture will elevate the end user’s ability to improve business performance and success through modern technologies.
 
The first step for any user of IEC 61499 is to become familiar with the event-driven nature of the logic tools and the Execution Control Chart (ECC) capabilities. Once the user realizes that the system clock and a scheduled interval (e.g., 100ms, 1sec, etc.) can be used as the event, the leap from purely time-based scan typical of IEC 61131-based products to event-driven execution in IEC 61499 is straight forward and enlightening. The freedom and independence from a specified, cyclical schedule offer new options for how to schedule and configure control logic applications that are more tailored to the actual process automation requirements. 

The standard of measure for companies like ExxonMobil includes the ability to safely automate, manage and operate highly energetic chemical processes that are typically found in hydrocarbon refining and chemical manufacturing operations. The key criteria used for evaluating control systems and components include determinism, scalability, reliability, failure modes and recovery, as well as a robust capability to implement existing and trusted process control code and methods, including the creation of new and novel control algorithms.  

The usage of control logic in the UAO Runtime has been demonstrated and tested down to 1ms time cycles in a pilot plant prototype. While the responses from physical equipment, like valves and pumps, to control signal changes suffer from delay and deadtime that make 1ms timing impractical for general usage, the goal was to test whether control logic and the UAO Runtime based on the IEC 61499 standard, could be used for some types of rotating machinery timing requirements.

The integration of basic control logic with Model Predictive Control (MPC) was also tested, demonstrating several orders of magnitude faster response (from 15sec to 40 msec) than in a legacy automation system. The advanced control algorithms were operated through connections to the PID logic blocks and directly communicated to the output I/O channels.  

Several end user companies have successfully repeated these use-case demonstrations in their prototypes and test beds and have gained confidence that IEC 61499-based products offer an excellent, standards-based choice for automation in an open architecture.

The performance and capability of the IEC 61499 standard have convinced ExxonMobil to use the UAO Runtime in control logic applications for their previously announced Open Process Automation Program’s Lighthouse Project, which is scheduled for commission and startup at the end of 3Q24.

 The project will replace an automation system composed of PLCs and DCS equipment with an OPA-based automation system to support the on-going operations of an existing manufacturing plant. This open system integrates the features of IEC 61499 and the UAO Runtime, OPC UA, DMTF Redfish, IEC 62443 security, container-based execution, Kubernetes/Orchestration and existing control strategies and systems to create new levels of capability. This demonstrates that an open, standards-based, secure and interoperable automation architecture will elevate the end user’s ability to improve business performance and success through modern technologies.

High availability. In addition to determinism, a key requirement in the process industries is high availability. Processor failure is a critical event that must be addressed immediately to ensure process safety and continuity. 

The traditional approach to achieving high availability has been to use a primary and secondary processor with a high-speed data link between them to share process state data. If the primary processor fails, control of the process I/O systems is handed over to the secondary processor. During the transfer, the current process states are maintained achieving a bump-less transfer.

This approach has two drawbacks:

1. It relies on very specific and expensive proprietary hardware.
2. The operator must schedule a maintenance order to replace the failed processor and manually re-establish the secondary backup.

Schneider Electric and Red Hat overcame these drawbacks by implementing a software-defined high-availability scheme using the RedHat for Edge Linux distribution and the Schneider Electric Soft dPAC HA solution (a containerized version of the UAO EcoRT). This high-availability solution is fully software-defined, meaning it is no longer hardware-dependent. Soft dPAC HA provides the bump-less transfer of the control load automatically as soon a failure is detected and transfers process state data over a high-speed Ethernet connection. 

In addition, because of the software-defined nature of the solution, Schneider Electric and Red Hat were able to redeploy the application to another compute resource on the network, thereby automatically replacing the failed processor with zero human intervention. This is possible as long as the network compute resource has the capabilities to run the workload to be deployed. 

This approach to high availability ensures process continuity without human intervention and allows the customer to remove obsolete hardware and software without process interruption.

Conclusion

The long march to open automation systems is underway driven by user demands for more innovative solutions and the continuing acceleration of technology that has tended to leave proprietary automation systems lagging in its wake. The use of an open technology like Linux on automation platforms has already become mainstream and open automation architecture initiatives like OPAF are fully embracing emerging hardware-independent technologies like UniversalAutomation.org and OPC UA.

Concerns over performance have proven to be unfounded because new, open technologies have been successfully executing demanding applications.

So the question is not whether open systems will become mainstream. The question is how long will it take for open systems to become the dominant automation architecture. For many end users across industry segments, who already trust the determinism, availability and flexibility open, IEC61499-based offers provide, the answer is now.

_{This feature was originally published in AUTOMATION 2024: 9th Annual Industrial Automation & Control Trends Report.}