Building Industrial MQTT Networks at Scale with Edge Computing

Summary

MQTT (formerly MQ Telemetry Transport) consistently ranks as the most popular IoT-specific messaging protocol. However, there are several obstacles to bringing the power of MQTT to an industrial environment.

MQTT’s innate flexibility requires stronger guarantees of interoperability and state management to meet the needs of a diverse industrial network. Likewise, while MQTT provides a framework for large-scale integration and addresses fundamental cybersecurity issues, MQTT by itself cannot create a secure Industrial Internet of Things (IIoT) infrastructure.

For system integrators, developers, engineers, and managers wondering what MQTT can do for them, this article explains the protocol’s fundamentals, demonstrates how the Sparkplug B specification adapts MQTT to industrial applications, and shows how to establish and scale MQTT networks using industrial edge computing.

Fundamentals of MQTT v3.1.1

Traditional communication models use a poll-response mechanism to interrogate field devices and maintain state awareness. Data is refreshed when a master device or application requests new data, causing field devices to respond with the requested data. When IBM and Arcom Control Systems developed MQTT, they abandoned this model in favor of a brokered publish-subscribe model—MQTT’s central strength.

In that scheme, a central server acts as a broker, managing data delivery for the entire network. Rather than being prompted by command, MQTT-enabled devices publish data to the broker only when a monitored value changes, a behavior called report by exception.

Because some devices may not publish very often, state awareness is maintained by periodically sending a small keep-alive packet to the broker, plus other mechanisms that we’ll discuss later. Other clients, including software and other field devices, can register with the broker as subscribers to any data published on the network.

Flexible topic paths
Although each client identifies itself to the broker with a unique client ID, there is no device addressing scheme. Instead, clients identify published data using individual, hierarchical topic paths represented as plain text strings. For example, “CellA/Oven/Temperature” could represent the internal temperature of an oven in a particular work cell. Single- and multilevel wildcard characters (+, #) provide more flexibility.

The broker stores these topic filters with each client’s network session and routes matching updates from publishers to all matching subscribers: one-to-one or one-to-many. This system is something like Twitter for machines: a free flow of anonymous, interest-based, eventdriven, bidirectional communication.

Efficient message payload
Both MQTT and HTTP are commonly used as IoT messaging transports, but while HTTP is famously heavyweight, MQTT is data-agnostic, with a streamlined on-the-wire footprint that devices with limited power and processing capabilities can process efficiently.

MQTT uses a simple byte array payload with a fixed 2-byte header and variable-length header fields (a few additional bytes) to indicate packet length or control codes. A packet is up to 256 MB in size and can transport anything from process variables to your pet’s photo.

Fault tolerance mechanisms
MQTT’s designers also included features to gracefully handle unintended client disconnections and ensure data delivery to subscribers:

▶ Messages can be optionally transmitted at higher qualityof-service (QoS) levels that carry more overhead but have a stronger guarantee of delivery.
▶ Published topics can be flagged as retained messages. The MQTT broker keeps the most recent update available for new subscribers.
▶ If a client connects using a persistent session, the MQTT broker stores its configuration and high QoS messages should it disconnect unexpectedly.
▶ Each client can also register a special last will and testament (LWT) message with the broker, which is distributed to subscribers should the client connection be interrupted unexpectedly. The client’s keep-alive timer determines when the broker considers the connection lost.
 

Advantages of MQTT

Efficient communication
Combined with its streamlined payload, the MQTT communications model delivers an 80–90 percent reduction in bandwidth consumption compared to poll-response models, according to coinventor Arlen Nipper. This efficiency creates room for existing networks to grow—up to millions of connections.

Decoupled data
But it is the decoupled nature of MQTT data exchange that unlocks industrial network scalability. Without point-to-point connections or direct addressing, MQTT networks can grow and share data flexibly. Any client wanting access to published data—a maintenance database, enterprise resource planning, or a supervisory control and data acquisition (SCADA) system, for instance—can simply point to the MQTT broker and subscribe to any desired topics, without needing details of the publishing source.




Security
MQTT’s basic architecture also improves IoT security. The broker alone manages client authentication, data access rights, and message delivery, simplifying network management and allowing clients to remain anonymous. And because MQTT connections are deviceoriginating (outgoing), only the broker requires open firewall ports. Field devices can be completely locked down while still permitting bidirectional communication.

MQTT also supports username and password requirements but usually relies on transport layer security (TLS) encryption (port 8883 is registered for MQTT TLS). With certificates of trust authenticating connected endpoints, secure site-to-site MQTT communication is feasible even over public networks.

Faster, better with Sparkplug B

Although this simple framework allows MQTT to support many kinds of IoT applications, configuring the large, diverse networks characteristic of industrial installations can require significant effort akin to mapping conventional tag data between applications. Further, there is no guarantee that a device vendor will use the features available to mitigate connection risks and maintain data quality.

These weaknesses inspired Arlen Nipper and Cirrus Link Solutions to develop the open-source “Sparkplug MQTT Topic Namespace and Payload Definition.” The current version, Sparkplug B (SpB), expands on the basic MQTT architecture to address common industrial use cases and adds important implementation details for MQTT clients.

Components of an MQTT/Sparkplug B network
Sparkplug adds to and clarifies the roles of basic MQTT clients, enabling new features and more explicit messaging that are the basis for other enhancements in the specification. The specification distinguishes between two essential types of MQTT clients:

▶ MQTT/Sparkplug B Edge of Network (EoN) Nodes: These clients provide physical and/or logical gateway functions to enable MQTT/ Sparkplug B communications for legacy devices and sensors. EoN nodes also include smart devices and sensors capable of publishing their own Sparkplug B data, process variables, or metrics directly to an MQTT broker.
▶ MQTT/Sparkplug B Application Nodes: These are software clients, optionally including one primary application responsible for sending commands and receiving historical data. An MQTT/ SpB application node may also be a gateway to legacy software systems.

Any MQTT 3.1.1–compliant broker supports Sparkplug B payloads, and one or more can be used for redundancy, high availability, or scalability

Topic namespace
Sparkplug defines a standard format for MQTT topic paths:

spBv1.0/<Group ID>/<MESSAGE TYPE >/<Edge Node>/<Device ID>



Payload definition
Sparkplug B also defines a standard, structured, data-rich yet efficient payload format. The core payload contains an array of metrics and associated metadata as well as optional fields and custom properties for each metric: name, description, data type, engineering units, scaling limits, and more. Data type indicators accommodate complex types common in industrial applications, like matrices and user-defined types (UDTs). The full payload is then time stamped, sequenced, and encoded using an efficient binary representation of the structured data. The decoded payload is typically represented in JavaScript Object Notation (JSON).

State management
Sparkplug defines birth and death certificate messages to enforce state monitoring. Death certificates use MQTT’s LWT message option and the special message types NDEATH or DDEATH, indicating whether the certificate pertains to a node or to an attached device, respectively. When first online, each client must also publish birth certificates (message types NBIRTH or DBIRTH) for itself and its attached devices. Birth messages define all available topics for that client/device and inform all subscribers that it is online.

Primary application
While an MQTT network may contain any number of application clients, the Sparkplug specification allows one application node to be designated as the primary application, typically a SCADA or IIoT host that sends commands and logs historical data. It publishes birth and death certificates using the unique topic STATE/ with elevated quality-of-service and retained message status, so any MQTT client can identify its state at any time.

Advantages of Sparkplug B

Low administration
Because a birth certificate is required from each client whenever it connects to the network or updates its topic structure, subscribers can automatically map out available MQTT topics in a few moments. And thanks to Sparkplug B’s payload structure, certificates include all metadata and UDT definitions. No more tag mapping.

Interoperability
The Sparkplug B specification addresses the potential for inconsistency in MQTT implementations by defining standard client roles and data interfaces designed around industrial applications. By standardizing on Sparkplug, MQTT clients from different vendors can identify, interpret, and use published data without needing to know details of the source. Even legacy automation networks and devices can leverage Sparkplug because of its explicit support for gateway-attached devices.

Unified namespace
This level of interoperability satisfies a critical requirement for digital transformation by making it possible to have an enterprisespanning namespace defining all business data. Enterprise clients can consume data from the field via the broker, using Sparkplug to provide a common exchange format and uniform context, then publish results back into the network to be detected, interpreted, and acted on by other clients.

Enhanced data integrity
By requiring LWT messages, Sparkplug ensures all subscribers are notified when data becomes stale. Likewise, birth certificates let subscribers know when data is fresh again. And the primary application mechanism creates new failover options to avoid critical data loss.

All clients can detect when the primary application’s connection is interrupted and search any alternate connections for a broker showing an “ONLINE” status for the application. Clients with storage capabilities can also use store-and-forward historization to queue up messages, flag them as historical data, and publish them into the network once the primary application is available again.

Building mission-critical MQTT networks

Despite MQTT and Sparkplug B’s inherent advantages, they cannot function without robust support from all network clients. Sparkplug’s features, like legacy device integration and store-and-forward historization, cannot be realized without field device clients that implement the necessary gateway and storage functions.

And fundamental network reliability is undermined unless every network participant implements complementary cybersecurity and connectivity features. Even the sheer volume of data a large IIoT network generates can be overwhelming unless clients can filter and sanitize data locally before publishing.

Unfortunately, traditional industrial devices were designed for a narrow scope of operation and often lack general-purpose processing, communication, and storage.


Industrial edge computing
Many industrial automation vendors are drawing inspiration from information technology to address these gaps, turning to an approach known as edge computing—a form of distributed computing that moves computing resources closer to high-demand areas on the network’s edge. With more resources at the edge, data can be prepared locally before broad distribution; modern security measures can be layered onto legacy systems; and advanced functionality can be embedded in the process to support a resilient network.

Opto 22’s groov devices support edge data processing with embedded tools like Node-RED, IBM’s free, open-source IoT platform, and Ignition Edge from Inductive Automation. They provide modern IT security options like user authentication, TLS encryption, and device firewalling. And they support store-and-forward historization with local fault-tolerant storage. All groov devices provide robust MQTT/Sparkplug B client options, designed in collaboration with industry leaders like Cirrus Link.

From concept to cash
Using edge devices like these, users can build a secure MQTT/Sparkplug B network following a basic pattern:
1. First, instrument assets and connect I/O to an edge device like groov EPIC.



2. If controls already exist, connect them to a segmented network interface on the EPIC, and use OPC UA drivers to retrieve data.
3. Then, the EPIC can open a secured connection to an MQTT broker, ideally in the corporate network’s demilitarized zone (an isolated network that bridges internal and external networks).
4. Now, plug applications into the infrastructure. A gateway application like Cirrus Link’s MQTT Engine can integrate legacy applications, if needed.
5. With this foundation, the system can scale up using additional edge devices or brokers to build out a highly resilient, high-performance data exchange and command-and-control system.

By relying on a flexible, efficient, open-source standard like MQTT/ Sparkplug B, this design reduces licensing and configuration costs, allows users to pick from a variety of components, provides multiple options for scaling up, and delivers subsecond response times even across global networks.

Complete solution

MQTT is popular, proven, and well supported in enterprise and consumer applications. It makes organization-spanning data exchange possible by decoupling data producers and consumers and by defining a lightweight, data-agnostic communication format supporting millions of connections. But its usefulness for industrial applications is challenged by various interoperability and reliability risks.

Sparkplug B defines an MQTT implementation standard that ensures client interoperability, delivers usable data for operations, gracefully handles instability, and helps organizations scale by reducing administrative overhead. Industrial edge devices complete the architecture by providing physical layer reliability, integration options, and sufficient computing power to process and transmit field data efficiently and securely.

The combination of MQTT, Sparkplug B, and industrial edge devices forms a complete solution for building and scaling connected IT/OT data networks. It is open, flexible, and powerful enough to support the requirements of any connected application.

All figures courtesy of Opto 22.

This article comes from Volume 1 of the Automation 2021 Ebook: IIoT & Industry 4.0.