Diagnostics with EtherCAT - Part 1

July 2014 - Besides features such as high performance or speed, the availability of network diagnostic tools is extremely important when it comes to selecting a communication technology for industrial environments. EtherCAT comprises a wide range of system-inherent diagnostic features which help detect and locate system errors precisely.

Basically-speaking, the diagnostic features of a fieldbus should meet two general demands:

Fast response
The control system should be able to immediately detect an anomalous condition in the network, in order to react as fast as possible and to prevent mechanical damage, loss of materials, or loss of synchronicity within the application.

Precise analysis
The network should provide the controller with detailed status information, enabling the master application to determine if errors are caused by EMC disturbances, by a hardware failure of a device, or by a firmware behavior.

Due to its protocol properties and its slave structure, EtherCAT fulfills both requirements equally.

The EtherCAT frame: structure and processing
EtherCAT makes use of standard IEEE 802.3 Ethernet. The payload of EtherCAT frames transports a variable number of so-called EtherCAT datagrams (Fig. 1).

Fig. 1: Structure of an EtherCAT frame

Each EtherCAT frame propagates through the slaves in a distinctive order, providing output data and collecting input data, and then returns back to the master (Fig. 2).

Fig 2: Processing order of the EtherCAT frames

On the slave side, frame processing is performed by a dedicated component called an EtherCAT Slave Controller (ESC) (Fig. 3).
 

Fig. 3: Internal structure of the EtherCAT Slave Controller (ESC)

The frame routing is done by the ports, as shown in Fig. 3, while the EtherCAT Processing Unit reads and/or writes data from/to datagrams and the slave’s memory. The ESC memory space is used for both application data (process data, mailbox) and registers: several standard registers are defined in this area for both hardware and firmware diagnostics.

Each datagram specifies which operation is performed on the slave’s memory by means of a command code. Fig. 4 shows the datagram structure.

 
Fig 4: Command code in the datagram

EtherCAT defines 15 different standard command codes:

Each EtherCAT datagram, and thus each EtherCAT command, has a so-called Working Counter. With this Working Counter, the master application can check within the same cycle if the data exchange was successful. Therefore, the Working Counter is a very effective monitoring mechanism.

Working Counter
The Working Counter is a 16-bit field placed at the end of each EtherCAT datagram (Fig. 5).
 

Fig 5: Working Counter field at the end of a datagram.

When a frame is sent by the master, the Working Counter is zero for all EtherCAT datagrams. Frames then sequentially encounter all the slaves. If a slave is addressed by a specific datagram, and if the corresponding command was processed correctly (that means if slave memory could be read/written successfully), the associated Working Counter is incremented in a well-defined way:

Based on the network configuration, the master knows the theoretically expected value of the Working Counter for each datagram and can compare it to the value actually received in the incoming frame. Working Counter errors can be caused by different reasons preventing one or more slaves from performing a read or write action on the ESC memory: when the slave is switched-off or physically disconnected from the network, the slave registers an internal firmware error and thus moves to a state in which some read or write memory accesses are blocked, or the EtherCAT cycle time was too fast with respect to the minimum internal firmware cycle time.

A Working Counter mismatch does not directly clarify the reason for a problem: it states only that one or more of the addressed slaves did not process a datagram properly. Though the information is a direct part of the datagram structure, it is received cyclically and synchronously with the communication cycle. Therefore, it is immediately available for the master application; the cause for error can be further investigated by the master device by means of specific register address procedures.

Being a physical field within datagrams, Working Counter information can also be captured directly by network analyzer traces such as Wireshark scans (Fig. 6).

 Fig 6: Working Counter within a Wireshark scan

A Working Counter mismatch is particularly critical for datagrams carrying process data because in this case the network slaves would not work with synchronous data. The master can react immediately in the most suitable way for the specific application, in order to prevent mechanical damage, waste of valuable or potentially dangerous materials, or misleading information provided to the user interface.

In the case of less critical acyclic commands, a Working Counter mismatch indicates to the master that the parameter data were not successfully exchanged, and thus that it must be sent again. Therefore, the master has always a reliable indication on whether data actually reached the recipient or not.

Example 1: Motion control system
Motion control applications typically consist of several axes working synchronously with each other (3-dimensional interpolation). At hardware level, each axis corresponds to a motor drive with an EtherCAT slave interface. Each drive exchanges both output and input process data (control word, set point position or velocity, status word, actual position), so Logical Read Write commands are typically used. Usually a single datagram is used to exchange process data of all the three drives. If an internal hardware error occurs in one of the drives, like a short circuit in the power stage, the slave leaves the Operational state and stops processing cyclic data. The motion control application immediately detects that the Working Counter associated to the datagram is no longer received with the correct value and can stop the other axes involved in the interpolation task (Fig. 7).
 
Fig 7: Cycle-synchronous reaction of the master application

Example 2: Loss of synchronization
Distributed Clock technology is used in applications that require very high synchronization degrees among EtherCAT slaves: all slave clocks are aligned to the same time reference and generate synchronous events within each slave. For proper operation, the synchronous event triggering a process data update should always follow the reception of the frame carrying new data values. However, frame transmission timing can jitter according to the master implementation and, therefore, the frame carrying new output data could reach the slave after the internal sync-event. Every time this condition occurs, the slave increments an event missed counter. If the counter overcomes a specific user-defined threshold the slave falls back to SafeOperational state. The PLC program can immediately detect the change in the Working Counter of the datagram carrying output process data which is no longer processed by the slave and manage the loss of synchronization within the software code in the most suitable way for the specific application. Additionally, the slave provides the reason for the error in a defined register and thus enables the master to offer the user helpful information for error handling (Fig. 8).

 
Fig 8: Cycle-synchronous detection of synchronization error

Example 3: Detection of topology changes
In order to cyclically monitor changes in the network topology that could be caused by hardware failures or disconnection of parts of the slaves, Working Counter information of broadcast datagrams can be used. Broadcast datagrams are processed by all network slaves: therefore, the theoretical Working Counter value expected for those datagrams is always directly proportional to the number of configured slaves. If a part of the network is disconnected or switched off, not all configured slaves will process the frames and thus the received Working Counter value of broadcast datagrams will decrease. This event can be used by the master to trigger polling the link status registers of each slave, precisely detecting where the network topology changed. Fig. 9 shows the Working Counter field before and after the topology change (Fig. 9).
 
Fig 9: Cycle-synchronous detection of a topology change

Conclusion
With its Working Counter concept, EtherCAT provides a powerful diagnostic mechanism that can be used by master applications to monitor the network status synchronously with the communication cycle. Master applications such as PLC programs can therefore react immediately to any error condition within the network with a simple evaluation of the Working Counter in the application software. Precise detection of specific errors is possible by accessing the standard diagnostic registers provided by each slave. The description and usage of these registers will be described in the second part of the article.