Industrial events and alarms: delivery, persistence and recovery

A gateway loses its connection for two minutes. In that time a protection relay raises an alarm and clears it again. When the connection returns, does the gateway know?

The answer depends on the protocol, the configuration and the device, and it ranges from “yes, with the exact time of both changes” to “no, and there is no way to find out”. That is the central question of this article: when something changes while we are disconnected, can we find out afterwards?

It is part of the series that starts with 14 industrial protocols, 5 communication patterns.

Measurements, state changes, events and alarms

Four words are often used as if they meant the same thing.

• A measurement is a value at a moment: 47.3 kW. The next one replaces it.
• A state change is a transition: the breaker went from closed to open. Losing it means losing a fact about the world, even if the current state is known.
• An event is a record of something that happened, usually with a time stamp and a cause. It is meant to be kept, ordered and processed once.
• An alarm is a condition that needs attention. It has a lifecycle of its own: active, acknowledged, cleared, and sometimes shelved or suppressed.

A protocol that delivers measurements reliably may still lose state changes, and a protocol that delivers events may still have no notion of an alarm lifecycle. Asking “does it support events?” is not precise enough. The useful questions are: is a change delivered, is it kept if delivery fails, for how long, and can the receiver tell that it missed something?

Change-of-value versus event generation

BACnet has both mechanisms, and they behave differently. A COV notification reports that a property changed, to whoever subscribed; it is about the current value. Event notifications are generated by an object’s event-reporting logic, intrinsic or algorithmic, when a value crosses a limit or a state becomes abnormal. They are sent to the recipients listed in a Notification Class object, carry a time stamp, a priority and an event type, and say whether an acknowledgement is required.

OPC UA separates the two the same way. A monitored item on a variable produces data change notifications. A monitored item on an object with an event filter produces event notifications, and the Alarms and Conditions model adds conditions with states, severity, acknowledgement and confirmation.

The distinction matters for recovery. A data change can be replaced by a fresh read. An event cannot: once it is lost, the fact is lost.

Three acknowledgements that mean different things

“The alarm was acknowledged” can mean three unrelated things.

1. Transport acknowledgement: a TCP ACK, the acknowledgement slot of a CAN frame, a KNXnet/IP tunnelling acknowledgement. The bytes arrived at the next hop. Nothing more.

2. Protocol confirmation: the sender may now release its copy, but what it releases differs per protocol.

   • An IEC 104 acknowledgement (the receive sequence number in an S-frame or I-frame) confirms that I-format APDUs arrived within the current connection. The sender can drop them from its transmission buffer. It is not a confirmation of processing, and it does not carry over to a new connection.
   • A DNP3 application confirmation lets the outstation release the events that were in the confirmed response.
   • An OPC UA acknowledgement in the next Publish request lets the server drop that notification message from its retransmission queue.
   • A BACnet SimpleACK to a confirmed event notification tells the device the notification was received.

   None of these mechanisms on its own guarantees that the consuming application stored the event durably.

3. Operator acknowledgement: BACnet AcknowledgeAlarm, the Acknowledge method on an OPC UA condition. A person or a rule has seen the alarm, and the device records that it was acknowledged.

Once the protocol confirmation is sent, the sender may release the relevant transmission or retransmission record. The alarm condition itself may still be active or retained in the device, but the notification will not be sent again, so in practice the gateway has taken over responsibility for it. If the gateway then crashes before writing the event somewhere durable, the device will not send it again. Where the protocol and the library let the application decide when to confirm, confirming only after the event is safely stored is one of the simplest and most valuable rules in event handling. Where they do not, for example because a library confirms automatically or the acknowledgement is part of the transport window, the gateway needs another way to close that gap, such as a durable queue directly behind the adapter, and should document the window in which an event can still be lost.

Buffered events and time stamps

DNP3 is the reference design for buffered events. The outstation records each change of a point assigned to an event class, with the time it happened if the variation includes time. Events stay in the outstation’s buffer until a response carrying them is confirmed by the master. If the master is away, they accumulate, as long as the points are assigned to an event class and the buffer is large enough for the outage.

IEC 60870-5-104 has spontaneous transmission and time-tagged information types, and many controlled stations buffer spontaneous data while no connection is in data transfer. How much they buffer, and what they do when the buffer is full, is a property of the device rather than something the standard makes uniform; at the protocol level, IEC 104’s acknowledgements cover the I-frames of the current connection, whose sequence numbers start again on every new connection.

IEC 61850 offers two kinds of reports. An unbuffered report control block sends reports while a client is connected and enabled; what happens while nobody listens is lost. A buffered report control block keeps report entries in the server while the client is away, up to a configured buffer, and the server identifies each entry by an EntryID. Whether that EntryID appears in the transmitted reports is controlled by the optional fields of the report control block. When it does, a client that saves the EntryID of the last report it processed can resume after a reconnect and receive everything after it, provided it can reserve the same report control block again and the server still holds that entry. If the buffer overflowed, the next report says so with the BufOvfl flag.

OCPP applies the same idea to billing-relevant messages: a charging station that loses its connection keeps transaction-related messages, such as the start, the meter values and the end of a transaction, and delivers them in order when it reconnects. OCPP 2.0.1 gives every TransactionEvent a sequence number per transaction, so the CSMS can detect a gap. Its optional offline field says that the event occurred while the station was disconnected; it describes when the event happened, not how the message was sent, and a CSMS must not assume it is present on every delayed event. Status notifications, by contrast, describe the current state and are sent again after reconnecting rather than replayed.

Queue overflow and missed changes

Every buffer is finite, and a design that relies on buffered events must handle overflow:

• DNP3 sets the EVENT_BUFFER_OVERFLOW internal indication when the outstation had to discard events. The correct response is an integrity poll to resynchronize static data, and an entry in your own records that history has a gap.
• IEC 61850 sets BufOvfl in the next report of a buffered report control block.
• OPC UA monitored items have a queue size and a discard policy, oldest or newest. When the queue overflows, the overflow is indicated in the status of the value. For events, an overflow is reported with a special EventQueueOverflowEventType event.

A gap you know about is manageable: mark the period as incomplete, resynchronize the state, raise an operational warning. A gap you do not know about corrupts reports, energy totals and incident timelines without anyone noticing.

OPC UA sequence numbers and retransmission

An OPC UA subscription numbers its notification messages. The client acknowledges sequence numbers in its next Publish requests, and the server keeps unacknowledged messages in a retransmission queue. If the client notices a missing sequence number, or reconnects after a short break, it can call Republish to get a message again, as long as the server still has it. The retransmission queue is finite and its size is up to the server. Durable subscriptions, which keep notifications for hours, are an optional extension, not something every server offers.

Two limits keep this honest:

• The subscription lives only as long as its lifetime count allows without Publish requests. If the client is away longer, the server deletes the subscription and its queue, and what changed in the meantime is gone.
• A subscription belongs to a session. After a reconnect, the client must reactivate the session, or move the subscription to a new session with TransferSubscriptions, before it can collect anything. If neither works, it recreates the subscription and starts fresh.

For alarms, OPC UA offers ConditionRefresh: the server resends the current state of every retained condition, so a client that lost track knows which alarms are active and which still need acknowledgement. It restores the present, not the history; past transitions are available only if the server supports historical events and the client reads them.

DNP3 event classes and application confirmations

DNP3 gives the integrator two controls over event delivery:

• Class assignment decides which points generate events in which class. Class 1 often holds the most important changes, which the outstation may be configured to report immediately; classes 2 and 3 can be collected less often.
• Unsolicited reporting can be enabled per class. The outstation sends unsolicited responses with events and waits for an application confirm; without one, it retries according to its configuration and keeps the events.

A careful master starts a session by disabling unsolicited responses, running an integrity poll to learn the current state, and only then enabling unsolicited reporting for the classes it wants. That order prevents events from arriving before the master has a baseline to apply them to.

IEC 61850 buffered reports versus GOOSE retransmissions

GOOSE looks like an event mechanism, but its reliability model is entirely different from a buffered report. A GOOSE publisher sends its dataset as a multicast Ethernet frame when a value changes, then repeats it at increasing intervals until it reaches a steady heartbeat. Each state change increases stNum; each repetition increases sqNum.

A subscriber that misses a frame gets the same state in the next repetition. A subscriber that was away for two minutes gets the current state on the next heartbeat. In a publisher that has been running continuously, a jump in stNum tells it that changes happened which it did not see. A restart of the publisher, a configuration change or a counter wrap also changes stNum, so a jump is a reason to check, not proof of missed events. What it cannot get is the intermediate states: GOOSE carries state, not history. Combined with a time-allowed-to-live that marks the data invalid if no frame arrives in time, this is ideal for fast protection signalling and wrong for an event log.

CANopen emergency frames and heartbeat supervision

CANopen devices report errors with EMCY messages: an emergency error code, the error register and manufacturer-specific bytes. An EMCY is a single CAN frame produced for any node that listens. There is no confirmation and no retransmission at the application level; a node that was not listening misses it. Devices keep a pre-defined error field, object 0x1003, with a short history of recent errors, which a master can read by SDO after the fact.

Heartbeat messages provide the supervision. Each node produces its NMT state at a configured interval, and consumers expect it within a timeout. A missing heartbeat is itself an event, the loss of a node, and is often more important than any single EMCY.

KNX: state without history

KNX group telegrams are not stored anywhere. A gateway that was disconnected from the bus or from its KNXnet/IP interface missed every telegram in that time. To recover, it reads the group objects that have the read flag set, and their devices answer with GroupValue_Response. Objects without the read flag can only be learned when they next change. Some actuators send their status periodically, which helps. KNX group communication has no standard way to replay missed group telegrams; any history comes from a product or supervisory system that logs the bus itself.

Recovering after network interruption

Back to the two-minute disconnect on the cover. A practical recovery strategy, per protocol family:

Mechanism	What the gateway does on reconnect	What it gets back
Polling (Modbus, S7)	Poll again	Current values; nothing in between
BACnet COV and events	Renew subscriptions, read values, GetEventInformation	Current values and active events; history only from an Event Log object
OPC UA subscriptions	Reactivate or transfer, Republish, ConditionRefresh	The gap, if the subscription survived; otherwise current alarm states
IEC 104	STARTDT, collect buffered data, general interrogation	Whatever the station chose to buffer, then its configured interrogatable points
DNP3	Event poll or integrity poll, then enable unsolicited	Buffered events with times, if they fit in the buffer, or a flagged overflow
IEC 61850 buffered reports	Reserve the report control block, resume from the last EntryID if reports carry it, general interrogation if the block supports the GI trigger	Every report still buffered, or a BufOvfl flag
GOOSE	Wait for the next repetition	Current state; a stNum jump suggests a gap
CANopen	Check heartbeats, read the error field by SDO	Current NMT state and recent errors
KNX	Read status objects	Current state of readable objects
OCPP	Accept the queued transaction messages, check sequence numbers	Transaction history; status as it is now

Practical lesson

A protocol’s support for notifications does not necessarily mean guaranteed event history. For every event source, write down three things: where events wait when you are away, how long they can wait, and how you learn that some were lost. Where you control the confirmation, confirm events only after they are stored; deduplicate on replay, and record gaps as data. Reconnecting is not recovering builds the reconnect state machine around these rules, and Time, freshness and ordering covers the time stamps that make recovered events usable.