Designing a multi-protocol OT gateway without hiding the differences

Sooner or later, every OT software team builds a gateway: something that talks to the equipment on one side and to historians, message brokers, APIs and increasingly AI agents on the other. The first version usually talks one protocol. The second talks three, and that is when the design decisions of the first version start to hurt.

The central question of this article: how should a software engineer design an application that talks to all these protocols?

It is the last part of the series that starts with 14 industrial protocols, 5 communication patterns, and it draws on all the earlier ones.

Why a universal device client becomes a problem

The tempting design is one interface for every device:

// The interface that looks simple and hides everything that matters.
type Device interface {
  Read(ctx context.Context, point string) (float64, error)
  Write(ctx context.Context, point string, v float64) error
  Subscribe(ctx context.Context, point string, fn func(float64)) error
}

Every method on it is wrong for some protocol:

• Read on a DNP3 or IEC 104 point either polls a protocol that was designed to report, or returns a cached value with no time and no quality. Either way, the event history and its time stamps are gone.
• Subscribe on Modbus becomes a hidden poll loop at a rate nobody chose, competing with every other hidden poll loop for the device’s single request slot.
• Write returns nil for a PROFINET output that has merely been placed in the next cycle, for a breaker command that was accepted but not executed, and for an OCPP remote start that may never happen.
• Nothing in it can express a cyclic relationship, a watchdog, a session that is connected but not synchronized, or a command that is scheduled for tomorrow.
• float64 loses the quality flags, the source time and the difference between “no value” and “stale value”.

The problem is not abstraction as such. It is an abstraction that pretends all protocols have the same contract.

Capability-driven protocol adapters

A better starting point is a small set of capabilities, each with an honest contract, and adapters that implement only the ones their protocol actually has. In Go, the conceptual shapes might look like this:

// Value is what every acquisition path produces.
type Value struct {
  Point      PointRef       // provenance: adapter, device, protocol address
  Data       any            // typed value; nil when there is none
  SourceTime time.Time      // zero when no source time is available
  AcqTime    time.Time      // when this process obtained it
  Comm       CommStatus     // communication: fresh, stale, not synchronized, link down
  Quality    Quality        // process quality from the device, raw flags kept
  Path       AcquisitionPath // Polled, Subscribed, Event, Cyclic, Group
}

// Reader: explicit request/response reads (Modbus, S7, OPC UA Read, SDO).
type Reader interface {
  ReadBatch(ctx context.Context, points []PointRef) ([]Value, error)
}

// Subscriber: the server decides when, the client decides what (OPC UA, BACnet COV).
type Subscriber interface {
  Subscribe(ctx context.Context, req SubscriptionRequest) (Subscription, error)
}

// EventSource: device-originated events (DNP3, IEC 104, IEC 61850 reports, OCPP).
// No promise of durable replay, total order or exactly-once delivery.
type EventSource interface {
  Events() <-chan Event // in delivery order, may contain replays;
                        // carries its DeliveryToken where the protocol has one
}

// EventCheckpointer: optional, only where the protocol and the library let
// the application decide when a delivery is confirmed (for example a DNP3
// master with application-controlled confirmation). A DeliveryToken stands
// for one protocol-specific confirmable delivery, such as a DNP3 response
// fragment: not an individual event and not an ordered offset. Adapters that
// cannot offer it, such as IEC 104, whose acknowledgements are part of the
// transport window, simply do not implement it.
type EventCheckpointer interface {
  ConfirmDelivery(ctx context.Context, token DeliveryToken) error // after durable storage
}

// ResumableEvents: optional, where a source can resume from a position
// (an IEC 61850 buffered report EntryID, an OPC UA sequence number).
type ResumableEvents interface {
  ResumeFrom(ctx context.Context, pos EventPosition) error
}

// CyclicIO: a configured relationship with a deadline (PROFINET, EtherNet/IP Class 1, CANopen PDOs).
type CyclicIO interface {
  Inputs() InputImage     // latest image with validity and cycle counter
  SetOutputs(OutputImage) // accepted into the local output image; says nothing about the device
  State() RelationshipState
}

// Session: lifecycle that the rest of the gateway can observe.
type Session interface {
  State() SessionState // Disconnected, TransportReady, ProtocolReady, Synchronized, Operational
  Changes() <-chan SessionState
}

// Controller: commands with stages, not writes with an error.
type Controller interface {
  Submit(ctx context.Context, cmd Command) (CommandHandle, error)
}

These are conceptual capabilities, not a mandate to define one interface per operation or to wrap every protocol library behind them. A Modbus adapter is a Reader and a Controller; a DNP3 adapter is an EventSource, an EventCheckpointer if its library lets the application confirm, a Reader for integrity polls and a Controller; an IEC 61850 client adds ResumableEvents for buffered reports; a PROFINET adapter is a CyclicIO with a Reader for records. Code that needs a capability asks for it; code that receives an adapter without it knows, at compile time or at configuration time, that the protocol does not offer it.

Protocol-specific APIs remain available underneath. The gateway uses capabilities for the common paths and reaches into the adapter for anything protocol-specific, such as IEC 61850 setting groups or OCPP smart-charging profiles, rather than inventing lowest-common-denominator versions of them.

Separate polling, subscription, event and cyclic execution

The cover of this article shows the central idea: four execution paths with different scheduling and failure models.

• Acquisition scheduler for Readers. It groups points into poll groups with rates and deadlines, merges contiguous addresses into batch reads, respects per-device concurrency limits (one outstanding request for many Modbus devices and every DNP3 association, the negotiated job count for S7), and spreads polls over time instead of firing them all on the second.
• Subscription manager for Subscribers. It creates, renews and supervises subscriptions, watches keep-alives, and recreates subscriptions after a session change.
• Event pipeline for EventSources. It stores events durably and, where the adapter is an EventCheckpointer, releases them only afterwards. It deduplicates replays where a protocol identity allows it, keeps arrival order and sequence information next to source time instead of re-sorting blindly, applies backpressure when consumers fall behind, and turns overflow indications into explicit gap records.
• Cyclic I/O loop for CyclicIO. It runs isolated from the rest: its own goroutine or process, no allocation-heavy work on its path, outputs computed by fast local logic or set by the command service and applied at the next cycle, and inputs published to the rest of the gateway as snapshots with validity.

Separating them means a slow Modbus device cannot delay DNP3 events, an event storm cannot starve the cyclic loop, and each path can report its own health.

Explicit lifecycle and recovery state machines

Each adapter owns one state machine per session, with the states from Reconnecting is not recovering: disconnected, transport ready, protocol ready, synchronized, operational. The surrounding components react to state changes:

• while a session is not synchronized, its values are marked uncertain and its commands are held or refused;
• on synchronization, the event pipeline expects replays and the subscription manager restores subscriptions;
• on disconnection, the scheduler stops polling that device and the freshness monitor starts counting.

An actor per session is a natural fit: one goroutine owns the connection and its state, receives requests on a channel, and publishes state changes. It makes the protocol’s own concurrency rules, such as one DNP3 request at a time or one OCPP call per direction, easy to enforce.

Acquisition timestamps, quality and provenance

Every value that leaves the gateway carries:

• source time when the protocol provides it, and acquisition time always;
• communication status, kept apart from process quality: a value can be good at the device and stale because the link is down;
• quality, mapped to a common good, uncertain, bad scale, with the protocol’s raw flags kept for anyone who needs them;
• freshness against a deadline derived from how the value is acquired;
• provenance: adapter, device, protocol address and acquisition path.

This is what makes the data usable downstream, from historians to analytics to agents that act on it. Time, freshness and ordering explains each field.

Event deduplication and backpressure

The event pipeline is where reliability is won or lost:

• Store, then release, where you can. Where the protocol and the library leave the confirmation to the application, confirm DNP3 events and answer OCPP transaction messages only after they are written durably. An IEC 61850 EntryID is not a per-report acknowledgement: record the last EntryID you stored, so you can resume from it after a reconnect if the report control block and the server’s buffer allow. Where confirmation happens below the application, as with IEC 104, put a durable queue directly behind the adapter and document the window in which an event can still be lost.
• Deduplicate by protocol identity where one exists: EntryID, OPC UA sequence numbers, OCPP transaction ids and sequence numbers. Where none exists, as for DNP3 events, use a gateway-side identity from the delivery context and a persistent ingestion sequence, and treat point plus source time plus value as a heuristic, not a guarantee.
• Bound every queue. When downstream consumers are slow, slow down confirmation where the protocol lets you, so that devices keep events in their own buffers, rather than growing memory without limit. Surface the backlog as a metric.
• Record gaps from overflow flags and counter jumps as first-class data.

Command acceptance, execution and verification

The command service implements the stages from Writing to industrial equipment:

type CommandStage int

const (
  Submitted CommandStage = iota
  Delivered  // the transport or protocol confirmed receipt
  Accepted   // the device agreed: echo, ACTCON, SUCCESS, Accepted
  Executed   // only from a message whose semantics mean completion, such as
             // IEC 61850 CommandTermination; never from acceptance alone
  Verified   // independent process feedback matched within tolerance and time
  Failed     // with the stage at which it failed and why
  Unknown    // timeout: outcome not known, needs reconciliation
)

Not every protocol reaches every stage: a Modbus write jumps from accepted to verified, and an OCPP remote start is followed by a TransactionEvent whose timing depends on the station’s configured start point, which tells you a transaction exists, not that energy flows. Each command has an id, a target, an idempotency policy, a timeout per stage and a verification rule: which point to watch, what value, what tolerance, by when. A timeout moves the command to Unknown, not Failed, and triggers a read of the target state before any retry. Finding the target already in the requested state means there is nothing to retry; it does not prove that this command caused it. Only idempotent commands are retried automatically.

Cancellation, timeouts and bounded concurrency

Go’s context.Context maps well onto industrial request lifecycles, with two cautions. First, cancelling a context does not cancel an operation that the device already received: a cancelled breaker command may still execute. Second, a cancelled request must not leave the protocol in a broken state; a library should finish or cleanly abandon the exchange without tearing down the association. Budget concurrency per device and per protocol, not per gateway: the limits are set by the equipment.

Observability and protocol-specific metrics

Generic request counters are not enough. Useful metrics are specific to the contract:

• session state and time in each state, reconnects and their causes;
• poll cycle duration versus target, late polls, timeouts and exception codes per device;
• subscription keep-alives, notification gaps, republish and transfer results;
• event backlog, confirmation lag, duplicates dropped, overflow and gap records;
• cyclic watchdog expiries, lost frames, cycle jitter;
• commands per stage, time to each stage, unknown outcomes;
• values past their freshness deadline.

Traces that follow a command from API call to verification are worth their cost the first time someone asks why a breaker did not open.

Testing: simulators, interoperability, faults and races

A gateway is only as good as the failures it has seen:

• Simulators for each protocol make development possible without hardware and allow writes without risk.
• Interoperability tests against independent implementations catch the misreadings that a library shares with its own test server.
• Fault injection: drop and delay packets, kill connections mid-command, restart devices, overflow event buffers, skew clocks, and check that the state machines, deduplication and gap records behave.
• Race testing: run with the race detector, hammer adapters with concurrent requests, reconnect during subscription setup.
• Captures: record real traffic, then replay and analyse it to find poll patterns, latencies and anomalies before they become incidents.

Configuration: point models per protocol

A gateway is mostly configuration: which devices, which points, how each one is acquired. A single flat point list with an address string and a poll rate forces every protocol into the polling model again. A point model that follows the capabilities keeps the differences visible:

devices:
  - id: inverter-7
    protocol: modbus
    endpoint: tcp://10.0.4.17:502
    unit: 1
    limits: { outstanding: 1, minGap: 20ms }
    poll:
      - group: fast
        every: 1s
        points: [active_power, reactive_power]
      - group: slow
        every: 60s
        points: [energy_total]
  - id: rtu-north
    protocol: dnp3
    endpoint: tcp://10.0.9.2:20000
    startup: { integrity: true, unsolicited: [1, 2, 3], timeSync: lan }
    events: { confirmAfter: stored }
  - id: press-line
    protocol: profinet
    station: press-io-1
    cycle: 4ms
    watchdog: 3
    onLoss: substitute

The Modbus device has poll groups and concurrency limits. The DNP3 outstation has a startup sequence and an event policy. The PROFINET device has a cycle, a watchdog and a loss behaviour. None of them pretends to be the others, and a reviewer can see each contract in the configuration.

The three running examples, end to end

The series used three running examples. Here is where each one lands in the gateway:

• Reading an inverter. The Modbus adapter is a Reader scheduled by the acquisition scheduler; its values carry acquisition time and a freshness deadline of a few poll intervals. If the same inverter sits behind a DNP3 RTU, the adapter is an EventSource, and the event pipeline stores the fault events with their source time before the adapter confirms them, if the DNP3 library lets the application control confirmation.
• Operating a breaker. The command service submits a select-before-operate through the DNP3 or IEC 61850 adapter, records delivered and accepted from the responses, executed from the IEC 61850 CommandTermination where one exists, and verified from the position report that arrives through the event pipeline. A timeout moves the command to Unknown and triggers a read of the position before anything is retried.
• Managing an EV charger. The OCPP adapter is a server for thousands of WebSocket sessions, each with its own state machine. Remote commands wait in the command service until their station is operational, or expire. The queued transaction messages that arrive after a reconnect go through the event pipeline and are deduplicated by transaction and sequence number. Scheduled limits from OpenADR or IEEE 2030.5 enter as commands with a start time, and the command service turns them into charging profiles for the affected stations.

The protocols differ, and so do the components that handle them. The value model, storage, observability and testing are shared.

What a protocol library does, and what the gateway does

The line between the two is worth drawing explicitly.

Keeping the protocol library honest and protocol-shaped, and putting policy in the gateway, makes both easier to test and to trust.

Practical lesson

Share engineering infrastructure where possible, and preserve protocol semantics wherever correctness depends on them. Share the value model, the storage, the observability, the configuration and the testing harness. Keep separate the things that differ: how data is acquired, how events are confirmed, how cycles are timed and how commands complete. A gateway built that way can grow to fourteen protocols without hiding the differences that matter.