Artificial intelligence is leaving the data center and entering the world. Cameras which comprehend their surroundings, sensors which make decisions, industrial systems which respond in fractions of a second - these are edge AI, distributed in the world at the edges of human control: warehouses, vehicles, and other distant locations. The compute power is impressive, and the networking which makes it all work is the secret and challenging foundation.

The reason is simple: an edge device is only useful if it can be reached, reliably and securely, over its lifetime. The task is non-trivial for any distributed system which must be managed from a distance. This article focuses on the techniques used to connect fleets of edge-AI devices securely and manage them over their lifetimes, with a particular emphasis on the unique challenges of the domain.

Specifically, the article considers the following four questions which are of utmost importance to operators of distributed systems:

  1. Private connectivity: How can devices which are hard to reach be reliably connected privately?
  2. Identity: How can devices which must be accessed from the outside be known to be who they are?
  3. Management: How can hundreds of devices be managed, at scale, over their lifetimes?
  4. Operations: How can fleets of devices be maintained and operated effectively?

The article discusses these topics in detail in the sections below.

1. Private connectivity: How can devices which are hard to reach be reliably connected privately?

The first pillar of a secure connection to hard-to-reach devices is a private overlay network, implemented as a modern, well-documented, and audited open-source virtual private network (VPN). All of the devices in the fleet and the operators' infrastructure connect to this private network, encrypted end-to-end. WireGuard carries encrypted IP packets as UDP traffic over an underlay network, which may include the public Internet. The application traffic is protected inside the tunnel, while the underlay still carries encrypted packets and exposes the selected peer endpoints.

Two constraints particular to distributed systems inform how operators should think about their private network:

Edge devices cannot be reached.
A camera or other device which is connected through a cellular modem or behind a firewall cannot be reached on a public IP address. This means that traditional approaches to establishing connections - connecting from the center out to devices - are often ineffective or impossible. In a hub-and-spoke deployment, devices instead initiate outbound sessions to a reachable coordination point. That is an architecture choice, not a WireGuard requirement; a routed mesh or another topology may be appropriate. Limiting public exposure can reduce the number of Internet-facing endpoints, but the reachable coordination point still needs hardening, monitoring, and a recovery plan.

Local networks of remote sites are often indistinguishable.
The local network behind a firewall or router at a remote site is often similar or identical to the local network at another remote site. This means that, for example, the "device in the corner" at either site may have the same IP address. This is desirable for operators: from the perspective of the device, its local network is opaque and should be treated as a "black box." A naïve network design would not account for the possibility of collision and would have unpredictable results when they occur. A good design would eliminate the possibility of collision and provide operators with the abstraction that the local networks of remote sites are isolated, invisible, and self-contained. In practice, an overlay controller can assign site-scoped addresses and routes or translate between overlapping site networks. WireGuard supplies encrypted peer interfaces and cryptokey routing; it does not itself define the fleet's address-allocation or site-translation policy.

Given these constraints, operators of distributed systems should consider two options:

Background access to multiple sites.
With a private overlay network, an operator can access multiple remote sites concurrently for tasks such as monitoring or orchestration. The actual throughput and contention depend on the site uplinks, relay or hub capacity, routing, and any quality-of-service policy; the VPN does not remove those constraints.

Foreground sessions to isolated sites.
When operators need to perform tasks which require the full capability of a device, such as interacting with other equipment on the local network, they should establish a direct session to the remote device. This can provide a similar network path to being on the local site network, but access should still be bounded by identity, authorization, audit, and least-privilege controls. These sessions can support maintenance and diagnostics without making every service publicly reachable.

It is important to note that these two methods are complementary. An operator may simultaneously access multiple remote sites in the background while maintaining a foreground session to one. In addition, the overlay network provides secure, uninterrupted access for operators while still allowing the device to maintain its connection to the control plane. Operators can continue using their regular Internet access when the overlay uses split-tunnel routes; a full-tunnel configuration changes that routing behavior and should be documented explicitly.

The design principle: The design principle which informs this approach is that operators should not have to think about the hard problems. The most robust systems are those in which the implementation details are hidden and encapsulated. By providing operators with convenient yet secure network access, fleet operators can empower their users while providing them with flexibility.

2. Identity: How can devices which must be accessed from the outside be known to be who they are?

A private network provides a vital degree of security, but it is not impenetrable. An outsider which tricks the fleet into believing that they are one of the operator's devices can access all of the traffic on the private network. For fleet operators, this is an acute problem: their devices are often physically accessible to other individuals and may even be located in insecure environments. The ability to impersonate a device is a serious vulnerability, and fleet operators should take measures to prevent it.

At a technical level, the threat is similar to a man-in-the-middle (MITM) attack on a TLS certificate. The simplest way to prevent such an attack is to prevent outsiders from presenting valid certificates altogether. Fleet operators can do this by tying the identity of the device to the device itself in a manner which cannot be forged.

A stronger approach is to bind a device identity to a hardware-backed private key or certificate where the platform supports it, and to use secure boot or measured boot when the threat model requires image integrity. A software image alone is not an unforgeable identity, and a legitimate device may still be compromised. Operators should therefore rotate and revoke credentials, restrict peer permissions, and limit the blast radius of a compromised component.

In practice, a device should only accept connections from known and trusted sources, such as operator tooling and authorized control-plane services. If an unauthorized device appears, quarantine or restrict it according to the incident policy rather than assuming that dropping traffic is the only safe response. A limited quarantine state can preserve forensic visibility while blocking access to production services.

The same principles which apply to devices should be considered by operators when managing people. Individual operators should have accounts, and an account can be associated with multiple devices. Access can be restricted at the site level, so that, for example, a regional technician can only see and manage devices within her jurisdiction. An operator can revoke access to a device or a person at any time, with the change taking effect as peer configuration and credentials propagate. This is the principle of least privilege applied to distributed systems: operators have only the access they need, and no more, to do their jobs.

The design principle: The design principle which informs this approach is that operators should trust the device rather than the file. Trust should be tied to a specific device whenever possible, and the blast radius of a single compromised device should be limited.

3. Management: How can hundreds of devices be managed, at scale, over their lifetimes?

Adding and removing devices from a fleet is a laborious process. A fleet with only three devices is simple to manage, but a production fleet may have hundreds of devices which are added or removed every week. Local networks of remote sites change or are taken down, and operators come and go. Operators of distributed systems should design their infrastructure to accommodate these realities gracefully.

There are two approaches to adding new devices to a fleet: operators can add them deliberately, or new devices can enroll themselves. Self-enrolling devices are useful when a fleet of identical devices must be deployed at scale. When these devices are initially powered on, they will enroll themselves with the fleet, greatly accelerating the deployment process. However, operators must consider the risks which self-enrollment presents. Self-enrolling devices should only have access to the network in a limited capacity, perhaps in a "pending" state, until they are manually approved. Approval can be revoked at any time. In this way, the operator can enjoy the convenience of automatic enrollment while minimizing the risks which it presents. These two approaches - deliberate onboarding and self-enrollment - are distinct but complementary.

Operators can onboard new operators or decommission devices at any time, without disruption to current operations. When operators are added, they can see and manage only the devices to which they are assigned, with no further configuration. Similarly, when operators are removed, all of their access to the fleet is revoked immediately. Likewise, operators can update the local network or other components of a remote site without disrupting the connection. These changes to the fleet take effect while the connection is maintained – there is no need to establish a new connection for every change.

Operators can self-heal and self-correct. The state of the fleet is a single source of truth, and the fleet operates according to this state. If an individual component restarts – for example, when a device is rebooted – it should restore itself to the correct state, if possible. This eliminates the need for operators to manually configure every change which occurs to the fleet.

The design principle: The design principle which informs this approach is that changes to the fleet should be simple and reversible. In particular, the desired state is the system of record, and the actual state is always reconciled with it. Operators can grow their distributed system, update their components, and maintain their connections without disrupting their operations. This greatly reduces the complexity of managing such a large-scale system.

4. Operations: How can fleets of devices be maintained and operated effectively?

Edge devices are distributed throughout the world, connected through networks which operators cannot control. It is inevitable that these devices will experience connectivity issues or failures due to local network outages or other problems beyond the control of the operators. Operations are the mechanisms which operators use to monitor and manage the fleet. The design of the operations layer is particularly important to the long-term success of the fleet, as operators will use it to detect and remediate outages.

Devices can report their status and perform diagnostics.
In a distributed system, operators should be able to monitor the status of their remote devices and detect outages as they occur. Ideally, this information should be available in a centralized location so that operators can see the status of all of the devices in the fleet at a glance. If a remote site goes offline, operators should receive a notification.

Devices should perform diagnostics and report this information to operators so that they can understand and respond to outages. In practice, this means that a device should routinely check in with the central server to report its status. The check-in should include information about the device's condition and connectivity. Operators can use this information to understand how the device is performing, troubleshoot issues, and determine whether the device is truly offline or simply unreachable.

The same mechanism can be used to perform remote maintenance. Because operators use the same channel which the device uses to check in, a control plane can request updates without requiring a publicly reachable service on the device. The VPN is the transport; enrollment, authorization, update integrity, and rollback belong to the control plane. When operators cannot knock on a device's door, the device can establish the path back to the control plane.

Operators can perform remote hands-on maintenance.
Operators should be able to perform maintenance and diagnostic tasks on a remote device as if they were there in person. This capability should be exercised through the private network, if possible, in order to avoid exposing the device to threats. This way, operators can use the convenience of a remote connection without the risks which it entails.

Operators should treat the unglamorous infrastructure as infrastructure.
Security is a priority, but it is not the only consideration. Operators must ensure that sensitive data is stored securely on devices, that dangerous operations are confirmed and audited, and that access to the system is rate-limited to prevent attacks. These considerations may not be as glamorous, but they are nonetheless essential to the security and stability of the system.

The design principle: The design principle which informs this approach is that operators should always consider the edge devices as black boxes. Monitoring and maintenance should be possible, but the details of the implementation should be hidden. This way, operators can focus on the tasks which matter most to them without being overwhelmed by unimportant details.

What good looks like

It is instructive to step back from the details and examine several principles which unify the approach:

  1. Expose as little as possible to the public Internet: In a hub-and-spoke design, the coordination point may be the only public endpoint while peers initiate outbound encrypted sessions. Other topologies may require a different endpoint model.
  2. Make hard problems self-contained: Network collisions, addressing, and reachability should be problems of the network, not the operator.
  3. Tie trust to a device rather than to a file or an individual, and limit damage when security is compromised.
  4. Make access to specific rather than general: Operators should have only the access they need to do their jobs.
  5. Record the desired state, and allow the system to reconcile the actual state with it.
  6. Make the fleet observable and steerable from a distance: Operators should be able to observe and maintain the fleet without being on-site.

Edge AI is only the beginning. These same principles apply to any domain which involves distributed systems of any sort, whether they involve AI or not. Fleet operators should always consider their infrastructure from this perspective in order to maximize flexibility, security, and convenience.

This is a high-level overview of design philosophy intended for a general audience. It intentionally omits implementation details.