About this document

These notes are distilled from production fixes and operational lessons gathered while operating a multi-stage DeepStream inference pipeline on an NVIDIA Jetson in production. The system is a headless edge vision system which had to sustain constant inference for days at a time; this is a set of problems specific to a deployment environment in which a Jetson is used in the field without reliable on-site recovery. The document focuses on DeepStream specifics, and lessons were derived from production incidents which could only be diagnosed at the site.

The document is field-focused: the application scenarios of squeezing a few more FPS out of a benchmark for the blog post are not the priority, rather the items that cause a week-long production outage are covered. It also assumes that the audience is already familiar with the process of building a basic pipeline graph (decode + mux + primary detector + tracker + secondary detector + on-screen display + sink). In the following points, the symptoms, root cause, action and outcome of every lesson are detailed in turn, with the goal of covering the most informative examples of each category.

1. Pipeline is easy; memory model is hard

The graph of deepstream-app examples is mostly boilerplate code: decode → mux → primary inference → tracker → secondary inference → on-screen-display → sink. The interesting part consists of memory considerations: the memory type of a buffer in the pipeline constrains which elements can read it, and not all memory types are accessible to all elements.

1.1. GPU-rendered overlays are dropped silently on encode

Symptom. The overlays rendered by an on-screen-display element in the pipeline were visible on the local Jetson display, but completely absent from the video stream re-encoded and re-exported over RTSP by the same pipeline.

Root cause. The OSD and encode branches were not using a compatible render and buffer path. DeepStream's default buffer memory differs by platform: Jetson commonly uses surface-array memory, while dGPU defaults may use CUDA device memory. An OSD mode and downstream converter that do not agree on format or memory accessibility can produce a visible local display while the encoded branch lacks the overlay.

Action. Render to CPU-accessible memory on the encode branch by setting osd.set_property(“process-mode”, 0) and ensure the branch uses the RGBA format required by CPU OSD mode. 

# On Jetson, the OSD's default GPU render mode produces device-only
# output the post-OSD converter can't read back for the encode path.
if is_aarch64() and output_over_rtsp:
    osd.set_property('process-mode', 0)  # 0 = CPU render

Outcome. CPU rendering produces CPU-accessible output that can be read by the video converter → encoder → RTSP chain. The performance impact depends on stream count, resolution, and overlay density, so it should be measured on the target Jetson rather than assumed to be insignificant.

Observation. "It appears on the screen" is not a sufficient test for visibility in a multi-sink pipeline. Two sinks may have different requirements for overlay rendering - if an overlay is rendered in GPU memory (which is inaccessible), it will not appear in downstream elements that are not rendered by GPU, such as encoders. Whenever an overlay is rendered and fails to appear in a sink, check the OSD process mode, pixel format, buffer memory type, and converter/encoder requirements.

1.2. CPU-accessible frame access differs between host and Jetson

The code often runs on a host PC with a discrete GPU during development, while it runs on Jetson during production. To enable frame access from Python in the development host, unified memory is used: 

if not is_aarch64():
    mem_type = int(pyds.NVBUF_MEM_CUDA_UNIFIED)

    for el in (streammux, nvvidconv, nvvidconv_postosd):
        el.set_property("nvbuf-memory-type", mem_type)

On Jetson, the default surface-array memory is not equivalent to an ordinary CPU pointer. CPU access requires the supported surface mapping and synchronization steps; the exact path depends on the DeepStream release and element configuration.

Observation. Code that wishes to access frame contents through the surface API must branch on the platform. Memory-type properties are not portable between host and Jetson.

2. DeepStream is not version-portable; target versions are a deployment choice

The application is built to run on top of several DeepStream versions at once - different Jetsons run different versions. Configuration is not shared between targets; it is a serious production hazard to attempt to assume that a configuration works across versions.

In practice, the code contains branches for every version divergence, keyed off a parsed version string. Some specific divergence points which are encoded in the code:

  1. The name of the secondary-model config directory changed between versions
  2. A property is required for batch processing of tracker objects in older versions but absent in newer ones.
  3. The default render mode of the OSD element differs between versions (covered in 1.1)

Observation. The default properties of elements, plugins, and config directories are not version portable. The version must be parsed and branched on as early as possible and deployment images must be version specific. Pin the DeepStream, JetPack, CUDA, and driver compatibility set for each image, test the combination on target hardware, and branch only where a documented compatibility difference requires it. A single image should not be assumed to run unchanged across every DeepStream and JetPack release.

3. Expect the pipeline to become silent; watchdogs and crash recovery are mandatory

A GStreamer pipeline can become silent: a probe can stop pushing buffers, the source camera can become unavailable, or the driver can encounter an internal error. This can cause the pipeline to become unresponsive and enter a state where it does not produce any output. This is especially dangerous in the field, as no logs are available to diagnose the issue.

What we run. Watchdog processes that can kill the pipeline process and cause it to restart: a per-frame watchdog that terminates the process if the per-frame callback stops updating a timestamp, and a dependency health watchdog that terminates the process if the state bus becomes inaccessible.

Observation. The most reliable way to restart a pipeline that has stalled in an unresponsive state is to have the process terminate and let the container restart policy recover it. Having a watchdog process that kills the pipeline as a separate process allows us to cleanly exit the pipeline process with SIGTERM.

Observation. Watchdogs must have independent exit codes so that the restart reason is known. The restart policy should include a sensible backoff, health checks, and an escalation path; immediate unlimited restarts can create a tight failure loop.

3.1. Cheap watchdog bug: self-rearming timers with tens of thousands of threads

A self-rearming watchdog thread would create a new thread every watchdog interval, which created tens of thousands of threads per day. The watchdog was reimplemented as a long-running daemon thread that blocks on a wait-with-timeout. 

while not stop_event.wait(interval):
    if seconds_since(last_heartbeat) > timeout:
        log.critical("pipeline stalled; terminating")
        os._exit(1)  # use the exit code expected by the selected restart policy

Observation. On an unwatchable device, the pipeline's liveness is best asserted by a heartbeat that the per-frame callback must continuously update. Once the watchdog has expired, the process can be terminated and restarted by the restart policy.

4. On resource-constrained platforms, design explicit degraded modes

A vision system that runs a primary detector plus a few secondary models at frame rate on an edge device is usually memory bound. Having a slight hiccup in the model processing pipeline can cause frames to accumulate and increase the processing load of the pipeline, which leads to more stalls and eventually triggers the pipeline watchdog.

Backpressure as a first-class state. The system was instrumented to track the number of frames in flight and have an overload flag that could prevent the pipeline from emitting results. This overload flag is tripped by a high value of the frame latency metric and cleared when the value drops below a lower threshold, thus preventing oscillation.

Observation. Degradation should always have a first-class representation in the pipeline: latency is a better metric to use than throughput to avoid stalls, and a hysteresis should be used to clear the overload state.

Other methods of degradation: the most memory-intensive stages can be skipped on a per-frame basis to reduce memory pressure while maintaining overall performance; a warmup gate can be used to prevent any results from being published until the system has had a chance to stabilize; the muxer can use a pushed timeout to avoid blocking the pipeline when a source is dropped and the muxer has to wait for the batch to fill.

Observation. Design the system to have an explicit overload state which causes it to shed memory-intensive or computationally expensive operations while maintaining correctness and reducing memory pressure.

5. Sensor fusion is all about timestamps

A pipeline that must fuse a time-series sensor reading (position, etc.) with the video frames has two possible definitions of "latest" - the sensor timestamp and the frame timestamp. The pipeline adds real latency between the capture of a frame and the moment a sensor observation is available for use by the pipeline.

What we do. The sensor values are stored in a ring buffer keyed off approximate timestamps. For each frame, the current timestamp is used to query the ring buffer for the closest value available at or before the frame's presentation timestamp, with an option to use a timestamp offset to account for the latency between the sensor reading and the frame capture: 

keys = [quantize(frame_ts) - i for i in range(history)]
for k in keys:
    if k in buffer:
        return buffer[k]

Observation. Sensor fusion should be performed by correlating frame timestamps with the sensor timestamp, taking into account the pipeline latency. If the timestamps are not aligned, the offset should be exposed as a configuration parameter so that it can be adjusted at deployment time.

Observation. A ring buffer keyed off approximately the frame timestamp is a good way to implement this. Using an approximate timestamp allows lookups to be O(1) instead of O(n).

Observation. It is essential to expose this offset as a configuration parameter that can be tweaked at deployment time. The offset is often non-zero and depends on the sensor characteristics and the pipeline's internal latency.

Observation. Avoid using state that is shared between pipelines. This can lead to data races when multiple pipeline instances are running at the same time and can cause incorrect sensor fusion results. State should be tracked as instance variables so that each pipeline can maintain its own state without interfering with others.

6. Invisible drops are the worst. Observability is a feature.

The class of bugs that had the biggest cost in context were cases where an invisible suppression of a result, a sensor reading, or a label occurred. Such bugs have no visible manifestation and no debugging technique beyond finding the point in the code where suppression happened by process of elimination.

The following are some examples of cases where a suppression of a result was done invisibly:

  1. A stale flag was used to suppress a result at publish time. The flag had no logging or metric associated with it and could only be diagnosed once suppression was made visible.
  2. A reported state had two mutually exclusive flags set that could not have been set by a single entity at a time. This could only be diagnosed by inspecting the state bus and seeing what entities had accessed shared state.
  3. A startup probe of an external hardware sensor failed silently and fell back to using a default value that was incorrect. This could only be diagnosed by checking the startup probe and verifying that it had not succeeded in finding the sensor.

Observation. Every suppression path should emit an observable reason through logs, metrics, or state inspection so operators can distinguish intentional filtering from a defect.

Observation. Live state inspection and the ability to see what processes are writing to shared state or suppressing results is critical to debugging production issues. The state bus inspection capability is the reason why such bugs are diagnosed at the site.

Observation. Assumptions made about external hardware must be re-evaluated at runtime to ensure they are correct. This is especially important for hardware that is difficult to test, such as cameras, sensors, and network connections. Observations about such hardware must be made at the site to confirm that the assumptions are correct.

Observation. Live video overlay should be used to make visible decisions the pipeline is making. This can greatly reduce the amount of time spent trying to determine why a particular candidate was selected or rejected.

7. The per-frame callback is a hot loop. Treat it as one.

The per-frame callback is invoked once for every frame, for every stream, at the camera frame rate. Any allocation or synchronous processing in the callback will be multiplied by the frame rate and penalized in terms of latency.

Some changes made during a hot-path audit of the code base, many of which were used in every frame:

  1. Blocking I/O is not allowed in the callback. A synchronous read from the bus to get a shared flag was converted into a background thread that updated a cached copy of the variable.
  2. A value that was derived from a slow source was cached with a short TTL.
  3. An invariant value that was recalculated for every item in a list was moved out of the loop.
  4. A list that was growing without bound was replaced with a list that had a single item that was replaced every frame.

Observation. Performance-sensitive code should be profiled as if it were an interrupt handler. No I/O should be performed synchronously. Expensive operations should be moved to background threads and cached, if possible.

8. Field devices have no control panel. Everything must be tuned or observed from a distance.

A field device should support reloadable configuration that can be tuned without requiring an image change and re-deploy. The following features help tuning from a distance:

  1. Config file-based hot reload: a watcher process watches for changes to the configuration file and reloads it when it changes. The reload is a full restart, but it allows for quick iteration and testing of changes.
  2. The state bus can be used to implement live control of the pipeline.

Observation. The code should be structured in a way that allows for rapid iteration and tuning of the behavior of the system in the field. This is done by separating code that is shipped in the versioned image and configuration that is reloaded at runtime and using a config file watcher to watch for changes to the configuration file.

9. Operational odds and ends that ate up hours.

Some of the less exciting but important discoveries that were made:

  1. Stale plugin cache: "could not create element X" means the plugin cache is corrupt and needs to be deleted. This is a common problem after updating the image.
  2. The restart policy is part of the recovery design. It is what makes the watchdog a recovery mechanism and not just a crash. The watchdog and the restart policy are always discussed together: the watchdog process should kill and restart the pipeline as needed.
  3. One process per stream, cleanly compartmentalized: a single variable controls what inputs, outputs, and state bus keys are used for each instance. Each instance has its own process and does not share any state with others. This makes it possible to kill one process without affecting others.

The short version

The set of notes above represents a list of recommendations that are applicable to any DeepStream-on-Jetson deployment. The most important recommendations are summarized below:

  1. Memory model, not pipeline, defines constraints → branch on the platform to get at CPU-accessible memory
  2. Target versions, not one version → build one image per target
  3. Use heartbeat to prove liveness; exit on failure to recover with restart policy
  4. Use delay as a metric for overload; have a budget to shed processing
  5. Correlate on capture timestamps; keep frame state instance-scoped
  6. Make suppression of results or observations visible: no silent skips; use metrics and bus state inspection to find what suppresses; use live overlay to explain suppression
  7. Watchdog is the interrupt handler; be lean in per-frame callback - no blocking I/O; cache slow reads; bound per-frame allocation
  8. Make configuration reloadable by watcher, not versioned - hot-reload config by modification time; live control via bus inspection

The pipeline diagram takes an afternoon; what matters is what the team has learned in the subsequent six months.