HELIX runs inside a live GO2's ROS 2 graph and audits every recovery command

Learned robot policies fail in ways that look normal from the outside. Sensors go stale while still publishing. Recurrent state freezes. Topic rates degrade without dropping to zero. Policy outputs stay numerically plausible long after they stopped meaning anything.

A deployment stack needs a layer that notices the system is becoming untrustworthy before a user or a robot is exposed to the consequence, and that can prove afterwards what it noticed and when.

Plenty of robotics work ships a trained policy or a navigation stack. Far less ships the operational layer around it: the thing that detects silent failure, separates confirmed behavior from inferred behavior, and records whether a recovery command was physically acted on. Simulation validation does not establish hardware safety. For an edge robot the reliability layer has to run inside ROS 2, fit a companion computer's budget, survive real topic timing, and leave an audit trail. Most monitoring tooling assumes a datacenter, not a Jetson on a robot.

Separate sensing from acting, and make the split explicit in the evidence. That decision is what lets HELIX prove what it detected and which recovery commands it published, without claiming the robot physically executed a safe hold. When the actuation path turned out to be unsubscribed, the architecture surfaced it as a finding instead of hiding it behind a green dashboard.

Heartbeat monitoring, topic-rate anomaly detection, and log-pattern parsing feed a diagnosis stage that emits structured FaultEvents. Diagnosis produces recovery hints; a cooldown-aware auditor decides whether an action is permitted, records it, and publishes the zero-velocity command on a dedicated topic.

Architecture

Designed the reliability-layer split and the verified / inferred / not-validated evidence ledger that governs every claim in the repo.

Detection paths

Anomaly detector, heartbeat monitor, log parser, diagnosis rules, and the recovery-audit path.

Live integration

Found and fixed the R1 schema mismatch that silently blocked live diagnosis across earlier sessions.

C++ port

Ported the anomaly detector to C++ for the Jetson budget, then reported the failed promotion gate rather than shipping it on.

How it was validated

Live GO2 graph

Observed from both companion and development machines across eight sessions.

Jetson sessions

Measured persistent runtime, resource use, and thermal behavior on the Orin NX.

Topic adapter

Bridged real robot topics into HELIX metrics rather than replaying synthetic traces.

Separated benchmarks

Pure algorithm speed, ROS 2 callback latency, and hardware-session behavior measured independently, so no single number does double duty.

Results

Live graph attachability

HELIX nodes observed and processed live GO2 topic-rate streams through an adapter.

Recovery control plane (software)

Session 8, 439 s, robot standing idle: 30 ANOMALY alerts, all natural idle-jitter threshold crossings rather than confirmed faults, produced 14 recovery hints and 14 audited actions (7 published, 7 suppressed by cooldown), with no allowlist or cooldown violation, and 3,064 zero-twist /helix/cmd_vel messages. Published, not physically acted on: the stream had no downstream subscriber.

Latency

1.16 ms mean, 1.24 ms p95 through the actual rclpy callback path: 50 synthetic metric messages in a single-process benchmark, excluding inter-process transport.

Detector realism

96.5% TPR at 0.0% FPR on near-threshold synthetic anomalies, at the benchmark threshold (Z = 3.0), and 0% TPR on 3-sigma anomalies in Laplace noise. Heavy-tailed noise and slow drift remain known weak cases.

C++ port

30-minute hardware parity run: mean RSS 42.51 MB (Python, 1-h plateau) to 19.81 MB, −53.4% (−47.2% against Python at cold start); mean CPU 2.21% to 0.80%. It still missed the 30%-of-Python RSS target I set, so the port stays launch-gated.

◆Verified on hardware

• GO2 and Jetson sessions captured live ROS 2 topic landscapes, latency, resource, and HELIX runtime behavior.
• HELIX emitted structured FaultEvents from bridged GO2 rate metrics (unlabeled fluctuations, not confirmed faults) and from an injected log pattern, detected end to end in about 1.8 s.
• Diagnosis and recovery-audit messages flowed after the R1 schema fix.
• The C++ anomaly detector produced a lower-resource live fault stream.

When a robot misbehaves, the useful evidence is seconds old and usually gone. BlackBoxRS is the flight recorder beside HELIX: it builds a reproducible evidence bundle and replays recorded robot data through the same detectors offline, against a virtual clock pinned to bag time.

Real data

Offline replay of a real 330 s GO2 recording (about 94k messages) raised zero anomalies.

Injected fault

Finds a topic dropout injected into that recording from bag timing alone, offline.

Open

The real-bag evidence is offline replay; a live onboard capture is the largest remaining gap. A C++20 runtime exists on a branch, with hardware validation pending.

BlackBoxRS on GitHub

Wire a real subscriber to /helix/cmd_vel in a controlled lab session and demonstrate that an injected fault physically holds the robot, with the motion path instrumented on both ends. That single session converts the largest pending row in the ledger into a verified one.