Embedded systems · real-time firmware

I crashed the network chip 11 times. The control loop didn’t miss a deadline.

Kestrel-RT is a real-time runtime I wrote in C for a 48 MHz Arm Cortex-M4F, from the reset vector up. Phase 2 moved networking onto the board's ESP32-S3 and treated it as untrusted. In one 10,003,440-period soak with 11 forced crashes of that chip, the real-time side had 0 resets and 0 deadline misses; separate campaigns with Wi-Fi floods and 15,000 malformed frames also caused 0 resets and 0 critical misses.

Repository Demo video Engineering log

A live screen recording of the physical board running the demo script; the lines are read from the board as it runs.

10.43M: Phase 1: control periods in one 2.9 h soak, 0 deadline misses (10,429,500)
11: Phase 2: forced ESP32-S3 crashes in one 10,003,440-period soak, 0 controller resets
0: Phase 2: of 15,000 malformed frames accepted, in a separate fault campaign
+3.3 µs: control start p99, the measured cost of the network link

Status: v1.0 released; every result measured on the physical board
Role: Solo: runtime, drivers, protocol, ESP32-S3 firmware changes, test harness, campaigns
Stack: C, Arm Cortex-M4F (Renesas RA4M1, 48 MHz), ESP32-S3, Python host tools
Evidence: Frozen gates, per-run result files, raw-capture hashes, engineering log of every failure

The problem

A robot controller puts a hard-timed control loop next to code that misbehaves: networking, logging, telemetry. I wanted to own every layer of that on one microcontroller, from the reset handler to the scheduler to the drivers, and measure what each design choice actually costs.

What I built

Runtime: Own startup and linker script, a tickless fixed-priority scheduler on a 64-bit timebase, PendSV context switching, unprivileged tasks under 8 MPU regions, no heap.
Fault handling: Crash records that survive the reset, a contained fault parks one task while control keeps running, and a supervisor feeds the watchdog only while the critical tasks vote.
I/O and tracing: Register-level DMA for all UART traffic; telemetry drops records (counted) instead of blocking; a binary flight recorder.
Network isolation: The ESP32-S3 reaches the controller only through a DMA ring and a budgeted task: 300 µs per job, sessions, a sequence window, replay rejection, 400 frames/s rate limit.

Results

Phase 1 soak: One 2.9 h run on the physical board: 10,429,500 control periods with 0 misses, 0 overruns, 0 faults across all six tasks; control start latency p99 45.0 µs, max 50.5 µs.
Faults: 17/17 injected cases on the final firmware (earlier runs 16/17, traced to UART receive overruns and fixed): MPU violations, bus and usage faults, stack overflow, asserts, hangs. Contained faults cost control 0 deadlines once a handler bug that missed one was fixed.
Comparison: Same workload on FreeRTOS 11.1.0, Zephyr 4.2.0 and a superloop. Under 7x overload Kestrel's control task degraded to half rate while safety, sensor and comms kept their deadlines; the FreeRTOS baseline missed every control job and starved the other tasks.
Phase 2 soak: One 10,003,440-period run with a forced ESP32-S3 panic every 15 minutes (11 in total): 0 controller resets, 0 deadline misses on any task, 0 protocol desyncs.
Phase 2 campaigns: Separate short runs: Wi-Fi flood, CPU and heap exhaustion, an 83,711-frame mailbox flood, 15,000 malformed frames (0 accepted), a held-low line and reboot loops, each with 0 controller resets and 0 critical misses.

The crash, as the board logged it

-- force an ESP32-S3 panic (NULL write on the ESP)
[live] ESP reply: ok crash
[live] ESP gone: no telemetry for 2.2 s (ESP reboot, Wi-Fi rejoin, new session)
[live] telemetry resumed: RA4 epoch 585065 (same RA4 boot), link session 46789 -> 13097
[live] RA4 control ran through it: +2,300 jobs in 2.3 s (1 ms period), misses 0 -> 0

Verbatim excerpt from the live board demo log.

What did not work

The failures are kept in the repository with root causes. The network chip's download mode once produced about 1.17 million receive-error interrupts on the controller. The first link budget let a job reach 507 µs against a 400 µs gate. The protocol desynchronized permanently after 15 s of silence. Each was fixed and has a regression test.

Limitations

Trade-off: FreeRTOS had lower control-start latency (p50 15.9 vs 24.6 µs). Kestrel spends that time on per-job deadline accounting and MPU switching.
Security: Fault isolation, not a security boundary: the ESP32-S3 still controls the controller's reset and debug lines.
Scope: Measured on one board, with no motor driven. The cause of the +3.3 µs link cost is not yet isolated.

At a glance