■FIG. A.3 · Self project · C++ on-device pipeline
Deaf Audio Assistant
For deaf and hard-of-hearing users, sound that matters — a spoken sentence, a smoke alarm, a siren behind you — arrives with no visual channel. This system builds one: streaming speech-to-text plus hazard-sound classification, running entirely on-device on CPU, with no audio ever leaving the machine. The engineering constraint is latency: the design budget is 500 ms p95 end-to-end, and the bench measures 102.136 ms.
PER-STAGE LATENCY · p95 (ms) — bench/report.md, committed
102.1 ms
caption e2e · p95
500 ms
design budget · p95
7.9×
real-time factor
0/0
ring drops in run
Measured on Intel Core Ultra 5 225H, 14 logical cores, ASR 2 / SED 4 inference threads. Every value above is transcribed from the bench report committed to the repository — nothing on this page is a re-enacted or illustrative number.
The real system runs capture → inference → UI on three threads joined by bounded single-producer / single-consumer rings (capacity 4); when a producer outruns its consumer, the oldest item is dropped rather than blocking the mic. This simulation replays that contract: starve the inference thread and watch drops accumulate in the ring, not on the microphone.
SED EVAL · SAMPLES FROM THE REPO’s OWN EVALUATION SUITE
Siren · hazard · tier 3 · θ 0.3
Smoke detector, smoke alarm · hazard · tier 3 · θ 0.3
Explosion · hazard · tier 3 · θ 0.3
Wind · ambient · θ 0.4
silence — negative control
CURATED AUDIOSET CLASS TABLE — HAZARD TIER (EXCERPT)
| class | tier | threshold |
| Smoke detector, smoke alarm | 3 | 0.3 |
| Fire alarm | 3 | 0.3 |
| Fire | 3 | 0.35 |
| Siren | 3 | 0.3 |
| Civil defense siren | 3 | 0.3 |
| Police car (siren) | 3 | 0.3 |
| Ambulance (siren) | 3 | 0.3 |
| Fire engine, fire truck (siren) | 3 | 0.3 |
Sigmoid probabilities per 4.8 s window, 0.48 s hop; hazard classes feed the ranked alert list.
THE PIPELINE
Three threads — capture, inference, UI — are joined by lock-free single-producer/single-consumer ring buffers, and the hot path is allocation-free. Speech runs through Nemotron 0.6B FastConformer-CacheAware-RNNT via sherpa-onnx in 560 ms chunks at around 8% WER, more than 6× CPU real-time. Sound events come from an Int8-quantized ONNX CNN over log-mel spectrograms covering 20+ hazard-relevant classes, with sliding-window classification, event debouncing, and alerts ranked by confidence and recency.
WHAT HAPPENS WHEN THE CPU CAN’t KEEP UP
Backpressure is drop-oldest: under CPU starvation the rings shed their oldest frames instead of blocking the microphone, so the user loses a stale caption rather than live audio. Chaos tests under synthetic load verify the degradation is graceful. The lab above models that contract; the bench chart beside it is the measured reality.
Prev: FIG. A.2 · GoRESP Edge CacheNext: FIG. A.4 · Slack Data Copilot