Aura
On-device multimodal accessibility assistant for blind/low-vision and deaf/hard-of-hearing users — real-time scene description, captioning, and sound awareness with zero data leaving the device.
Overview
A private, real-time perceptual aid: it describes the scene and reads text aloud, captions and enhances speech, and fuses sound events with visual context — running entirely on-device on Apple's native stack.
Problem
Existing aids for blind/low-vision and deaf/hard-of-hearing users either require a human or stream the most intimate data imaginable — everything the user sees and hears — to the cloud. That's a privacy violation and too slow for real-time use.
Approach
YOLOv8n detection + OCR + spoken description end-to-end at 40ms (29.4 FPS); on-device speech captioning at 3.6% WER; the speech denoiser moved onto the Neural Engine for an ~88× speedup (1149ms → 13ms per 4s chunk). Provably private: 0 bytes written to disk, 0 bytes on the network (verified with lsof). Fairness-audited across 6 English accents and 5 lighting conditions with significance testing.
Impact
Continues the assistive-tech thread from Acoustic Horizons with modern on-device ML: measured performance, fairness auditing, and privacy by construction.