Project Overview
My Role
Lead Developer & Audio DSP Architect responsible for C++ infrastructure, real-time thread safety, and optimization of low-latency model inference under strict audio buffer limits.
The Problem Context
Existing neural text-to-speech and voice-cloning engines suffer from high latency (often over 250ms), making real-time live performance, recording, or interactive DAW control impossible without noticeable delay and digital jitter.
- Sub-optimal multi-thread safety causes audio drops and clicks when inference workloads hit the CPU concurrently with the DAW buffer cycle.
- Unoptimized memory allocation in standard neural inference frameworks causes memory fragmentation inside memory-constrained hosts.
- High processing overhead forces high sample buffer configurations, limiting application in low-latency recording setups.
We bypassed heavy external framework overhead by designing a native C++ runtime that directly links the model weights with quantized CPU execution vectors, lowering internal buffer delay to 5.8 milliseconds.
Key Features & Workflow
Sub-6ms Total Latency
Utilizes optimized execution vectors to calculate audio buffers in parallel, running voice synthesis within live recording requirements.
Parametric Expression
Exposes core vocal parameters (tensity, breathiness, and resonance) to external MIDI automation lanes within any standard DAW environment.
Quantized Local Engine
Operates fully on-device using a compressed INT8 model footprint. No network roundtrip or remote cloud resources required.
Technical Architecture
How the system components fit together.
JUCE Audio Engine Wrapper
Houses custom multi-channel ring buffers to split audio streaming processing from high-priority execution lanes.
ONNX Runtime Integration
Leverages ONNX CPU quantization pathways to run low-level vector matrix operations using SSE and AVX instructions.
Vocoder Synthesis Pipeline
Employs an audio-conditioned generative adversarial vocoder structure to maintain human-like phase relationships.
Design Decisions
To keep the plugin responsive under heavy load, we isolated the system UI thread from the critical realtime audio callbacks. Decoupling operations prevented render-blocking interruptions during active visual interface parameter updates.
- Created lock-free, single-producer single-consumer circular buffers to pass control parameters safely between processing and user interface threads.
- Implemented runtime model compilation with targeted hardware acceleration paths based on target CPU instructions (e.g. AVX-512 vs. Apple Silicon NEON).
- Programmed a structured dry-wet mix dynamic bypass to automatically route unaffected signals around the inference blocks if system dropouts are imminent.
Future Roadmap
- Implementation of CLAP (Clever Audio Plug-in) support for robust host automation handling.
- Real-time acoustic feedback cancellation to prevent local mic-spill from corrupting synthesis accuracy.
- GPU-accelerated desktop execution paths for high-density multi-mic configurations.
Technology Stack
Core technologies used to build this project.
Capabilities Demonstrated
Key areas of expertise highlighted by this project.