AuK Technical Report — An Open-Source Foundational Model for Speech Generation and Editing
Source Content
AuK is an open-source foundational model that unifies speech generation and editing through one interface: natural-language instructions plus audio context. The paper is a technical report (arXiv cs.SD, submitted 8 Sep 2026) announcing the model, its training recipe, and an open release of code and weights.
Scale
- ~3.03 billion instruction–audio instances
- 1.95 million hours of effective supervision
- Five task families: speech generation, content editing, enhancement and separation, paralinguistic editing, and acoustic editing
Architecture
Three coordinated components:
- Multimodal LLM — semantic conditioning (turns instructions into intent)
- VAE — acoustic conditioning, jointly trained on speech, general audio, and music
- Hybrid rectified-flow Transformer — dual-stream MMDiT blocks followed by unified single-stream DiT blocks for generation
Training & Post-training
- Generation-only warm-up → joint generation–editing pre-training
- Human-feedback preference optimization for open-ended editing
- Reward-based reinforcement learning for speech generation
Distillation (AuK-Flash)
To cut inference cost, the team distills the model with consistency initialization and task-routed Decoupled DMD. The result, AuK-Flash, does 4-step inference without classifier-free guidance and achieves a 4.5× wall-clock speedup over the full model under matched conditions.
Results
Leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing; competitive on signal-level restoration tasks. Code and weights released for reproducibility.