MiDashengLM-Gen
Unified Audio Scene Generation via LLM-Driven Autoregressive Flow Matching
Overview
MiDashengLM-Gen is an end-to-end framework that uses a pre-trained Large Language Model and audio tokenizer as the backbone, combined with per-token conditional flow matching for autoregressive, variable-length mixed-audio scene generation. It generates coherent 16 kHz audio scenes that simultaneously blend speech, music, sound effects, and environmental acoustics from structured text descriptions.
Highlights
- Autoregressive generation with per-token flow matching — combines LLM sequence modeling with continuous flow-based generation, enabling variable-length output via a learned stop head
- LLM-conditioned high-dimensional audio latents — generates over 768-dimensional semantic-acoustic latents (25 Hz) conditioned on LLM hidden states, achieving rich acoustic modeling without quantization artifacts
- Audio-text alignment pre-training — bridges the modality gap by mapping audio latents into the LLM's token space before generation training, critical for cross-modal synthesis
- Strong speech intelligibility with competitive mixed-audio quality — approaches dedicated TTS performance on Seed-TTS while maintaining mixed-audio scene generation, and supports 9 languages with emotion control
Architecture
| Component | Details |
|---|---|
| Audio Tokenizer | DashengTokenizer → 768-dim latents @ 25 Hz, downsampled to 5 Hz via audio projector |
| LLM Backbone | Qwen3-1.7B, fully fine-tuned |
| Flow Matching DiT | 16 layers, hidden dim 2048, 8 heads, MLP ratio 4.0 |
| Inference | 10-step Euler ODE solver, CFG scale 2.0 |
| Stop Head | Learned binary classifier for variable-length truncation |
| Output | 16 kHz mono WAV, auto-truncated |
Input Format
Input uses structured multi-view captions with special tokens to describe different aspects of an
audio scene. Use <|unknown|> for absent elements.
| Token | Description | Example |
|---|---|---|
<|caption|> | Overall scene description | A comedian delivering a punchline followed by uproarious crowd laughter and an upbeat jazz band hit |
<|asr|> | Speech transcript | And that is why I never buy cheap luggage anymore! |
<|speech|> | Speaker characteristics (voice, emotion, style) | expressive comedic male voice |
<|sfx|> | Sound effects | uproarious crowd laughter |
<|music|> | Music description | sudden upbeat jazz band sting |
<|env|> | Environment / ambience | intimate comedy club |
Audio Demonstrations
Audio samples across seven categories, demonstrating the model's unified generation capabilities.
Citation
@article{sun2026midashenglmgen,
title={MiDashengLM-Gen: Unified Audio Scene Generation via LLM-Driven Autoregressive Flow Matching},
author={Sun, Xingwei and Dinkel, Heinrich and Li, Gang and Mei, Jiahao and Niu, Yadong and Han, Zerui and Jiang, Yuepeng and Zhou, Jiahao and Fan, Lichun and Luan, Jian},
journal={arXiv preprint},
year={2026}
}Licensed under the Apache License 2.0.