CoSTA: Compressed Speech Token Adaptation for Efficient Spoken Dialogue Models
Spoken dialogue models can understand speech and answer with speech, but their responsiveness is constrained by a basic representation problem. The same speech-token space often serves two different jobs: autoregressive generation and waveform synthesis. Because speech requires many tokens, the language model must generate long sequences one step at a time before the decoder can produce audio. That increases latency, consumes generation budgets, and makes sustained conversation more difficult.
In CoSTA: Compressed Speech Token Adaptation for Efficient Spoken Dialogue Models, Yingzhuo Deng and colleagues propose a way to shorten the sequence generated by the model without replacing the existing speech tokenizer or decoder. Their central idea is to separate the token space used during generation from the token space required for synthesis.
The problem with coupling generation and synthesis
Existing spoken dialogue systems commonly generate speech tokens that are directly consumed by a speech decoder. Lower-rate tokenizers can reduce sequence length, but the authors note two drawbacks: aggressive compression can affect speech quality, and changing the tokenizer may require coordinated adaptation of the tokenizer, decoder, and dialogue model.
CoSTA instead treats speech-token sequences as a losslessly compressible representation. It applies byte-pair encoding, or BPE, to adjacent speech tokens. A generated BPE unit can represent several original tokens, while a deterministic expansion step can always recover the original sequence before synthesis. The decoder therefore receives the format it already expects, even though the dialogue model generates fewer units.
How CoSTA adapts an existing model
The method is designed as a post-hoc adaptation pipeline with three main stages.
- Vocabulary extension. The authors add BPE speech units to the model vocabulary. Each new unit is initialized by averaging the embedding and language-model output rows of the original tokens in its deterministic expansion. Their analysis reports that mean initialization converges faster and produces more stable gradients than random or single-token initialization.
- Pronunciation adaptation. The model is trained on compressed text-to-speech data. Text tokens remain unchanged, while speech tokens are replaced with their BPE representations. A progressive schedule first updates the embedding and output layers, then unfreezes intermediate layers.
- Dialogue adaptation. The adapted vocabulary is then used in speech-to-speech dialogue training. This stage updates the language-model parameters while leaving the speech encoder, adaptor, and decoder frozen. During inference, generated BPE speech units are expanded deterministically and passed to the unchanged decoder.
This design preserves the original synthesis interface. The additional mechanism at inference is a deterministic expansion of the generated speech stream, rather than a new neural decoder.
What the experiments show
The authors evaluate CoSTA primarily with the 8B Step-Audio 2 mini model on bilingual Chinese and English spoken-dialogue benchmarks. They use an 80K-unit BPE extension in the main configuration, along with 300 hours of Chinese and English text-to-speech data and roughly 50K speech-dialogue examples per language. The evaluation covers content quality, speech quality, text-speech alignment, latency, throughput, and human judgments.
On the model side, CoSTA-ZH reaches a generated-token compression rate of 66.81% and 1.736 times the expanded speech-token generation speed of the base model. CoSTA-EN reaches 70.17% and 1.659 times the corresponding speed. Matched vanilla fine-tuning retains the original token sequence length and produces nearly unchanged generation speed, suggesting that the gains come from compressed-token generation rather than fine-tuning alone.
The system-level results are also notable. On a single A100 GPU, the authors report full-response latency reductions of 32.8% to 39.0% for Chinese and 22.4% to 26.0% for English across the tested concurrency levels. Throughput increases by 51.6% to 59.2% in Chinese and 25.8% to 38.6% in English. CoSTA uses at most 0.68 GiB more peak memory than the matched vanilla fine-tuned model in those comparisons.
Efficiency did not come at the expense of the measured quality metrics. On the reported benchmarks, CoSTA generally improves or preserves content scores and substantially lowers character or word error rates relative to matched vanilla fine-tuning, while UTMOS speech-quality scores remain comparable. A listening study with six evaluators produced more CoSTA wins than losses against both the base model and vanilla fine-tuning for speech fluency and semantic quality.
Important trade-offs and ablations
The paper does not suggest that a larger compressed vocabulary is always better. Increasing the added vocabulary from 10K to 80K improves compression and decoding speed in the tested setting. Increasing it to 160K produces only a small additional speed gain while lowering the reported G-Eval content score. The authors therefore identify 80K as a practical balance for their configuration, not as a universal optimum.
The ablations also show that dialogue adaptation is essential. Removing it raises the generated-token compression rate from about 66.73% to 81.05% and causes the G-Eval score to fall from 72.27 to 24.35 on the cited Chinese evaluation. In other words, learning to pronounce compressed units is not enough; the model must also learn to use them while answering spoken queries.
CoSTA transfers to OpenS2S, a different spoken-dialogue architecture with a different tokenizer and decoder. There, the authors report reductions in generated speech-token steps of 21.17% in Chinese and 18.64% in English, alongside improvements in selected content and alignment measures over matched fine-tuning. They interpret this as evidence that compressed-token adaptation can complement, rather than replace, lower-rate speech tokenization.
Limitations and practical implications
The strongest evidence is still bounded by the experimental setup. The dialogue adaptation data is largely synthesized speech-to-speech data, so performance on spontaneous, multi-speaker, emotionally varied, or noisy real-world conversations remains less certain. The evaluation focuses on general spoken dialogue rather than personalized speaking styles, emotional interaction, or diverse acoustic environments.
There is also a real engineering trade-off. Larger vocabularies shorten sequences but increase output-layer computation and memory. The one-time adaptation process requires substantial data preparation and training, so teams must compare those costs with expected inference savings.
For practitioners with an existing spoken dialogue model, CoSTA offers a compelling compatibility-preserving option: retain the tokenizer and decoder, expand the generation vocabulary, train the model to use compressed speech units, and measure quality and latency jointly. The authors’ results show that this can make responses faster and interactions more sustainable, but the best vocabulary size and adaptation recipe should be validated on the target language, hardware, and conversational data.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.