Confucius4-R2T2: A Stable-Prefix Streaming ASR Model for Real Real-Time Transcription
Streaming automatic speech recognition is often judged by accuracy and speed, but systems that revise already displayed text create a third problem: instability. A caption may flicker, a translation system may redo work, and a voice agent may act on words that later disappear. The authors of Confucius4-R2T2 propose a different target for streaming ASR: once text is released, it should never be retracted.
The problem: partial speech is not final speech
A full-context recognizer can use the entire utterance to resolve ambiguity. A streaming recognizer cannot. When it receives only the first few hundred milliseconds, it may guess a continuation and then revise that guess as more audio arrives. Conventional commitment rules reduce this flicker by waiting for agreement between hypotheses, but they do not guarantee that the underlying model will stop changing previously released text.
The paper treats stability as a behavioral property of the complete recognizer and decoding policy. At every point in time, the new output must extend the old output. Stability is separate from accuracy and latency: a system that emits nothing until the end is stable but not useful in real time, while a system that emits arbitrary fixed text is stable but inaccurate.
Longest Stable Prefix learning
Confucius4-R2T2 is adapted from Qwen3-ASR-1.7B through post-training on approximately 1,500 hours of Chinese and English audio. The authors freeze the pretrained audio encoder and projector and train the language decoder instead. This preserves the base model’s bidirectional acoustic representations rather than converting the encoder into a causal architecture.
The central idea is the Longest Stable Prefix, or LSP. The researchers first run the base model on progressively longer prefixes of an utterance and also obtain its full-audio output. At each point, the LSP is the longest earlier output that is entirely consistent with that final output. Crucially, the label must correspond to a complete output the model actually produced, rather than a substring assembled by comparing partial hypotheses.
This creates a conservative but useful training signal. If the model has not produced a trustworthy continuation, the target includes a special Wait token. The decoder therefore learns two related decisions: what text is supported by the available audio, and when it should defer the uncertain continuation. The authors use supervised fine-tuning followed by reinforcement learning to improve recognition, waiting behavior, and resistance to repetition, hallucination, language inconsistency, and unsupported tails.
The approach requires no architectural changes. At inference, users can choose chunk sizes from 80 milliseconds to two seconds and adjust a provisional-token budget that controls how much text can remain internally revisable before commitment. The released output, however, remains append-only.
What the evaluations show
The authors evaluate English and Chinese benchmarks, compare against open-source and commercial systems, and test transfer to ten additional languages. Their main streaming configuration uses 160-millisecond chunks.
On the reported English and Chinese test sets, R2T2’s streaming recognition approaches the base model’s full-context quality. The paper reports a combined macro error of 5.76 for R2T2 with one provisional token, compared with 5.15 for the base model in offline mode. The authors also report that R2T2 improves on the unadapted base model at the same short chunk size across the listed English and Chinese sets. In their comparison, its combined error is lower than the evaluated open-source and proprietary streaming configurations, although the offline and streaming references are not identical operating points.
The model also transfers beyond the languages used for post-training. According to the paper, R2T2’s error is lower than the base model’s at the same 160-millisecond setting on all 18 test sets covering ten additional languages. That result suggests that the learned behavior may be more general than memorizing language-specific streaming patterns, but it does not establish universal multilingual performance.
Latency results are promising but need careful interpretation. The authors report low retrospective emission lag, with an 80-millisecond, zero-provisional-token mode staying below half a second on some English and Chinese measurements. They also show a practical trade-off: withholding one provisional token can improve recognition while adding some delay. Smaller chunks are not automatically faster, because the model may still wait for enough context to make a stable commitment.
Important limitations
The strongest guarantee concerns released text, not necessarily the model’s internal hypotheses. R2T2 can keep a provisional suffix private and reconsider it, while ensuring that committed text never changes. Applications must still decide how to flush final output and detect the end of speech.
The latency numbers are retrospective audio-timeline measurements, not end-to-end serving benchmarks. The paper explicitly excludes execution, queueing, transport, and rendering time, and notes that some API timing estimates rely on undisclosed buffering. Real deployment performance therefore remains unverified. The evaluation also uses approximately 1,500 hours of proprietary post-training data, and the authors do not provide controlled ablations for every reward component or inference optimization. Finally, the LSP target is anchored to the base model’s own full-context output; it does not guarantee that the eventual transcript is correct. WER and CER remain essential.
Why this matters in practice
For live captions, incremental translation, meeting systems, and voice agents, stable-prefix output changes the integration contract. Downstream components can process committed text once instead of repeatedly undoing and rebuilding work. This can simplify state management and reduce the risk that an agent triggers an action from a transient hypothesis.
The paper’s broader lesson is that streaming conversion does not always require redesigning the acoustic encoder. A full-context model can be taught to wait, using its own outputs to generate stability-aware supervision. That is a compelling engineering path, but teams considering deployment should benchmark real-time factor, memory use, tail latency, silence handling, domain robustness, and error recovery on their own workloads. R2T2 demonstrates a strong research result: stable output, useful latency, and near-full-context recognition can be pursued together rather than treated as mutually exclusive goals.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.