VeriSpoof: Beyond Real or Fake with One Audio LLM
Most speech deepfake detectors answer a narrow question: does this recording sound real or synthetic? The technical report by Yang Xiao and Ting Dang, from the University of Melbourne, argues that forensic analysis needs more. Investigators may also need to identify the generating system, compare two recordings, locate a manipulated segment, understand recording conditions, and explain why a decision was made.
VeriSpoof is designed as one audio language model for those related tasks. Rather than adding a separate output head for every question, it frames each task as scored question answering. The model assigns likelihoods to candidate answer sentences, allowing binary decisions such as genuine versus spoofed or same source versus different source to be thresholded using an equal-error-rate operating point.
One model for several forensic questions
The system combines a speech encoder, an audio tokenizer, and a language model. The encoder processes an eight-second waveform and produces two kinds of representations: utterance-level tokens intended to capture global source information, and frame-level tokens that preserve timing information. The latter support localization of synthesized speech within a recording.
Those tokens are passed to a language model that has been adapted for forensic questions. VeriSpoof can address authenticity, generator attribution, acoustic-model and vocoder attribution, recording-condition analysis, and source verification for pairs of recordings. The authors also test whether it can compare recordings from generators that were not present during training.
The model is trained in three stages. First, the audio representation is aligned with detection and source questions. Next, the language model is tuned on the broader set of forensic questions. Finally, the system learns to produce grounded rationales. Only a small fraction of the underlying parameters is updated: the report says 29.7 million of 2.85 billion parameters are trained.
Rationales that can be checked
The most distinctive part of VeriSpoof is its approach to explanation. Its rationales use fixed fields rather than unrestricted prose. Observations such as signal-to-noise ratio, speech level, bandwidth, and clipping are checked against waveform measurements. Synthesized spans are checked against annotated locations, while source profiles are checked against generator metadata. Recording-condition statements are evaluated against the logged augmentation or condition.
This does not make every statement correct. In one example, the reported bandwidth is wrong even though the localization and final decision are correct. That failure is visible because the statement has an independent check. The approach therefore treats an explanation as an auditable collection of claims, not simply as persuasive text.
The authors add counterfactual training to make the final decision depend on the stated evidence. In some training examples, the localization is deliberately reversed and the model is trained only on the decision that should follow. In evaluation, the model followed paraphrased or otherwise unseen localization edits in 99.5% of cases, compared with 73.8% without counterfactual training.
Reported results
On the 14 test sets in Speech-DF-Arena, VeriSpoof-Plus achieved a pooled equal error rate of 2.14%. The authors place it third among 28 systems in the cited public snapshot and first among open-weight systems. VeriSpoof-Base, trained only on official training splits from eight corpora, reached 4.43% pooled EER. These results are strong, but the comparison includes proprietary systems whose training data are not disclosed, and the authors did not submit the models to the leaderboards; they computed results using the published protocol files.
The same weights also performed source verification. On pairs from generators unseen during training, VeriSpoof-Plus reached 6.17% EER for new models using known architectures and 15.00% for new architectures. Performance weakened on noisy, reverberant, or codec-processed copies, where source-specific details are harder to preserve.
Grounded reasoning was especially useful for partial fakes. On PartialEdit, the decision after reasoning reached 99.6% accuracy, while the direct one-step answer reached 78.4%. The report attributes this gap to the rationale process: writing a synthesized span first encourages frame-level inspection instead of summarizing the entire recording in one judgment.
What the controlled studies suggest
The component studies separate several effects that are often conflated. According to the authors, the audio representation and its training largely determine detection quality. The language model contributes more to comparing two recordings and to composing evidence into a useful answer. This matters for system design: a larger language model alone may not solve missing or weak audio evidence.
Source supervision also had a nuanced effect. It helped on many test sets when compared with an authenticity-only model receiving the same number of authenticity examples, and it was particularly important for neural-codec fakes. However, when the authenticity-only model received roughly five times more authenticity training examples, source supervision often became equivalent rather than clearly better. It also hurt on one evaluation set, ASVspoof 2021 LA.
Limitations and practical implications
The results remain benchmark results from a technical report. Generalization to operational forensic evidence, unfamiliar languages, microphones, transmission paths, and adversarial manipulation is not established here. The Plus model also shows calibration problems across domains: its single pooled threshold produced substantial false positives on some genuine speech subsets, leading the authors to recommend threshold calibration on genuine audio from the deployment domain.
Rationales are not a substitute for expert review. Their checks depend on available labels, metadata, or waveform measurements, and localization quality declines when many short manipulated segments must be described in text. Source verification also degrades under channel changes. The report estimates that training and inference can fit relatively modest GPUs, but some of those lower-memory figures are estimates rather than completed training runs.
For practitioners, VeriSpoof points toward a useful workflow rather than an autonomous verdict. A fast score could screen large collections, while grounded rationales could prioritize recordings for human examination. The released models, inference code, demo, and VeriSpoof-Bench may make it easier to evaluate that workflow. The broader lesson is that audio deepfake detection becomes more valuable when it answers the follow-up questions—what was altered, where, by what kind of system, and what evidence supports the conclusion—without confusing fluent explanation with proof.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.