Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes

Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes

Audio-visual target speaker extraction aims to isolate one person’s speech from a mixture by combining the microphone signal with visual evidence, typically the target speaker’s lip movements. This capability could improve video conferencing, hearing-assistance devices, and other latency-sensitive systems. But deployment creates a difficult trade-off: a model must handle changing rooms, noise, reverberation, and multiple interferers while processing audio continuously on limited hardware.

The authors argue that existing benchmarks and causal models leave two important gaps. Common datasets use exactly two speakers, relatively controlled mixtures, and no independent real-recorded evaluation set. Meanwhile, stronger causal architectures can be too computationally expensive for edge devices. Some lightweight systems reduce parameter counts by repeatedly applying a small separator, but those extra cycles still increase computation and latency.

A benchmark designed for less controlled scenes

To address the data problem, the authors introduce RealSSA, which has two parts. RealSSA-Sim is a controllable simulated suite covering four scene types: small meeting rooms, medium indoor spaces, large public areas, and outdoor environments. It varies room geometry, reverberation, environmental noise, sudden sound events, and speaker positions together rather than changing only one factor at a time.

The simulated scenes contain one to five dynamic near-field interfering speakers. Scenes with far-field interference add 10 to 15 additional far-field speakers. The benchmark contains 100,826 training samples, 9,133 validation samples, and 9,169 test samples, totaling approximately 153 hours. The authors also provide RealSSA-Real, a held-out real-recorded evaluation set covering 54 sessions and approximately 84 minutes across eight indoor, public, and outdoor environments. It includes uncontrolled background noise and overlapping speech, and was not used for training or validation.

This design is intended to test whether a model trained on simulation can generalize to deployment-like recordings. It also makes the difficulty of near-field interference more explicit. Nearby speakers can share similar reverberation and spectral characteristics with the target, making them harder to suppress than distant sources.

Falcon’s single-pass approach

The proposed model, Falcon, is a causal audio-visual extraction system. Its audio branch converts the mixture into a time-frequency representation, while a frozen visual backbone extracts mouth-region features. The two streams are fused before a separator estimates a representation of the target speaker and produces a complex-valued time-frequency mask. An audio decoder then reconstructs the waveform.

Falcon’s central architectural choice is a single-pass, multi-scale separator. Rather than repeatedly invoking a compact separator, it uses an asymmetric encoder-decoder with several resolutions. Time-frequency refinement blocks model frequency structure, causal temporal context, and joint time-frequency relationships. Skip connections preserve finer spectral detail while global context is injected into different decoder levels. According to the authors, this arrangement captures multiple time-frequency scales in one forward pass.

Training adds an encoder-space speaker contrastive loss to the usual SI-SNR reconstruction objective. The estimated target is pulled toward an encoded clean target and pushed away from encoded near-field interferers. The contrastive branch is used only during training, so it adds no inference-time computation. The authors also report that the loss can be attached to different audio-visual backbones without changing their inference architecture.

Results across simulated and real data

On the simulated benchmarks, Falcon achieved the highest reported SI-SNR improvement and SDR improvement among the compared causal systems on RealSSA-Sim, LRS2-2Mix, LRS3-2Mix, and VoxCeleb2-2Mix. On RealSSA-Sim, it reached 7.7 dB SI-SNR improvement, compared with 7.6 dB for causal AV-TFGridNet. On VoxCeleb2-2Mix, its SI-SNR improvement was 10.5 dB, versus 9.4 dB for Swift-Net-12.

The real-recorded results are more nuanced. Falcon achieved a DNSMOS score of 3.276 on RealSSA-Real, tying Swift-Net-12 and exceeding the reported score for causal AV-TFGridNet. Its word error rate was 32.74%, the second-lowest among the listed extraction systems, while its target-speaker similarity score of 42.23 was the highest. These measurements suggest useful generalization, but they do not mean that the model fully matches clean speech: RealSSA-Real has no isolated target waveform for direct signal-level comparison, and the authors note a substantial gap between Falcon’s speaker-similarity score and the ground-truth reference measured on simulated data.

The efficiency results are a major part of the paper’s contribution. Under the authors’ measurement protocol, Falcon used 0.5 million parameters, 10.6 billion MACs, and 51.3 milliseconds of GPU latency for a two-second input. Causal AV-TFGridNet used 6.2 million parameters, 213.8 billion MACs, and 567.8 milliseconds. The authors summarize this as 91.9% fewer parameters, 20.2 times less computation, and 11.1 times lower GPU latency. Falcon also had the lowest reported memory footprint among the evaluated systems.

What the evidence does and does not show

Ablation results support both design choices. Increasing separator depth from one to three downsampling stages improved SI-SNR improvement from 7.1 to 7.7 dB. Adding the contrastive loss improved Falcon by 0.4 dB, and also improved two other tested architectures. The authors therefore interpret the loss as a broadly useful training signal for near-field leakage suppression, although the experiments do not establish that it will transfer unchanged to every backbone or deployment condition.

Several limitations matter for practical adoption:

  • Single-channel operation: The main experiments use one channel, despite the simulated dataset being generated from a microphone-array setup.
  • Limited real-world evaluation: RealSSA-Real contains approximately 84 minutes from 10 participants and eight environments. It is valuable as a held-out test, but not a comprehensive survey of devices, languages, rooms, or populations.
  • Simulation-dependent training: The contrastive loss relies on clean per-source waveforms for near-field interferers, which are available in simulation but generally unavailable in unlabeled real recordings.
  • Visual dependence: In a missing-video experiment, complete loss of visual information caused Falcon’s SI-SNR improvement to fall to -9.91 dB. Sustained missing frames were also more damaging than randomly distributed missing frames.

Practical implications

For developers of conferencing or assistive-audio products, the paper’s clearest lesson is that realistic evaluation should include variable speaker counts, near- and far-field geometry, sudden noise, reverberation, and genuinely held-out recordings. A strong score on a fixed two-speaker mixture may not predict behavior in a crowded room.

Falcon’s reported efficiency makes it a plausible candidate for experimentation on edge hardware, but the authors’ latency numbers are measured under a defined batch-one protocol and include amortized forward-pass measurements. End-to-end streaming behavior still depends on scheduling, buffering, camera tracking, and device-specific implementation. The work demonstrates a promising quality-efficiency trade-off; production readiness remains an engineering question rather than a conclusion established by these experiments.


Sources

AI-assisted research summary. Check the original paper for complete methods, results, and limitations.

Avatar photo

Tech News

Jeff is a passionate blog writer who shares clear, practical insights on technology, digital trends and AI industries. With a focus on simplicity and real-world experience, his writing helps readers understand complex topics in an accessible way. Through his blog, Jeff aims to inform, educate, and inspire curiosity, always valuing clarity, reliability, and continuous learning.

↑ ↓ navigate Enter open Esc close