MatAnyone-E: Efficient High-Resolution Video Matting via Sparse Memory Propagation
High-resolution video matting is a challenging computer vision task that involves extracting detailed alpha mattes from video frames. These mattes enable foreground subjects to be composited onto new backgrounds, preserving fine structures like hair strands and soft transparency. Achieving high-quality matting with temporal consistency across frames becomes computationally demanding as video resolution increases, limiting practical applications.
The Challenge of High-Resolution Video Matting
Video matting methods must balance three key demands: precise recovery of fine details, temporal stability to avoid flickering, and computational efficiency. Existing approaches often trade off these aspects. Efficiency-focused methods reduce computation but may lose detail or consistency, while memory-based methods leverage information from previous frames to improve quality but incur high computational costs, especially at native 4K or higher resolutions.
Memory-based techniques, such as the MatAnyone series, achieve strong matting quality by matching features from the current frame against a memory bank of features from earlier frames. However, the exhaustive matching process grows rapidly in cost as resolution increases, since both the query and memory feature grids enlarge. Additionally, decoding high-resolution alpha mattes is computationally intensive.
MatAnyone-E: Key Innovations
To address these challenges, the authors present MatAnyone-E, an efficient extension of MatAnyone 2 that significantly reduces inference time and memory usage while maintaining comparable matting quality. The core idea is to focus expensive high-resolution computations only where they are most needed.
- Sparse Memory Propagation (SMP): Instead of performing dense high-resolution matching across all spatial locations and memory entries, MatAnyone-E first conducts a low-resolution propagation pass. This inexpensive step estimates uncertainty and coarse affinities to identify spatial regions and memory candidates that require detailed inspection.
- Selective High-Resolution Matching: High-resolution matching is then performed only on the selected uncertain query locations and relevant memory candidates. For other regions, the model reuses upsampled low-resolution matching results. This dual pruning along both query and memory dimensions drastically reduces the number of expensive comparisons.
- Lightweight Decoder: Recognizing that decoding high-resolution mattes is another bottleneck, the authors design a lightweight decoder. Guided by profiling runtime bottlenecks, this decoder uses depthwise-separable convolutions in early stages and narrower dense convolutions in later stages to balance arithmetic cost and memory access overhead. It is trained via feature distillation from the original decoder to preserve matting quality.
Evidence and Results
The authors evaluate MatAnyone-E on several high-resolution video matting benchmarks, including VideoMatte, YouTubeMatte, and CRGNN, at 1920×1080 resolution and beyond. Key findings include:
- Matting Quality: MatAnyone-E achieves matting accuracy comparable to MatAnyone 2, with minimal differences in metrics such as mean absolute difference (MAD), mean squared error (MSE), gradient error, and temporal consistency error. It outperforms recent methods like VideoMaMa and SAM2Matting, especially in preserving fine hair structures and soft boundaries.
- Inference Efficiency: At native 4K resolution (3840×2160), MatAnyone-E achieves an estimated 7.4× speedup in inference time and reduces peak GPU memory usage to 7.6% of MatAnyone 2’s requirements. It reaches 8.1 frames per second (FPS) on an NVIDIA A800 GPU, with further improvements to 13.6 FPS when using compilation optimizations. At 1080p, it attains 26.6 FPS, doubling the speed of MatAnyone 2.
- Qualitative Improvements: Visual comparisons demonstrate that MatAnyone-E better preserves fine details such as thin hair strands and complex boundaries, particularly at 4K resolution where competing methods either run out of memory or operate at lower processing resolutions.
Methodological Insights
The Sparse Memory Propagation approach is central to MatAnyone-E’s efficiency. The process begins by downsampling queries and memory features by a factor (e.g., 2×). Low-resolution matching estimates uncertainty scores indicating which spatial locations require more precise matching. These uncertain locations are selected for high-resolution matching, while others rely on upsampled low-resolution results.
Memory pruning complements query pruning by selecting only the most relevant memory candidates associated with the uncertain queries. This two-sided pruning reduces the total number of query-memory pairs that must be compared at high resolution.
The lightweight decoder design is informed by profiling the original decoder’s runtime and memory access patterns. Early decoder stages are compute-bound and benefit from depthwise-separable convolutions to reduce arithmetic operations. Later stages are memory-bound, so the authors reduce channel widths instead of convolutional complexity to minimize memory traffic. Feature distillation from the original decoder ensures that the lightweight decoder maintains representational fidelity.
Limitations and Considerations
While MatAnyone-E demonstrates significant efficiency gains, the authors note that the benefits of Sparse Memory Propagation depend on the input resolution. At lower resolutions, the downsampled feature grids become too coarse to reliably identify uncertain regions or relevant memory, limiting pruning effectiveness. Thus, the method is most advantageous for high-resolution inputs (e.g., 4K and above).
Additionally, memory pruning yields more pronounced gains at very high resolutions where the memory bank size is large. At moderate resolutions, its impact is less significant.
Finally, the lightweight decoder trades some representational capacity for efficiency, though feature distillation mitigates quality loss. The approach assumes access to pretrained MatAnyone 2 models for distillation.
Practical Implications
MatAnyone-E offers a practical solution for deploying high-quality video matting in real-world applications requiring native 4K or higher resolution processing. Its substantial reductions in inference time and GPU memory usage enable smoother workflows in video editing, virtual production, and augmented reality scenarios where fine detail and temporal consistency are critical.
The selective computation paradigm introduced by Sparse Memory Propagation may inspire similar efficiency improvements in other memory-based video processing tasks, such as video object segmentation and tracking.
Moreover, the profiling-guided lightweight decoder design highlights the importance of considering hardware-specific runtime characteristics beyond theoretical operation counts when optimizing deep learning models for high-resolution tasks.
Conclusion
The authors of MatAnyone-E successfully address the computational bottlenecks of memory-based high-resolution video matting by introducing a sparse memory propagation strategy and a lightweight decoder. Their method preserves the strong matting quality of prior work while achieving up to 7.4× speedup and over 13× reduction in peak GPU memory at 4K resolution. This work represents a significant step toward practical, real-time high-resolution video matting.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.