LociArt: A Unified Paradigm for Free-Style Local Video Stylization

LociArt: A Unified Paradigm for Free-Style Local Video Stylization

Video stylization has become a vibrant area in computer vision and graphics, enabling the transformation of ordinary footage into artistic renditions. While many existing methods focus on global stylization—applying a uniform style across an entire video—artists and creators often require more nuanced control. They may want to stylize only specific regions, such as turning a moving subject into a pencil sketch while leaving the background untouched or applying different artistic styles to multiple objects within the same scene. This challenge, known as free-style local video stylization, demands precise spatial and temporal control to maintain the integrity of non-target regions and ensure smooth transitions over time.

The Problem: Localized Artistic Control in Video Stylization

Local video stylization involves two key aspects: where to apply the style and what style to apply. Achieving this requires accurate region segmentation, reliable association of styles to regions, and temporal consistency to avoid flickering or artifacts across frames. Existing approaches often struggle with these requirements because binary masks used for segmentation do not encode style identity, and mask representations can lose spatial fidelity or temporal context during processing. Moreover, paired training data that explicitly demonstrate local style changes while preserving the rest of the scene are scarce, complicating supervised learning.

Introducing LociArt: A Unified Framework

To address these challenges, the authors propose LociArt, a comprehensive paradigm that integrates data construction, model design, and evaluation for free-style local video stylization. The framework consists of three main components:

  • LociArt-100K Dataset: An automatically constructed dataset of 100,000 video pairs with localized stylization, combining global stylization, object removal, matting, and compositing to generate high-quality training examples.
  • GRNStyle Model: A novel generative refinement network that explicitly binds style identities to regions using a polar style-aware mask and combines spatiotemporal context with geometric control.
  • LociArt Benchmark: A region-aware evaluation suite with 80 source videos and 84 object-mask units, measuring regional style fidelity, non-target content preservation, and temporal consistency.

Data Construction Pipeline

The LociArt-100K dataset is created through a multi-step automated pipeline:

  1. Object Identification and Removal: Salient objects are detected and segmented using state-of-the-art vision-language models and segmentation tools. These objects are then removed from the video to create clean backgrounds.
  2. Global Stylization: Both the original video and the object-removed background are globally stylized using multiple stylization models under the same style instructions.
  3. Matting and Localized Compositing: Alpha mattes are estimated to accurately capture object boundaries, enabling seamless compositing of stylized foregrounds or backgrounds with preserved complementary regions.
  4. Human Verification: Annotators review the composited videos to ensure style quality, content preservation, boundary coherence, and temporal stability.

This pipeline produces paired examples where only the target region is stylized, and the rest of the scene remains faithful to the source, providing strong supervision for local stylization.

GRNStyle: Model Architecture and Conditioning

At the core of LociArt is GRNStyle, a model built on Generative Refinement Networks (GRN). It introduces two key innovations:

  • Polar Style-Aware Mask: Instead of a simple binary mask, the model encodes style identity directly into the mask using a polar coordinate representation. Each style corresponds to a unique direction on a unit circle, embedded as two channels, while a third channel encodes mask occupancy. This design explicitly binds style and region, enabling the model to learn spatially localized style transformations.
  • Representation–Geometry Conditioning: To balance spatial fidelity and temporal context, the model combines encoded mask features with raw binary mask support. This approach preserves fine boundary details while providing rich spatiotemporal conditioning, enhancing the model’s ability to localize stylization effects accurately over time.

GRNStyle also leverages source evidence to guide content preservation, refining video tokens iteratively to produce stylized outputs that maintain non-target regions.

Evaluation: The LociArt Benchmark and Metrics

The authors establish the LociArt Benchmark to rigorously evaluate local video stylization. It includes:

  • 80 source videos with 84 object-mask units, supporting both foreground and background stylization protocols (168 cases total).
  • Metrics such as Style Prototype Similarity (SPS), which compares stylized regions to learned style prototypes; regional CLIP-T similarity, assessing alignment with style text prompts; PSNR for non-target preservation; and temporal consistency (TC) measuring frame-to-frame stability.
  • Complementary human evaluations using vision-language models to score style fidelity and preservation separately.

Experimental Results and Evidence

GRNStyle was trained on the LociArt-100K dataset and compared against several state-of-the-art methods, including image-based and video-based stylization approaches. Key findings include:

  • Superior Regional Style Quality: GRNStyle achieved the highest SPS scores for both foreground and background stylization, indicating more faithful and localized style application.
  • Strong Preservation and Temporal Consistency: It maintained high PSNR in non-target regions and competitive temporal consistency, balancing stylization with content fidelity.
  • Effective Multi-Region and Multi-Style Composition: Despite training only on single-region, single-style pairs, GRNStyle demonstrated zero-shot capability to stylize multiple regions with distinct styles simultaneously.
  • User Study Confirmation: Human raters favored GRNStyle over competitive video-based baselines in style fidelity and background preservation across multiple styles.

Additional ablations showed that the polar style-aware mask and representation–geometry conditioning both contribute significantly to performance, improving spatial control and style fidelity without sacrificing preservation.

Limitations and Future Directions

The authors acknowledge several limitations:

  • Temporal Consistency Challenges: Rapid object motion can degrade temporal stability, especially near boundaries.
  • Shadow Preservation: The object removal step may inadvertently remove shadows, leading to unnatural stylizations.
  • Boundary Evaluation: Current metrics do not explicitly measure boundary adherence, which is important for local editing quality.

Future work may focus on motion-aware conditioning, shadow-aware data construction, and style-aware boundary evaluation to further enhance local video stylization.

Practical Implications

LociArt provides a scalable and effective approach for artists, filmmakers, and content creators who seek precise and flexible control over video stylization. By enabling free-style local editing with strong spatial and temporal fidelity, it opens new possibilities in digital art, advertising, and film post-production. The publicly available dataset and benchmark also offer valuable resources for advancing research in controllable video generation and editing.

“Together, these components provide a unified foundation for learning and evaluating controllable local video stylization.” — Authors


Sources

AI-assisted research summary. Check the original paper for complete methods, results, and limitations.

Avatar photo

Tech News

Jeff is a passionate blog writer who shares clear, practical insights on technology, digital trends and AI industries. With a focus on simplicity and real-world experience, his writing helps readers understand complex topics in an accessible way. Through his blog, Jeff aims to inform, educate, and inspire curiosity, always valuing clarity, reliability, and continuous learning.

↑ ↓ navigate Enter open Esc close