How Transformers Revolutionize Image Style Transfer: Introducing StyTr²

Introduction

Image stylization, which renders a content image in the visual style of a reference image, has long been a popular research topic and is now widely deployed in short‑video applications such as Kuaishou’s main app, Quick App, Yitiao Camera, and Kuaiying. Traditional texture‑synthesis methods produce vivid results but are computationally heavy. CNN‑based neural style transfer simplifies the process by adjusting second‑order statistics of feature maps, yet it often fails to model content‑style relationships adequately, leading to unsatisfactory results. Recent works incorporate self‑attention mechanisms to improve stylization quality.

Background

Transformers, successful in natural language processing, have been adopted for vision tasks because self‑attention enables each layer to capture global information and relational structures, preserving fine details that CNNs may lose during deep feature extraction.

Method

The paper proposes StyTr², a Transformer‑based image stylization algorithm that addresses the content‑style bias of CNN methods. The architecture (Figure 2) consists of three components: a content Transformer encoder, a style Transformer encoder, and a Transformer decoder that fuses content features with style features to generate the stylized output.

To better suit visual tasks, the authors introduce Content‑Aware Positional Encoding (CAPE), which (1) incorporates semantic information into positional encoding for image generation, and (2) remains scale‑invariant for multi‑scale inputs, overcoming the limitations of traditional sinusoidal encodings.

Results

Qualitative comparisons (Figures 4 and 5) show that StyTr² outperforms state‑of‑the‑art CNN‑based methods, preserving content structure while delivering richer style patterns even after multiple stylization rounds. The Transformer‑based model captures long‑range dependencies, avoids detail loss, and produces high‑quality stylizations.

Conclusion

StyTr² demonstrates that Transformer architectures can advance the frontier of image stylization and broaden the application of Transformers in visual generation tasks.

Paper: https://arxiv.org/abs/2105.14576

Code: https://github.com/diyiiyiii/StyTR-2