AI Portrait Customization Technology for 3D Digital Humans

This paper presents an AI portrait customization system for 3D digital humans, specifically designed for the Taobao Life 2 application. The system addresses the challenge of generating personalized photos while maintaining identity consistency using Stable Diffusion technology.

The proposed solution offers several advantages over existing methods: significantly reduced processing time (approximately 20 seconds vs. 2-5 minutes for LoRA-based approaches), and improved diversity in both content and style. The system processes 3D model renderings through multiple stages including image preprocessing (face beautification and enhancement), feature extraction, and a novel reference image injection technique that preserves identity information while maintaining background diversity.

The technical approach involves generating intermediate layouts using prompt-only control, then applying a blend diffusion technique to incorporate diverse backgrounds while preserving facial identity. The system also includes post-processing steps to handle common Stable Diffusion artifacts and ensure high-quality output. The solution has been deployed in production with positive user feedback and supports multiple artistic styles through a library of LoRA models and prompt templates.

The system demonstrates effectiveness in generating customized portraits that maintain the user's identity while offering diverse artistic interpretations, making it suitable for applications in virtual try-on, social media, and digital avatar creation.