FreeMix: Personalized Structure and Appearance Control Without Finetuning

Abstract

Personalized image generation has gained significant attention with the advancement of text-to-image diffusion models. However, existing methods face challenges in effectively mixing multiple visual attributes, such as structure and appearance, from separate reference images. Finetuning-based methods are time-consuming and prone to overfitting, while finetuning-free approaches often suffer from feature entanglement, leading to distortions. To address these challenges, we propose FreeMix, a finetuning-free approach for multi-concept mixing in personalized image generation. Given separate references for structure and appearance, FreeMix generates a new image that integrates both. This is achieved through Disentangle-Mixing Self-Attention (DMSA). DMSA first disentangles the two concepts by applying spatial normalization to remove residual appearance from structure features, and then selectively injects appearance details via self-attention, guided by a cross-attention-derived mask to prevent background leakage. This mechanism ensures precise structural preservation and faithful appearance transfer. Extensive qualitative and quantitative experiments demonstrate that our method achieves superior structural consistency and appearance transfer compared to existing approaches. In addition to personalization, FreeMix can be adapted to exemplar-based image editing.