Preserving instance-level characteristics for multi-instance generation

Abstract

Recently, there have been efforts to explore instance-level control in diffusion models, where multiple instances are generated independently and then integrated into a single scene. However, several issues arise when instances are closely positioned or overlapping. First, independently generated instances frequently differ in style and lack coherence, leading to changes in their attributes as they influence each other when merged. Second, instances often merge with one another or become absorbed into others. To tackle these challenges, we propose a local latent refinement (LLR) that enforces each local latent to meet its conditions and remain distinct from others. We also propose a local latent injection (LLI) method that gradually integrates local latents during global latent generation for smoother fusion. Also, we find that the variance of latents changes significantly after instance fusion, which greatly impacts the quality of the generated images. To remedy this, we apply an instance normalization layer to regulate the variance of the fused latents, thereby producing high-quality images. Extensive experiments demonstrate that our approach achieves both high fidelity in instance layout and superior image quality, even in cases of high overlap among instances.