  Large-scale pre-training tasks like image classification, captioning, or
self-supervised techniques do not incentivize learning the semantic boundaries
of objects. However, recent generative foundation models built using text-based
latent diffusion techniques may learn semantic boundaries. This is because they
have to synthesize intricate details about all objects in an image based on a
text description. Therefore, we present a technique for segmenting real and
AI-generated images using latent diffusion models (LDMs) trained on
internet-scale datasets. First, we show that the latent space of LDMs (z-space)
is a better input representation compared to other feature representations like
RGB images or CLIP encodings for text-based image segmentation. By training the
segmentation models on the latent z-space, which creates a compressed
representation across several domains like different forms of art, cartoons,
illustrations, and photographs, we are also able to bridge the domain gap
between real and AI-generated images. We show that the internal features of
LDMs contain rich semantic information and present a technique in the form of
LD-ZNet to further boost the performance of text-based segmentation. Overall,
we show up to 6% improvement over standard baselines for text-to-image
segmentation on natural images. For AI-generated imagery, we show close to 20%
improvement compared to state-of-the-art techniques. The project is available
at https://koutilya-pnvr.github.io/LD-ZNet/.

Computer Vision and Pattern Recognition