Pattern Analogies V1.0 Model Card

This respository contains TriFuser --- a diffusion model trained for analogical editing of pattern images as a part of our recent tech report "Pattern Analogies: Learning to Perform Programmatic Image Edits by Analogy".

Abstract

Pattern images are everywhere in the digital and physical worlds, and tools to edit them are valuable. But editing pattern images is tricky: desired edits are often \emph{programmatic}: structure-aware edits that alter the underlying program which generates the pattern. One could attempt to infer this underlying program, but current methods for doing so struggle with complex images and produce unorganized programs that make editing tedious. In this work, we introduce a novel approach to perform programmatic edits on pattern images. By using a \emph{pattern analogy}---a pair of simple patterns to demonstrate the intended edit---and a learning-based generative model to execute these edits, our method allows users to intuitively edit patterns. To enable this paradigm, we introduce SplitWeaver, a domain-specific language that, combined with a framework for sampling synthetic pattern analogies, enables the creation of a large, high-quality synthetic training dataset.We also present TriFuser, a Latent Diffusion Model (LDM) designed to overcome critical issues that arise when naively deploying LDMs to this task. Extensive experiments on real-world, artist-sourced patterns reveals that our method faithfully performs the demonstrated edit while also generalizing to related pattern styles beyond its training distribution.

Please check out our preprint for more information.

Model Details

TriFuser model uses the image-variation model of Versatile Diffusion as the starting point. It takes three images as input, (A, A*, B), and generates image B* as output, which satisfies the analogical relation A:A*::B:B*. The figure below shows the architecture of TriFuser in detail. Please read our pre-print for more information.

One single flow of Versatile Diffusion contains a VAE, a diffuser, and a context encoder, and thus handles one task (e.g., text-to-image) under one data type (e.g., image) and one context type (e.g., text). The multi-flow structure of Versatile Diffusion shows in the following diagram:

Developed by: Aditya Ganeshan, Thibault Groueix, Paul Guerrero, Radomír Měch, Matthew Fisher and Daniel Ritchie
Model type: Diffusion-based image2image generative model
Language(s): English
License: Adobe Research License
Resources for more information: More information along with training code will be released in this GitHub Repository.

Citation

TBD

Usage

You can use the model with the 🧨Diffusers library.

PatternAnalogiesTrifuser

This repository contains example inputs to demonstrate the model's capabilities. Please change EXAMPLE_ID from 0-9 to check out the different examples. Please first clone our repository: https://github.com/adobe-research/pattern-analogies and add it to the path path (as shown below). This is a temporary fix until we figure out how to make diffusers custom pipeline with custom modules work directly.

import requests
import torch as th
from PIL import Image
from io import BytesIO
import matplotlib.pyplot as plt
from PIL import Image, ImageOps
from diffusers import DiffusionPipeline
import sys
# CLONE: https://github.com/adobe-research/pattern-analogies
sys.path.insert(0, "PATH_TO_REPOSITORY")
SEED = 1729
DEVICE = th.device("cuda")
DTYPE = th.float16
FIG_K = 3
EXAMPLE_ID = 1

# Now we need to do the trick
pretrained_path = "bardofcodes/pattern_analogies"
new_pipe = DiffusionPipeline.from_pretrained(
    pretrained_path, 
    custom_pipeline=pretrained_path, 
    trust_remote_code=True
)

img_urls = [
    f"https://huggingface.co/bardofcodes/pattern_analogies/resolve/main/examples/{EXAMPLE_ID}_a.png",
    f"https://huggingface.co/bardofcodes/pattern_analogies/resolve/main/examples/{EXAMPLE_ID}_a_star.png",
    f"https://huggingface.co/bardofcodes/pattern_analogies/resolve/main/examples/{EXAMPLE_ID}_b.png",
]
images = []
for url in img_urls:
    response = requests.get(url)
    image = Image.open(BytesIO(response.content)).convert("RGB")
    images.append(image)

pipe_input = [tuple(images)]

pipe = new_pipe.to(DEVICE, DTYPE)
var_images = pipe(pipe_input, num_inference_steps=50, num_images_per_prompt=3,).images

plt.figure(figsize=(3*FIG_K, 2*FIG_K))
plt.axis('off')
plt.legend(framealpha=1)
plt.rcParams['legend.fontsize'] = 'large'
for i in range(6):
    if i == 0:
        plt.subplot(2, 3, i+1)
        val_image = images[0]
        label_str = "A"
    elif i == 1:
        plt.subplot(2, 3, i+1)
        val_image = images[1]
        label_str = "A*"
    elif i == 2:
        plt.subplot(2, 3, i+1)
        val_image = images[2]
        label_str = "Target"
    else:
        plt.subplot(2, 3,i + 1)
        val_image = var_images[i-3]
        label_str = f"Variation {i-2}"
    
    val_image = ImageOps.expand(val_image,border=2,fill='black')
    plt.imshow(val_image)
    plt.scatter([], [], c="r", label=label_str)
    plt.legend(loc="lower right")
    plt.axis('off')
plt.subplots_adjust(wspace=0.01, hspace=0.01)

Full GitHub Repository

Will be released soon.

Cautions, Biases, and Content Acknowledgment

We would like the raise the awareness of users of this demo of its potential issues and concerns. Like previous large foundation models, the use of our model could be problematic in some cases, partially due to the imperfect training data and pretrained network (VAEs / context encoders) with limited scope. We welcome researchers and users to report issues with the HuggingFace community discussion feature or email the authors.

However, since our model targets the task of editing images, it is strongly guided by the user input. To the best of our knowledge, with sanitized inputs, our model outputs sanitized outputs consistently.

bardofcodes
/

pattern_analogies