Safetensors
clip

Long-CLIP-KO: Knocking Out Typographic Attacks in Long-CLIP πŸ’ͺπŸ€–

Finally, a Long-CLIP without a 'text obsession'! πŸ€—

❀️ this CLIP? Donate if you can / want. TY!


πŸ‘‰ CLICK ME to expand example benchmark code βš‘πŸ’»
from datasets import load_dataset
from transformers import CLIPModel, CLIPProcessor
import torch
from PIL import Image
from tqdm import tqdm
import pandas as pd

device = "cuda" if torch.cuda.is_available() else "cpu"

# BLISS / SCAM Typographic Attack Dataset
# https://huggingface.co/datasets/BLISS-e-V/SCAM
ds = load_dataset("BLISS-e-V/SCAM", split="train")

# Benchmark pre-trained model against my fine-tune
model_variants = [
    ("OpenAI ", "zer0int/LongCLIP-L-Diffusers", "zer0int/LongCLIP-L-Diffusers"),
    ("KO-CLIP", "zer0int/LongCLIP-KO-LITE-TypoAttack-Attn-ViT-L-14", "zer0int/LongCLIP-KO-LITE-TypoAttack-Attn-ViT-L-14"),
]

models = {}
for name, model_path, processor_path in model_variants:
    model = CLIPModel.from_pretrained(model_path).to(device).float()
    processor = CLIPProcessor.from_pretrained(processor_path)
    models[name] = (model, processor)

for variant in ["NoSCAM", "SCAM", "SynthSCAM"]:
    print(f"\n=== Evaluating var.: {variant} ===")
    idxs = [i for i, v in enumerate(ds['id']) if v.startswith(variant)]
    if not idxs:
        print(f"  No samples for {variant}")
        continue
    subset = [ds[i] for i in idxs]

    for model_name, (model, processor) in models.items():
        results = []
        for entry in tqdm(subset, desc=f"{model_name}", ncols=30, bar_format="{l_bar}{bar}| {n_fmt}/{total_fmt} |"):
            img = entry['image']
            object_label = entry['object_label']
            attack_word = entry['attack_word']

            texts = [f"a photo of a {object_label}", f"a photo of a {attack_word}"]
            inputs = processor(
                text=texts,
                images=img,
                return_tensors="pt",
                padding=True
            )
            for k in inputs:
                if isinstance(inputs[k], torch.Tensor):
                    inputs[k] = inputs[k].to(device)

            with torch.no_grad():
                outputs = model(**inputs)
                image_features = outputs.image_embeds
                text_features = outputs.text_embeds

                logits = image_features @ text_features.T
                probs = logits.softmax(dim=-1).cpu().numpy().flatten()
                pred_idx = probs.argmax()
                pred_label = [object_label, attack_word][pred_idx]
                is_correct = (pred_label == object_label)

            results.append({
                "id": entry['id'],
                "object_label": object_label,
                "attack_word": attack_word,
                "pred_label": pred_label,
                "is_correct": is_correct,
                "type": entry['type'],
                "model": model_name
            })

        n_total = len(results)
        n_correct = sum(r['is_correct'] for r in results)
        acc = n_correct / n_total if n_total else float('nan')
        print(f"| > > > > Zero-shot accuracy for {variant}, {model_name}: {n_correct}/{n_total} = {acc:.4f}")

Non-misleading Attention Heatmaps:

image/png


πŸ”₯ Example Images generated with Flux.1-dev:

image/jpeg

image/jpeg


πŸ“Š Benchmark Results πŸš€

Typographic Attack Pre-Trained Fine-Tuned
Typographic Attack
RTA-100
zero-shot acc. 0.5980 0.7400πŸŽ–οΈ
BLISS / SCAM
NoSCAM acc.: 0.9819 0.9905
SCAM acc.: 0.5912 0.7849πŸŽ–οΈ
SynthSCAM acc.: 0.5723 0.7573πŸŽ–οΈ
LAION/CLIP_Benchmark
VoC-2007-multilabel
mAP: 0.8083 0.8553
MSCOCO retrieval
image retr recall@5: 0.2760 0.3403
text retr recall@: 0.3312 0.4745
xm3600 retrieval
image retr recall@5: 0.3714 0.4453
text retr recall@: 0.2972 0.4348
ImageNet-1k
zero-shot acc1: 0.3398 0.4652
zero-shot acc5: 0.5182 0.6810
mAP: 0.3381 0.4633
MISC
ImageNet-1k
linear probe Top-1: 66.76% 71.54%
linear probe Top-5: 92.02% 93.76%
MVT ImageNet/ObjectNet
zero-shot acc. 0.8113 0.9026πŸŽ–οΈ
Flickr8k
Modality Gap: ↓ 1.0672 0.8119πŸŽ–οΈ
JSD: ↓ 0.3847 0.1759
Wasserstein Distance: ↓ 0.5755 0.3727
Image-Text Cos Sim (mean): ↑ 0.2666 0.3286
Image-Text Cos Sim, (std): 0.0191 0.0702
Text-Text Cos Sim (mean): 0.8421 0.7013
Text-Text Cos Sim (std): 0.0707 0.1486
Downloads last month
1,857
Safetensors
Model size
428M params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for zer0int/LongCLIP-KO-LITE-TypoAttack-Attn-ViT-L-14

Finetuned
(5)
this model

Datasets used to train zer0int/LongCLIP-KO-LITE-TypoAttack-Attn-ViT-L-14