Long-CLIP-KO: Knocking Out Typographic Attacks in Long-CLIP πͺπ€
Finally, a Long-CLIP without a 'text obsession'! π€
β€οΈ this CLIP? Donate if you can / want. TY!
- π Read the paper (PDF) here.
- πΌοΈ Download The Text Encoder for generative AI
- π€ Wanna fine-tune yourself? Get the code on my GitHub.
π CLICK ME to expand example benchmark code β‘π»
from datasets import load_dataset
from transformers import CLIPModel, CLIPProcessor
import torch
from PIL import Image
from tqdm import tqdm
import pandas as pd
device = "cuda" if torch.cuda.is_available() else "cpu"
# BLISS / SCAM Typographic Attack Dataset
# https://huggingface.co/datasets/BLISS-e-V/SCAM
ds = load_dataset("BLISS-e-V/SCAM", split="train")
# Benchmark pre-trained model against my fine-tune
model_variants = [
("OpenAI ", "zer0int/LongCLIP-L-Diffusers", "zer0int/LongCLIP-L-Diffusers"),
("KO-CLIP", "zer0int/LongCLIP-KO-LITE-TypoAttack-Attn-ViT-L-14", "zer0int/LongCLIP-KO-LITE-TypoAttack-Attn-ViT-L-14"),
]
models = {}
for name, model_path, processor_path in model_variants:
model = CLIPModel.from_pretrained(model_path).to(device).float()
processor = CLIPProcessor.from_pretrained(processor_path)
models[name] = (model, processor)
for variant in ["NoSCAM", "SCAM", "SynthSCAM"]:
print(f"\n=== Evaluating var.: {variant} ===")
idxs = [i for i, v in enumerate(ds['id']) if v.startswith(variant)]
if not idxs:
print(f" No samples for {variant}")
continue
subset = [ds[i] for i in idxs]
for model_name, (model, processor) in models.items():
results = []
for entry in tqdm(subset, desc=f"{model_name}", ncols=30, bar_format="{l_bar}{bar}| {n_fmt}/{total_fmt} |"):
img = entry['image']
object_label = entry['object_label']
attack_word = entry['attack_word']
texts = [f"a photo of a {object_label}", f"a photo of a {attack_word}"]
inputs = processor(
text=texts,
images=img,
return_tensors="pt",
padding=True
)
for k in inputs:
if isinstance(inputs[k], torch.Tensor):
inputs[k] = inputs[k].to(device)
with torch.no_grad():
outputs = model(**inputs)
image_features = outputs.image_embeds
text_features = outputs.text_embeds
logits = image_features @ text_features.T
probs = logits.softmax(dim=-1).cpu().numpy().flatten()
pred_idx = probs.argmax()
pred_label = [object_label, attack_word][pred_idx]
is_correct = (pred_label == object_label)
results.append({
"id": entry['id'],
"object_label": object_label,
"attack_word": attack_word,
"pred_label": pred_label,
"is_correct": is_correct,
"type": entry['type'],
"model": model_name
})
n_total = len(results)
n_correct = sum(r['is_correct'] for r in results)
acc = n_correct / n_total if n_total else float('nan')
print(f"| > > > > Zero-shot accuracy for {variant}, {model_name}: {n_correct}/{n_total} = {acc:.4f}")
Non-misleading Attention Heatmaps:
π₯ Example Images generated with Flux.1-dev:
π Benchmark Results π
Typographic Attack | Pre-Trained | Fine-Tuned |
---|---|---|
Typographic Attack | ||
RTA-100 | ||
zero-shot acc. | 0.5980 | 0.7400ποΈ |
BLISS / SCAM | ||
NoSCAM acc.: | 0.9819 | 0.9905 |
SCAM acc.: | 0.5912 | 0.7849ποΈ |
SynthSCAM acc.: | 0.5723 | 0.7573ποΈ |
LAION/CLIP_Benchmark | ||
VoC-2007-multilabel | ||
mAP: | 0.8083 | 0.8553 |
MSCOCO retrieval | ||
image retr recall@5: | 0.2760 | 0.3403 |
text retr recall@: | 0.3312 | 0.4745 |
xm3600 retrieval | ||
image retr recall@5: | 0.3714 | 0.4453 |
text retr recall@: | 0.2972 | 0.4348 |
ImageNet-1k | ||
zero-shot acc1: | 0.3398 | 0.4652 |
zero-shot acc5: | 0.5182 | 0.6810 |
mAP: | 0.3381 | 0.4633 |
MISC | ||
ImageNet-1k | ||
linear probe Top-1: | 66.76% | 71.54% |
linear probe Top-5: | 92.02% | 93.76% |
MVT ImageNet/ObjectNet | ||
zero-shot acc. | 0.8113 | 0.9026ποΈ |
Flickr8k | ||
Modality Gap: β | 1.0672 | 0.8119ποΈ |
JSD: β | 0.3847 | 0.1759 |
Wasserstein Distance: β | 0.5755 | 0.3727 |
Image-Text Cos Sim (mean): β | 0.2666 | 0.3286 |
Image-Text Cos Sim, (std): | 0.0191 | 0.0702 |
Text-Text Cos Sim (mean): | 0.8421 | 0.7013 |
Text-Text Cos Sim (std): | 0.0707 | 0.1486 |
- Downloads last month
- 1,857
Inference Providers
NEW
This model isn't deployed by any Inference Provider.
π
Ask for provider support
Model tree for zer0int/LongCLIP-KO-LITE-TypoAttack-Attn-ViT-L-14
Base model
BeichenZhang/LongCLIP-L