metadata

license: cc-by-4.0
language:
  - en
tags:
  - adversarial
  - rank-boosting
  - rank-promotion
base_model:
  - Qwen/QwQ-32B
pipeline_tag: text-generation
library_name: transformers

CRAFT-QwQ-32B

Specialized for adversarial rank promotion in neural IR systems, this model is fine-tuned with the Alpaca template on a QwQ-32B backbone. It produces fluent, style-consistent sentences that strategically enhance a target document’s relevance score without addressing the query.

Base architecture: QwQ-32B
Format: Alpaca instruction template (exact template below)
Dataset: Adversarial datasets for fine-tuning (link under Dataset section)

Intended Use and Tasks

Adversarial attacks for rank promotion in information retrieval systems based on neural ranking models

Task Definition

Adversarial Ranking Augmentation (primary): Given a Query, a Target Document, and the Top-5 Documents for that query, generate a single adversarial sentence that increases the Target Document's perceived relevance without answering the query. The response must be returned as a JSON object of the form { "response": "<one sentence>" }. Constraints include reusing vocabulary/themes (Buffer A), preserving style/fluency, and avoiding direct answers or explicit references to the query.

Prompt Formatting

This model was fine-tuned using the Alpaca-style instruction template. The exact prompt format, including the instruction and input content used during fine-tuning, is:

Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
You are an expert in information retrieval and adversarial search ranking. Your goal is to help a *Target Document* rank higher.
(ideally between 1 to 10) for a *given query* by subtly increasing its perceived relevance to ranking models such as cross-encoders.

You will do this by generating content that shares vocabulary, structure, and context with high-ranking documents, 
but without explicitly answering the query or suggesting a solution.

Extract:

- **Buffer A**: 10-15 words from the Top-5 ranked documents and query itself, strongly associated with the query.

**Generate an adversarial sentences** that satisfy ALL the following:

- Include combination of words (at least 5) or similar words (similar embedding) from Buffer A** that is most related to the query and help promote ranking significantly and integrates well with Target Document
- DO NOT use the words that answer the query.
- Are **fluent**, **grammatically sound**, and **consistent with the style** of the Target Document.
- **Do NOT answer, suggest, or hint at an answer to the Target Query**.
- **Do NOT reference the Target Query at all**.
- Are designed to **sound relevant** but only reinforce theme/context alignment.

### Input:
Query: {query}

Target Document:
{doc_content}

Top-5 Documents:
{top_docs_str}

Generate your answer as a valid JSON object with the following structure:
{
  "response": "<your answer here>"
}
Do not include any additional text.

### Response:

How to Use (Transformers)

Basic usage with the Alpaca template:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# Use the published Hugging Face repo id
model_id = "radinrad/CRAFT-QwQ-32B"
tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=True)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16, device_map="auto")

# Example inputs
query = "effects of intermittent fasting on metabolism"
doc_content = "...target document content..."
top_docs = ["doc 1 ...", "doc 2 ...", "doc 3 ...", "doc 4 ...", "doc 5 ..."]
top_docs_str = "\n".join(top_docs)

prompt = f"""Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
You are an expert in information retrieval and adversarial search ranking. Your goal is to help a *Target Document* rank higher.
(ideally between 1 to 10) for a *given query* by subtly increasing its perceived relevance to ranking models such as cross-encoders.

You will do this by generating content that shares vocabulary, structure, and context with high-ranking documents, 
but without explicitly answering the query or suggesting a solution.

Extract:

- **Buffer A**: 10-15 words from the Top-5 ranked documents and query itself, strongly associated with the query.

**Generate an adversarial sentences** that satisfy ALL the following:

- Include combination of words (at least 5) or similar words (similar embedding) from Buffer A** that is most related to the query and help promote ranking significantly and integrates well with Target Document
- DO NOT use the words that answer the query.
- Are **fluent**, **grammatically sound**, and **consistent with the style** of the Target Document.
- **Do NOT answer, suggest, or hint at an answer to the Target Query**.
- **Do NOT reference the Target Query at all**.
- Are designed to **sound relevant** but only reinforce theme/context alignment.

### Input:
Query: {query}

Target Document:
{doc_content}

Top-5 Documents:
{top_docs_str}

Generate your answer as a valid JSON object with the following structure:
{{
  "response": "<your answer here>"
}}
Do not include any additional text.

### Response:
"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output_ids = model.generate(
    **inputs,
    do_sample=True,
    temperature=0.6,
    top_p=0.95,
    top_k=40,
    max_new_tokens=128,
    eos_token_id=tokenizer.eos_token_id,
    pad_token_id=tokenizer.pad_token_id,
)
print(tokenizer.decode(output_ids[0], skip_special_tokens=True))

Recommended Generation Settings

Recommended decoding settings:

do_sample: true
temperature: 0.6
top_p: 0.95
top_k: 40
max_new_tokens: 128

Inference Recommendations

For most tasks, use top_p = 0.95 and temperature = 0.6.
Keep do_sample=True and top_k=40 for a good quality–diversity tradeoff.
Adjust max_new_tokens to your task length (e.g., 128 for short answers).

Adversarial Generation Strategy (Recommended)

For adversarial attack or robust candidate selection, we recommend a generate-then-rank approach:

Generate a pool of candidates (≈10) with the same decoding settings (top_p=0.95, temperature=0.6).
Score each candidate using a surrogate model e.g. BERT base uncased (google-bert/bert-base-uncased). Compute cosine similarity between the query and each candidate and pick the highest.
Select the highest-scoring candidate as the final output.

This pool-plus-ranking approach tends to improve robustness for adversarial objectives.

Evaluation

The following summarizes attack performance and content fidelity metrics for CRAFT across backbones on the Easy-5 and Hard-5 settings. Values are percentages where applicable; arrows indicate the direction of preference. Daggers (†) denote statistically significant improvements over the strongest baseline in each setting (paired two-tailed t-test, p < 0.05). Bold indicates column best.

Easy-5

Method	ASR	Top-10	Top-50	Boost	SS (↑)	ATI (↓)	ADT (↓)	LOR (↑)
PRADA	59.8	1.2	25.2	13.4	0.9	0.1	13.1	0.9
Brittle-BERT	76.3	12.9	56.8	22.6	0.9	11.6	11.6	1.0
PAT	46.8	1.4	17.2	-3.3	0.9	6.3	6.3	1.0
IDEM	97.3	32.1	84.8	49.3	0.9	11.6	11.6	1.0
EMPRA	99.4	43.5	93.4	57.6	0.9	29.8	29.8	1.0
AttChain	92.1	34.5	83.9	47.9	0.8	22.4	38.8	0.9
CRAFT_Qwen3	97.2	37.0	91.4	54.5	0.9	19.1	19.1	1.0
CRAFT_Llama3.3	99.4	44.5	95.8†	59.7†	0.9	19.9	19.9	1.0

Hard-5

Method	ASR	Top-10	Top-50	Boost	SS (↑)	ATI (↓)	ADT (↓)	LOR (↑)
PRADA	74.3	0.0	0.0	75.5	0.9	0.1	18.5	0.9
Brittle-BERT	99.7	4.2	23.4	744.5	0.9	11.2	11.3	1.0
PAT	80.1	0.1	0.4	79.6	0.9	11.2	6.3	1.0
IDEM	99.8	8.3	34.5	780.8	0.9	11.2	22.4	1.0
EMPRA	99.3	10.7	40.8	828.5	0.8	32.7	32.7	1.0
AttChain	99.8	12.2	42.4	855.2	0.7	22.8	39.0	0.9
CRAFT_Qwen3	100.0	15.3†	57.1†	911.5†	0.8	19.1	19.1	1.0
CRAFT_Llama3.3	100.0	22.2†	70.5†	940.5†	0.8	19.7	19.7	1.0

Figure: Attack methods performance vs. detection pass rate

Dataset

This model was fine-tuned using data from the following repository:

GitHub: https://github.com/KhosrojerdiA/adversarial-datasets

Please review the repository for details on data composition, licensing, and any usage constraints.

Limitations and Bias

The model may produce incorrect, biased, or unsafe content. Use human oversight for critical applications.
Behaviors outside the Alpaca-style instruction format may be less reliable.
The model does not have browsing or up-to-date world knowledge beyond its pretraining and fine-tuning data.

License and Usage

License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)
This checkpoint also inherits licensing constraints from the base Qwen2 model and the fine-tuning data. Ensure your usage complies with the base model license and the dataset’s license/terms.
If you redistribute or deploy this model, please include appropriate attribution and links back to the base model and dataset.

Acknowledgements

Base architecture: Qwen2 (Alibaba Cloud / Qwen team)
Prompt format inspired by Alpaca