Model Card for LRC-1.5B-SFT

LRC-1.5B-SFT is a Small Language Model (SLM) with approximately 1.5 billion parameters. It is the Supervised Fine-Tuned (SFT) version of LRC-1.5B-Base. The LRC method is an efficient knowledge distillation technique used to construct the base model from its teacher, Llama-3.2-3B-Instruct, using 10 billion tokens. This SFT version was then further fine-tuned on an instruction-following dataset ultrachat_200k.

The LRC approach trains a set of low-rank projection matrices that enable soft pruning by compressing teacher weights and an "activation clone" mechanism that aligns student activations (including FFN signals) with those of the teacher. The base model, LRC-1.5B-Base, was trained on 10 billion tokens.

Uses

Direct Use

LRC-1.5B-SFT is an instruction-tuned model and is intended for tasks requiring instruction following, question answering, and general chat capabilities.

Biases, Risks, and Limitations

• SFT Dataset Limitations: Our SFT model (LRC-1.5B-SFT) was fine-tuned solely on the UltraChat dataset (using 0.2B tokens, as per Table 8). While UltraChat enhances general instruction-following, it may not be sufficiently diverse or targeted to instill robust safety alignment or complex instruction adherence compared to models trained with more extensive or specialized alignment techniques (e.g., RLHF, or SFT on broader safety/instruction datasets). Consequently, the model might exhibit deficiencies in safety and its ability to follow highly complex or nuanced instructions.
• Inherited Biases: The model may reflect biases present in its pre-training data (Fineweb-Edu, OpenHermes 2.5) and the teacher model (Llama-3.2-3B-Instruct).
• Hallucination: Like all LLMs, LRC-1.5B-SFT can generate factually incorrect or nonsensical information (hallucinations).
• Limited Scope of Evaluation: The paper's primary evaluation focuses on pre-training efficiency and general downstream tasks. Extensive testing on safety benchmarks or complex reasoning tasks beyond the reported MMLU, ARC, etc., was not detailed.

How to Get Started with the Model

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained('JitaiHao/LRC-1.5B')
model = AutoModelForCausalLM.from_pretrained('JitaiHao/LRC-1.5B')

# Prepare a multi-turn chat history
messages = [
    {"role": "user", "content": "Hello, who are you?"},
    {"role": "assistant", "content": "Hello, I am an AI assistant."}
]

# Use apply_chat_template to create a prompt for the model
input_text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,        # Only generate the string prompt, do not tokenize yet
    add_generation_prompt=True  # Add a generation prompt for the assistant
)

print(input_text)  # View the generated prompt string

# If you want to generate a response with the model
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Details

Training Data

• Pre-training (for LRC-1.5B-Base): 10 billion tokens from the "Mixed-1.1" dataset (10B Fineweb-Edu, 450M OpenHermes 2.5).
• Supervised Fine-Tuning (SFT): 0.2 billion tokens from the UltraChat dataset.

Training Procedure

1. Pre-training (LRC-1.5B-Base): Trained using the Low-Rank Clone (LRC) method as detailed in the LRC-1.5B-Base model card.
   • Teacher Model: Llama-3.2-3B-Instruct
2. Supervised Fine-Tuning (SFT):
   • Dataset: UltraChat (0.2B tokens)
   • Learning Rate (SFT): 1.0 x 10⁻⁵ (as per Table 8)

Evaluation

Zero-Shot Comparison with other publicly available SFT models under 2B parameters (from Table 1 of the paper):

Model	# Tokens	ARC-E	ARC-C	LogiQA	CSQA	PIQA	WinoG	BoolQ	SciQ	MMLU	Avg.
InternLM2-1.8B	2T	71.04	42.06	28.42	70.11	74.27	63.77	75.50	94.50	43.75	62.60
LRC-1.7B-SFT	20B	74.62	44.20	30.88	70.19	73.07	63.30	79.82	93.80	54.93	64.98
Qwen3-1.7B	36T	72.47	43.00	28.42	64.78	72.20	61.48	77.65	93.10	55.44	63.17
SmolLM2-1.7B	11T	69.11	43.52	28.88	51.19	76.01	68.98	68.47	89.80	48.50	60.50
LRC-1.5B-SFT	10B	74.75	44.97	30.72	65.77	73.07	62.25	75.78	94.60	49.42	63.48
MiniCPM-1.2B	1T	70.16	39.68	30.88	64.29	74.65	60.77	67.58	91.50	44.23	60.42

Performance on safety and instruction-following tasks (from Table 14, LRC-1.5B is the SFT version):

Benchmark	Metric	Score (LRC-1.5B-SFT)	Score (LRC-1.5B-Base)
ToxiGen	Accuracy Norm	43.19	43.19
IFeval	Instance-Level Loose Acc	23.74	24.58
TruthfulQA	MC2	46.98	47.97

The limited gains on ToxiGen and the slight decrease on IFeval and TruthfulQA post-SFT (compared to its base model) may suggest that the UltraChat SFT data did not specifically target or enhance these particular capabilities beyond the pre-training stage, reinforcing the point made in "Biases, Risks, and Limitations."

Technical Specifications

Model Architecture and Objective

• Architecture: Transformer-based decoder-only model, adhering to the Llama architecture.
  • Number of Layers: 28
  • Hidden Size: 1,536
  • FFN Intermediate Size: 8,192
  • Attention Q Heads: 24
  • Attention KV Heads: 8
  • Head Dimension: 128
  • Vocabulary Size: 128,256
  • Word Embeddings: Tied