README.md

metadata

title: unsloth/DeepSeek-R1-Distill-Qwen-14B-unsloth-bnb-4bit (Research Training)
emoji: 🧪
colorFrom: blue
colorTo: indigo
sdk: gradio
sdk_version: 5.17.0
app_file: app.py
pinned: false
license: mit

Model Fine-Tuning Project

Overview

• Goal: Fine-tune unsloth/DeepSeek-R1-Distill-Qwen-14B-unsloth-bnb-4bit using pre-tokenized JSONL dataset
• Model: unsloth/DeepSeek-R1-Distill-Qwen-14B-unsloth-bnb-4bit
  • Important: Already 4-bit quantized - do not quantize further
• Dataset: phi4-cognitive-dataset

⚠️ RESEARCH TRAINING PHASE ONLY: This space is being used for training purposes and does not provide interactive model outputs.

Dataset Specs

• Entries under 2048 tokens
• Fields: prompt_number, article_id, conversations
• Process in ascending prompt_number order
• Pre-tokenized dataset - no additional tokenization needed

Hardware

• GPU: 1x L40S (48GB VRAM)
• RAM: 62GB
• CPU: 8 cores

Environment Variables (.env)

• HF_TOKEN: Hugging Face API token
• HF_USERNAME: Hugging Face username
• HF_SPACE_NAME: Target space name

Files

1. app.py

• Training status dashboard
• No interactive model demo (research phase only)

2. transformers_config.json

• Configuration for Hugging Face Transformers
• Contains: model parameters, hardware settings, optimizer details
• Specifies pre-tokenized dataset handling

3. run_cloud_training.py

• Loads pre-tokenized dataset, sorts by prompt_number, initiates training

1. Load and sort JSONL by prompt_number
2. Use pre-tokenized input_ids directly (no tokenization)
3. Initialize with parameters from config
4. Execute training with metrics, checkpoints, error handling

• Uses Hugging Face's Trainer API with custom pre-tokenized data collator

4. requirements.txt

• Python dependencies: transformers, datasets, torch, etc.
• Contains unsloth for optimized training

5. upload_to_space.py

• Update model and space directly using HF API

Implementation Notes

Best Practices

• Dataset is pre-tokenized and sorted by prompt_number
• Settings stored in config file, avoiding hardcoding
• Hardware-optimized training parameters
• Gradient checkpointing and mixed precision training
• Complete logging for monitoring progress

Model Repository

This space hosts a fine-tuned version of the unsloth/DeepSeek-R1-Distill-Qwen-14B-unsloth-bnb-4bit model.

Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference