Duplicate from jukofyork/DeepSeek-V3-0324-CODER-DRAFT-0.6B-v1.0

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,131 @@
+---
+license: apache-2.0
+base_model:
+- Qwen/Qwen2.5-Coder-0.5B-Instruct
+datasets:
+- agentlans/common-crawl-sample
+- bigcode/the-stack-smol-xl
+tags:
+- draft
+- speculative-decoding
+language:
+- zho
+- eng
+- fra
+- spa
+- por
+- deu
+- ita
+- rus
+- jpn
+- kor
+- vie
+- tha
+- ara
+---
+
+![image.webp](https://cdn-uploads.huggingface.co/production/uploads/65995c45539c808e84c38bf1/KL97x9lVuhmIPXbbKgvyY.webp)
+
+A `0.6B` parameter draft (speculative decoding) model for use with [deepseek-ai/DeepSeek-V3-0324](https://huggingface.co/deepseek-ai/DeepSeek-V3-0324) and [deepseek-ai/DeepSeek-V3](https://huggingface.co/deepseek-ai/DeepSeek-V3).
+
+**NOTES**:
+
+- This version (unlike the previous [jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0)), was trained using only the [agentlans/common-crawl-sample](https://huggingface.co/datasets/agentlans/common-crawl-sample) and [bigcode/the-stack-smol-xl](https://huggingface.co/datasets/bigcode/the-stack-smol-xl) datasets.
+- This version (unlike the previous [jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0)), doesn't trim the heads down from 14 to 12.
+
+See [jukofyork/DeepSeek-V3-0324-CODER-DRAFT-0.6B-v1.0-GGUF](https://huggingface.co/jukofyork/DeepSeek-V3-0324-CODER-DRAFT-0.6B-v1.0-GGUF) for the models in GGUF format.
+
+---
+
+# How the model was created
+
+## 1. The initial model was created from [Qwen/Qwen2.5-Coder-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-0.5B-Instruct) using [transplant-vocab](https://github.com/jukofyork/transplant-vocab):
+
+```sh
+python ./transplant_vocab.py \                                  
+        Qwen2.5-Coder-0.5B-Instruct \
+        DeepSeek-V3-0324-BF16 \
+        DeepSeek-V3-0324-CODER-DRAFT-0.6B-UNTRAINED \
+        --override "<｜▁pad▁｜>" "<|endoftext|>" \
+        --override "<｜fim▁hole｜>" "<|fim_middle|>" \
+        --override "<｜fim▁begin｜>" "<|fim_prefix|>" \
+        --override "<｜fim▁end｜>" "<|fim_suffix|>" \
+        --override "<｜User｜>" "<|im_start|>user\\n" \
+        --override "<｜Assistant｜>" "<|im_start|>assistant\\n" \
+        --override "<|EOT|>" "<|endoftext|>" \
+        --override "<｜tool▁calls▁begin｜>" "<tool_call>" \
+        --override "<｜tool▁call▁begin｜>" "<tool_call>" \
+        --override "<｜tool▁outputs▁begin｜>" "<tool_call>" \
+        --override "<｜tool▁output▁begin｜>" "<tool_call>" \
+        --override "<｜tool▁calls▁end｜>" "</tool_call>" \
+        --override "<｜tool▁call▁end｜>" "</tool_call>" \
+        --override "<｜tool▁outputs▁end｜>" "</tool_call>" \
+        --override "<｜tool▁output▁end｜>" "</tool_call>" \
+        --override "<｜tool▁sep｜>" "</tool_call>"
+```
+
+## 2. The following datasets were merged to create a fine-tuning dataset of ~2.5B tokens:
+
+- [agentlans/common-crawl-sample](https://huggingface.co/datasets/agentlans/common-crawl-sample)
+- [bigcode/the-stack-smol-xl](https://huggingface.co/datasets/bigcode/the-stack-smol-xl)
+
+formatted just between `<｜end▁of▁sentence｜>` tags.
+
+## 3. The model was then trained using [qlora-pipe](https://github.com/tdrussell/qlora-pipe) for 1 epoch with a batch size of 120 and a sequence length of 32k (~4M tokens per step):
+
+```toml
+# Resume a prior run
+resume_from_checkpoint = false
+
+# Paths
+model = 'DeepSeek-V3-0324-CODER-DRAFT-0.6B-UNTRAINED'
+output_dir = 'DeepSeek-V3-0324-CODER-DRAFT-0.6B'
+
+# Optimization configuration
+full_fine_tune = true
+epochs = 1
+lr_scheduler = 'cosine'
+warmup_steps = 100
+
+# Performance settings
+pipeline_stages = 1
+logging_steps = 1
+eval_steps = 100
+save_steps = 100
+checkpoint_every_n_minutes = 60
+eval_before_first_step = true
+eval_after_last_step = true
+model_weight_dtype = 'bfloat16'
+keep_states = 3
+group_by_length = true
+activation_checkpointing = 'unsloth'
+
+# Dataset configuration
+dataset_combination_mode = 'concatenate'
+eval_gradient_accumulation_steps = 20
+
+[optimizer]
+type = 'adamw_kahan'
+lr = 5e-5
+beta1 = 0.9
+beta2 = 0.999
+weight_decay = 0.01
+
+[[datasets]]
+name = 'mixed_data'
+dataset_type = 'textfile'
+dataset_path = 'mixed_data/*.txt'
+sequence_len = 32768
+eval_size = 0.01
+```
+
+```json
+{
+    "train_micro_batch_size_per_gpu": 1,
+    "gradient_accumulation_steps": 20,
+    "gradient_clipping": 1.0,
+    "steps_per_print": 1
+}
+```
+
+I used six `RTX A6000` GPUs over three nodes and hence the `120` batch size (`6 x 20 gradient accumulation steps = 120`).

config.json

@@ -0,0 +1,34 @@
+{
+  "_name_or_path": "DeepSeek-V3-0324-CODER-DRAFT-0.6B",
+  "architectures": [
+    "Qwen2ForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "eos_token_id": 1,
+  "hidden_act": "silu",
+  "hidden_size": 896,
+  "initializer_range": 0.02,
+  "intermediate_size": 4864,
+  "max_position_embeddings": 163840,
+  "max_window_layers": 21,
+  "model_type": "qwen2",
+  "num_attention_heads": 14,
+  "num_hidden_layers": 24,
+  "num_key_value_heads": 2,
+  "pad_token_id": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": {
+    "factor": 4.0,
+    "original_max_position_embeddings": 32768,
+    "type": "yarn"
+  },
+  "rope_theta": 1000000.0,
+  "sliding_window": null,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.44.2",
+  "use_cache": true,
+  "use_sliding_window": false,
+  "vocab_size": 129280
+}

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": 1,
+  "pad_token_id": 1,
+  "transformers_version": "4.44.2"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3827b3badd63324ecd8b2986b9f6739f6f0d40b23a1f62e282db65b2666787c8
+size 1179168272

special_tokens_map.json

@@ -0,0 +1,23 @@
+{
+  "bos_token": {
+    "content": "<｜begin▁of▁sentence｜>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<｜end▁of▁sentence｜>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<｜end▁of▁sentence｜>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}