[INFO|2025-04-22 23:31:16] tokenization_utils_base.py:2060 >> loading file tokenizer.model from cache at None

[INFO|2025-04-22 23:31:16] tokenization_utils_base.py:2060 >> loading file tokenizer.json from cache at /root/.cache/huggingface/hub/models--deepseek-ai--deepseek-llm-7b-base/snapshots/7683fea62db869066ddaff6a41d032262c490d4f/tokenizer.json

[INFO|2025-04-22 23:31:16] tokenization_utils_base.py:2060 >> loading file added_tokens.json from cache at None

[INFO|2025-04-22 23:31:16] tokenization_utils_base.py:2060 >> loading file special_tokens_map.json from cache at None

[INFO|2025-04-22 23:31:16] tokenization_utils_base.py:2060 >> loading file tokenizer_config.json from cache at /root/.cache/huggingface/hub/models--deepseek-ai--deepseek-llm-7b-base/snapshots/7683fea62db869066ddaff6a41d032262c490d4f/tokenizer_config.json

[INFO|2025-04-22 23:31:16] tokenization_utils_base.py:2060 >> loading file chat_template.jinja from cache at None

[INFO|2025-04-22 23:31:17] tokenization_utils_base.py:2323 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.

[INFO|2025-04-22 23:31:17] configuration_utils.py:693 >> loading configuration file config.json from cache at /root/.cache/huggingface/hub/models--deepseek-ai--deepseek-llm-7b-base/snapshots/7683fea62db869066ddaff6a41d032262c490d4f/config.json

[INFO|2025-04-22 23:31:17] configuration_utils.py:765 >> Model config LlamaConfig {
  "architectures": [
    "LlamaForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "bos_token_id": 1,
  "eos_token_id": 2,
  "head_dim": 128,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 11008,
  "max_position_embeddings": 4096,
  "mlp_bias": false,
  "model_type": "llama",
  "num_attention_heads": 32,
  "num_hidden_layers": 30,
  "num_key_value_heads": 32,
  "pretraining_tp": 1,
  "rms_norm_eps": 1e-06,
  "rope_scaling": null,
  "rope_theta": 10000.0,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.51.3",
  "use_cache": true,
  "vocab_size": 102400
}


[INFO|2025-04-22 23:31:17] tokenization_utils_base.py:2060 >> loading file tokenizer.model from cache at None

[INFO|2025-04-22 23:31:17] tokenization_utils_base.py:2060 >> loading file tokenizer.json from cache at /root/.cache/huggingface/hub/models--deepseek-ai--deepseek-llm-7b-base/snapshots/7683fea62db869066ddaff6a41d032262c490d4f/tokenizer.json

[INFO|2025-04-22 23:31:17] tokenization_utils_base.py:2060 >> loading file added_tokens.json from cache at None

[INFO|2025-04-22 23:31:17] tokenization_utils_base.py:2060 >> loading file special_tokens_map.json from cache at None

[INFO|2025-04-22 23:31:17] tokenization_utils_base.py:2060 >> loading file tokenizer_config.json from cache at /root/.cache/huggingface/hub/models--deepseek-ai--deepseek-llm-7b-base/snapshots/7683fea62db869066ddaff6a41d032262c490d4f/tokenizer_config.json

[INFO|2025-04-22 23:31:17] tokenization_utils_base.py:2060 >> loading file chat_template.jinja from cache at None

[INFO|2025-04-22 23:31:17] tokenization_utils_base.py:2323 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.

[INFO|2025-04-22 23:31:18] logging.py:143 >> Loading dataset alpaca_en_demo.json...

[INFO|2025-04-22 23:31:28] configuration_utils.py:693 >> loading configuration file config.json from cache at /root/.cache/huggingface/hub/models--deepseek-ai--deepseek-llm-7b-base/snapshots/7683fea62db869066ddaff6a41d032262c490d4f/config.json

[INFO|2025-04-22 23:31:28] configuration_utils.py:765 >> Model config LlamaConfig {
  "architectures": [
    "LlamaForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "bos_token_id": 1,
  "eos_token_id": 2,
  "head_dim": 128,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 11008,
  "max_position_embeddings": 4096,
  "mlp_bias": false,
  "model_type": "llama",
  "num_attention_heads": 32,
  "num_hidden_layers": 30,
  "num_key_value_heads": 32,
  "pretraining_tp": 1,
  "rms_norm_eps": 1e-06,
  "rope_scaling": null,
  "rope_theta": 10000.0,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.51.3",
  "use_cache": true,
  "vocab_size": 102400
}


[INFO|2025-04-22 23:31:28] logging.py:143 >> Quantizing model to 4 bit with bitsandbytes.

[INFO|2025-04-22 23:31:28] logging.py:143 >> KV cache is disabled during training.

[INFO|2025-04-22 23:31:29] modeling_utils.py:1124 >> loading weights file pytorch_model.bin from cache at /root/.cache/huggingface/hub/models--deepseek-ai--deepseek-llm-7b-base/snapshots/7683fea62db869066ddaff6a41d032262c490d4f/pytorch_model.bin.index.json

[INFO|2025-04-22 23:31:30] safetensors_conversion.py:61 >> Attempting to create safetensors variant

[INFO|2025-04-22 23:31:30] safetensors_conversion.py:74 >> Safetensors PR exists

[INFO|2025-04-22 23:33:11] modeling_utils.py:2167 >> Instantiating LlamaForCausalLM model under default dtype torch.bfloat16.

[INFO|2025-04-22 23:33:11] configuration_utils.py:1142 >> Generate config GenerationConfig {
  "bos_token_id": 1,
  "eos_token_id": 2,
  "use_cache": false
}


[INFO|2025-04-22 23:34:25] modeling_utils.py:4930 >> All model checkpoint weights were used when initializing LlamaForCausalLM.


[INFO|2025-04-22 23:34:25] modeling_utils.py:4938 >> All the weights of LlamaForCausalLM were initialized from the model checkpoint at deepseek-ai/deepseek-llm-7b-base.
If your task is similar to the task the model of the checkpoint was trained on, you can already use LlamaForCausalLM for predictions without further training.

[INFO|2025-04-22 23:34:25] configuration_utils.py:1097 >> loading configuration file generation_config.json from cache at /root/.cache/huggingface/hub/models--deepseek-ai--deepseek-llm-7b-base/snapshots/7683fea62db869066ddaff6a41d032262c490d4f/generation_config.json

[INFO|2025-04-22 23:34:25] configuration_utils.py:1142 >> Generate config GenerationConfig {
  "bos_token_id": 100000,
  "eos_token_id": 100001
}


[INFO|2025-04-22 23:34:25] logging.py:143 >> Gradient checkpointing enabled.

[INFO|2025-04-22 23:34:25] logging.py:143 >> Using torch SDPA for faster training and inference.

[INFO|2025-04-22 23:34:25] logging.py:143 >> Upcasting trainable params to float32.

[INFO|2025-04-22 23:34:25] logging.py:143 >> Fine-tuning method: LoRA

[INFO|2025-04-22 23:34:25] logging.py:143 >> Found linear modules: o_proj,q_proj,gate_proj,k_proj,down_proj,v_proj,up_proj

[INFO|2025-04-22 23:34:26] logging.py:143 >> trainable params: 18,739,200 || all params: 6,929,104,896 || trainable%: 0.2704

[INFO|2025-04-22 23:34:26] trainer.py:748 >> Using auto half precision backend

[INFO|2025-04-22 23:34:26] trainer.py:2414 >> ***** Running training *****

[INFO|2025-04-22 23:34:26] trainer.py:2415 >>   Num examples = 1,307

[INFO|2025-04-22 23:34:26] trainer.py:2416 >>   Num Epochs = 3

[INFO|2025-04-22 23:34:26] trainer.py:2417 >>   Instantaneous batch size per device = 2

[INFO|2025-04-22 23:34:26] trainer.py:2420 >>   Total train batch size (w. parallel, distributed & accumulation) = 16

[INFO|2025-04-22 23:34:26] trainer.py:2421 >>   Gradient Accumulation steps = 8

[INFO|2025-04-22 23:34:26] trainer.py:2422 >>   Total optimization steps = 243

[INFO|2025-04-22 23:34:26] trainer.py:2423 >>   Number of trainable parameters = 18,739,200

[INFO|2025-04-22 23:36:16] logging.py:143 >> {'loss': 2.3883, 'learning_rate': 1.9987e-04, 'epoch': 0.06, 'throughput': 36.53}

[INFO|2025-04-22 23:38:05] logging.py:143 >> {'loss': 1.4752, 'learning_rate': 1.9932e-04, 'epoch': 0.12, 'throughput': 36.29}

[INFO|2025-04-22 23:39:57] logging.py:143 >> {'loss': 1.1174, 'learning_rate': 1.9837e-04, 'epoch': 0.18, 'throughput': 36.12}

[INFO|2025-04-22 23:41:45] logging.py:143 >> {'loss': 0.7815, 'learning_rate': 1.9700e-04, 'epoch': 0.24, 'throughput': 36.03}

[INFO|2025-04-22 23:43:35] logging.py:143 >> {'loss': 0.6320, 'learning_rate': 1.9522e-04, 'epoch': 0.31, 'throughput': 36.22}

[INFO|2025-04-22 23:45:25] logging.py:143 >> {'loss': 0.4603, 'learning_rate': 1.9305e-04, 'epoch': 0.37, 'throughput': 36.20}

[INFO|2025-04-22 23:47:17] logging.py:143 >> {'loss': 0.4761, 'learning_rate': 1.9049e-04, 'epoch': 0.43, 'throughput': 36.25}

[INFO|2025-04-22 23:49:06] logging.py:143 >> {'loss': 0.4525, 'learning_rate': 1.8756e-04, 'epoch': 0.49, 'throughput': 36.22}

[INFO|2025-04-22 23:50:55] logging.py:143 >> {'loss': 0.4803, 'learning_rate': 1.8425e-04, 'epoch': 0.55, 'throughput': 36.12}

[INFO|2025-04-22 23:52:45] logging.py:143 >> {'loss': 0.3933, 'learning_rate': 1.8060e-04, 'epoch': 0.61, 'throughput': 36.11}

[INFO|2025-04-22 23:54:39] logging.py:143 >> {'loss': 0.3919, 'learning_rate': 1.7660e-04, 'epoch': 0.67, 'throughput': 36.22}

[INFO|2025-04-22 23:56:33] logging.py:143 >> {'loss': 0.3804, 'learning_rate': 1.7229e-04, 'epoch': 0.73, 'throughput': 36.31}

[INFO|2025-04-22 23:58:22] logging.py:143 >> {'loss': 0.3576, 'learning_rate': 1.6768e-04, 'epoch': 0.80, 'throughput': 36.28}

[INFO|2025-04-23 00:00:11] logging.py:143 >> {'loss': 0.3343, 'learning_rate': 1.6278e-04, 'epoch': 0.86, 'throughput': 36.32}

[INFO|2025-04-23 00:01:59] logging.py:143 >> {'loss': 0.4526, 'learning_rate': 1.5762e-04, 'epoch': 0.92, 'throughput': 36.27}

[INFO|2025-04-23 00:03:48] logging.py:143 >> {'loss': 0.4048, 'learning_rate': 1.5222e-04, 'epoch': 0.98, 'throughput': 36.24}

[INFO|2025-04-23 00:05:46] logging.py:143 >> {'loss': 0.3599, 'learning_rate': 1.4660e-04, 'epoch': 1.05, 'throughput': 36.26}

[INFO|2025-04-23 00:07:38] logging.py:143 >> {'loss': 0.2197, 'learning_rate': 1.4079e-04, 'epoch': 1.11, 'throughput': 36.31}

[INFO|2025-04-23 00:09:28] logging.py:143 >> {'loss': 0.1816, 'learning_rate': 1.3481e-04, 'epoch': 1.17, 'throughput': 36.33}

[INFO|2025-04-23 00:11:19] logging.py:143 >> {'loss': 0.1839, 'learning_rate': 1.2868e-04, 'epoch': 1.23, 'throughput': 36.33}

[INFO|2025-04-23 00:11:19] trainer.py:3984 >> Saving model checkpoint to saves/DeepSeek-LLM-7B-Base/lora/train_2025-04-22-23-24-06/checkpoint-100

[INFO|2025-04-23 00:11:19] configuration_utils.py:693 >> loading configuration file config.json from cache at /root/.cache/huggingface/hub/models--deepseek-ai--deepseek-llm-7b-base/snapshots/7683fea62db869066ddaff6a41d032262c490d4f/config.json

[INFO|2025-04-23 00:11:19] configuration_utils.py:765 >> Model config LlamaConfig {
  "architectures": [
    "LlamaForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "bos_token_id": 1,
  "eos_token_id": 2,
  "head_dim": 128,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 11008,
  "max_position_embeddings": 4096,
  "mlp_bias": false,
  "model_type": "llama",
  "num_attention_heads": 32,
  "num_hidden_layers": 30,
  "num_key_value_heads": 32,
  "pretraining_tp": 1,
  "rms_norm_eps": 1e-06,
  "rope_scaling": null,
  "rope_theta": 10000.0,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.51.3",
  "use_cache": true,
  "vocab_size": 102400
}


[INFO|2025-04-23 00:11:19] tokenization_utils_base.py:2510 >> tokenizer config file saved in saves/DeepSeek-LLM-7B-Base/lora/train_2025-04-22-23-24-06/checkpoint-100/tokenizer_config.json

[INFO|2025-04-23 00:11:19] tokenization_utils_base.py:2519 >> Special tokens file saved in saves/DeepSeek-LLM-7B-Base/lora/train_2025-04-22-23-24-06/checkpoint-100/special_tokens_map.json

[INFO|2025-04-23 00:13:14] logging.py:143 >> {'loss': 0.2100, 'learning_rate': 1.2243e-04, 'epoch': 1.29, 'throughput': 36.25}

[INFO|2025-04-23 00:15:04] logging.py:143 >> {'loss': 0.2568, 'learning_rate': 1.1609e-04, 'epoch': 1.35, 'throughput': 36.24}

[INFO|2025-04-23 00:16:56] logging.py:143 >> {'loss': 0.2049, 'learning_rate': 1.0968e-04, 'epoch': 1.42, 'throughput': 36.27}

[INFO|2025-04-23 00:18:44] logging.py:143 >> {'loss': 0.2213, 'learning_rate': 1.0323e-04, 'epoch': 1.48, 'throughput': 36.28}

[INFO|2025-04-23 00:20:34] logging.py:143 >> {'loss': 0.2836, 'learning_rate': 9.6768e-05, 'epoch': 1.54, 'throughput': 36.28}

[INFO|2025-04-23 00:22:25] logging.py:143 >> {'loss': 0.2280, 'learning_rate': 9.0319e-05, 'epoch': 1.60, 'throughput': 36.28}

[INFO|2025-04-23 00:24:15] logging.py:143 >> {'loss': 0.1778, 'learning_rate': 8.3910e-05, 'epoch': 1.66, 'throughput': 36.28}

[INFO|2025-04-23 00:26:03] logging.py:143 >> {'loss': 0.1871, 'learning_rate': 7.7568e-05, 'epoch': 1.72, 'throughput': 36.27}

[INFO|2025-04-23 00:27:52] logging.py:143 >> {'loss': 0.2289, 'learning_rate': 7.1320e-05, 'epoch': 1.78, 'throughput': 36.27}

[INFO|2025-04-23 00:29:41] logging.py:143 >> {'loss': 0.2889, 'learning_rate': 6.5191e-05, 'epoch': 1.84, 'throughput': 36.27}

[INFO|2025-04-23 00:31:32] logging.py:143 >> {'loss': 0.2724, 'learning_rate': 5.9208e-05, 'epoch': 1.91, 'throughput': 36.25}

[INFO|2025-04-23 00:33:23] logging.py:143 >> {'loss': 0.2359, 'learning_rate': 5.3396e-05, 'epoch': 1.97, 'throughput': 36.26}

[INFO|2025-04-23 00:35:23] logging.py:143 >> {'loss': 0.1864, 'learning_rate': 4.7778e-05, 'epoch': 2.04, 'throughput': 36.26}

[INFO|2025-04-23 00:37:13] logging.py:143 >> {'loss': 0.0671, 'learning_rate': 4.2378e-05, 'epoch': 2.10, 'throughput': 36.26}

[INFO|2025-04-23 00:39:02] logging.py:143 >> {'loss': 0.1465, 'learning_rate': 3.7219e-05, 'epoch': 2.16, 'throughput': 36.23}

[INFO|2025-04-23 00:40:51] logging.py:143 >> {'loss': 0.0910, 'learning_rate': 3.2322e-05, 'epoch': 2.22, 'throughput': 36.22}

[INFO|2025-04-23 00:42:41] logging.py:143 >> {'loss': 0.1140, 'learning_rate': 2.7708e-05, 'epoch': 2.28, 'throughput': 36.21}

[INFO|2025-04-23 00:44:33] logging.py:143 >> {'loss': 0.1381, 'learning_rate': 2.3396e-05, 'epoch': 2.34, 'throughput': 36.23}

[INFO|2025-04-23 00:46:22] logging.py:143 >> {'loss': 0.0799, 'learning_rate': 1.9403e-05, 'epoch': 2.40, 'throughput': 36.24}

[INFO|2025-04-23 00:48:14] logging.py:143 >> {'loss': 0.0722, 'learning_rate': 1.5748e-05, 'epoch': 2.46, 'throughput': 36.25}

[INFO|2025-04-23 00:48:14] trainer.py:3984 >> Saving model checkpoint to saves/DeepSeek-LLM-7B-Base/lora/train_2025-04-22-23-24-06/checkpoint-200

[INFO|2025-04-23 00:48:14] configuration_utils.py:693 >> loading configuration file config.json from cache at /root/.cache/huggingface/hub/models--deepseek-ai--deepseek-llm-7b-base/snapshots/7683fea62db869066ddaff6a41d032262c490d4f/config.json

[INFO|2025-04-23 00:48:14] configuration_utils.py:765 >> Model config LlamaConfig {
  "architectures": [
    "LlamaForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "bos_token_id": 1,
  "eos_token_id": 2,
  "head_dim": 128,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 11008,
  "max_position_embeddings": 4096,
  "mlp_bias": false,
  "model_type": "llama",
  "num_attention_heads": 32,
  "num_hidden_layers": 30,
  "num_key_value_heads": 32,
  "pretraining_tp": 1,
  "rms_norm_eps": 1e-06,
  "rope_scaling": null,
  "rope_theta": 10000.0,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.51.3",
  "use_cache": true,
  "vocab_size": 102400
}


[INFO|2025-04-23 00:48:15] tokenization_utils_base.py:2510 >> tokenizer config file saved in saves/DeepSeek-LLM-7B-Base/lora/train_2025-04-22-23-24-06/checkpoint-200/tokenizer_config.json

[INFO|2025-04-23 00:48:15] tokenization_utils_base.py:2519 >> Special tokens file saved in saves/DeepSeek-LLM-7B-Base/lora/train_2025-04-22-23-24-06/checkpoint-200/special_tokens_map.json

[INFO|2025-04-23 00:50:13] logging.py:143 >> {'loss': 0.1003, 'learning_rate': 1.2444e-05, 'epoch': 2.53, 'throughput': 36.23}

[INFO|2025-04-23 00:52:01] logging.py:143 >> {'loss': 0.1155, 'learning_rate': 9.5063e-06, 'epoch': 2.59, 'throughput': 36.21}

[INFO|2025-04-23 00:53:51] logging.py:143 >> {'loss': 0.0962, 'learning_rate': 6.9464e-06, 'epoch': 2.65, 'throughput': 36.22}

[INFO|2025-04-23 00:55:43] logging.py:143 >> {'loss': 0.0808, 'learning_rate': 4.7752e-06, 'epoch': 2.71, 'throughput': 36.23}

[INFO|2025-04-23 00:57:33] logging.py:143 >> {'loss': 0.1206, 'learning_rate': 3.0018e-06, 'epoch': 2.77, 'throughput': 36.22}

[INFO|2025-04-23 00:59:23] logging.py:143 >> {'loss': 0.1259, 'learning_rate': 1.6335e-06, 'epoch': 2.83, 'throughput': 36.22}

[INFO|2025-04-23 01:01:11] logging.py:143 >> {'loss': 0.1580, 'learning_rate': 6.7616e-07, 'epoch': 2.89, 'throughput': 36.22}

[INFO|2025-04-23 01:02:59] logging.py:143 >> {'loss': 0.0620, 'learning_rate': 1.3368e-07, 'epoch': 2.95, 'throughput': 36.21}

[INFO|2025-04-23 01:04:07] trainer.py:3984 >> Saving model checkpoint to saves/DeepSeek-LLM-7B-Base/lora/train_2025-04-22-23-24-06/checkpoint-243

[INFO|2025-04-23 01:04:07] configuration_utils.py:693 >> loading configuration file config.json from cache at /root/.cache/huggingface/hub/models--deepseek-ai--deepseek-llm-7b-base/snapshots/7683fea62db869066ddaff6a41d032262c490d4f/config.json

[INFO|2025-04-23 01:04:07] configuration_utils.py:765 >> Model config LlamaConfig {
  "architectures": [
    "LlamaForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "bos_token_id": 1,
  "eos_token_id": 2,
  "head_dim": 128,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 11008,
  "max_position_embeddings": 4096,
  "mlp_bias": false,
  "model_type": "llama",
  "num_attention_heads": 32,
  "num_hidden_layers": 30,
  "num_key_value_heads": 32,
  "pretraining_tp": 1,
  "rms_norm_eps": 1e-06,
  "rope_scaling": null,
  "rope_theta": 10000.0,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.51.3",
  "use_cache": true,
  "vocab_size": 102400
}


[INFO|2025-04-23 01:04:08] tokenization_utils_base.py:2510 >> tokenizer config file saved in saves/DeepSeek-LLM-7B-Base/lora/train_2025-04-22-23-24-06/checkpoint-243/tokenizer_config.json

[INFO|2025-04-23 01:04:08] tokenization_utils_base.py:2519 >> Special tokens file saved in saves/DeepSeek-LLM-7B-Base/lora/train_2025-04-22-23-24-06/checkpoint-243/special_tokens_map.json

[INFO|2025-04-23 01:04:09] trainer.py:2681 >> 

Training completed. Do not forget to share your model on huggingface.co/models =)



[INFO|2025-04-23 01:04:09] trainer.py:3984 >> Saving model checkpoint to saves/DeepSeek-LLM-7B-Base/lora/train_2025-04-22-23-24-06

[INFO|2025-04-23 01:04:09] configuration_utils.py:693 >> loading configuration file config.json from cache at /root/.cache/huggingface/hub/models--deepseek-ai--deepseek-llm-7b-base/snapshots/7683fea62db869066ddaff6a41d032262c490d4f/config.json

[INFO|2025-04-23 01:04:09] configuration_utils.py:765 >> Model config LlamaConfig {
  "architectures": [
    "LlamaForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "bos_token_id": 1,
  "eos_token_id": 2,
  "head_dim": 128,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 11008,
  "max_position_embeddings": 4096,
  "mlp_bias": false,
  "model_type": "llama",
  "num_attention_heads": 32,
  "num_hidden_layers": 30,
  "num_key_value_heads": 32,
  "pretraining_tp": 1,
  "rms_norm_eps": 1e-06,
  "rope_scaling": null,
  "rope_theta": 10000.0,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.51.3",
  "use_cache": true,
  "vocab_size": 102400
}


[INFO|2025-04-23 01:04:09] tokenization_utils_base.py:2510 >> tokenizer config file saved in saves/DeepSeek-LLM-7B-Base/lora/train_2025-04-22-23-24-06/tokenizer_config.json

[INFO|2025-04-23 01:04:09] tokenization_utils_base.py:2519 >> Special tokens file saved in saves/DeepSeek-LLM-7B-Base/lora/train_2025-04-22-23-24-06/special_tokens_map.json

[WARNING|2025-04-23 01:04:10] logging.py:148 >> No metric eval_loss to plot.

[WARNING|2025-04-23 01:04:10] logging.py:148 >> No metric eval_accuracy to plot.

[INFO|2025-04-23 01:04:10] modelcard.py:450 >> Dropping the following result as it does not have all the necessary fields:
{'task': {'name': 'Causal Language Modeling', 'type': 'text-generation'}}