Hugging Face's logo

hardlyworking
/
Games8B

Text Generation
Transformers
Safetensors
qwen3
axolotl
Generated from Trainer
conversational
text-generation-inference
Model card Files Community
hardlyworking commited on
Commit
9daab4a
·
verified ·
1 Parent(s): 9576c91

End of training

Browse files
Files changed (1) hide show
  1. README.md +172 -0
README.md ADDED
@@ -0,0 +1,172 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: transformers
3
+ license: apache-2.0
4
+ base_model: GreenerPastures/Bald-Beaver-8B
5
+ tags:
6
+ - axolotl
7
+ - generated_from_trainer
8
+ datasets:
9
+ - NewEden/Joe-Games
10
+ model-index:
11
+ - name: Games8B
12
+ results: []
13
+ ---
14
+
15
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
16
+ should probably proofread and complete it, then remove this comment. -->
17
+
18
+ [<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)
19
+ <details><summary>See axolotl config</summary>
20
+
21
+ axolotl version: `0.10.0.dev0`
22
+ ```yaml
23
+ base_model: GreenerPastures/Bald-Beaver-8B
24
+
25
+ load_in_8bit: false
26
+ load_in_4bit: false
27
+ strict: false
28
+
29
+ chat_template: qwen3
30
+ datasets:
31
+ - path: NewEden/Joe-Games
32
+ type: completion
33
+
34
+ val_set_size: 0.05
35
+ output_dir: ./outputs/out
36
+ dataset_prepared_path: last_run_prepared
37
+ shuffle_merged_datasets: true
38
+
39
+ hub_model_id: hardlyworking/Games8B
40
+ hub_strategy: "all_checkpoints"
41
+ push_dataset_to_hub:
42
+ hf_use_auth_token: true
43
+
44
+ plugins:
45
+ - axolotl.integrations.liger.LigerPlugin
46
+ - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin
47
+ liger_rope: true
48
+ liger_rms_norm: true
49
+ liger_layer_norm: true
50
+ liger_glu_activation: true
51
+ liger_fused_linear_cross_entropy: false
52
+ cut_cross_entropy: true
53
+
54
+ sequence_len: 32768
55
+ sample_packing: true
56
+ eval_sample_packing: true
57
+ pad_to_sequence_len: true
58
+
59
+ wandb_project: Qwen8B
60
+ wandb_entity:
61
+ wandb_watch:
62
+ wandb_name: Qwen8B
63
+ wandb_log_model:
64
+
65
+ evals_per_epoch: 8
66
+ eval_table_size:
67
+ eval_max_new_tokens: 128
68
+
69
+ max_grad_norm: 1.0
70
+ gradient_accumulation_steps: 1
71
+ micro_batch_size: 1
72
+ num_epochs: 2
73
+ optimizer: adamw_bnb_8bit
74
+ lr_scheduler: cosine
75
+ learning_rate: 1e-5
76
+
77
+ train_on_inputs: false
78
+ group_by_length: false
79
+ bf16: auto
80
+ fp16:
81
+ tf32: false
82
+
83
+ gradient_checkpointing: offload
84
+ gradient_checkpointing_kwargs:
85
+ use_reentrant: false
86
+ early_stopping_patience:
87
+ resume_from_checkpoint:
88
+ local_rank:
89
+ logging_steps: 1
90
+ xformers_attention:
91
+ flash_attention: true
92
+ s2_attention:
93
+
94
+ deepspeed:
95
+
96
+ warmup_ratio: 0.05
97
+ saves_per_epoch: 1
98
+ debug:
99
+ weight_decay: 0.01
100
+ fsdp:
101
+ fsdp_config:
102
+ special_tokens:
103
+ pad_token:
104
+ ```
105
+
106
+ </details><br>
107
+
108
+ # Games8B
109
+
110
+ This model is a fine-tuned version of [GreenerPastures/Bald-Beaver-8B](https://huggingface.co/GreenerPastures/Bald-Beaver-8B) on the NewEden/Joe-Games dataset.
111
+ It achieves the following results on the evaluation set:
112
+ - Loss: 1.9796
113
+
114
+ ## Model description
115
+
116
+ More information needed
117
+
118
+ ## Intended uses & limitations
119
+
120
+ More information needed
121
+
122
+ ## Training and evaluation data
123
+
124
+ More information needed
125
+
126
+ ## Training procedure
127
+
128
+ ### Training hyperparameters
129
+
130
+ The following hyperparameters were used during training:
131
+ - learning_rate: 1e-05
132
+ - train_batch_size: 1
133
+ - eval_batch_size: 1
134
+ - seed: 42
135
+ - distributed_type: multi-GPU
136
+ - num_devices: 4
137
+ - total_train_batch_size: 4
138
+ - total_eval_batch_size: 4
139
+ - optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
140
+ - lr_scheduler_type: cosine
141
+ - lr_scheduler_warmup_steps: 13
142
+ - num_epochs: 2.0
143
+
144
+ ### Training results
145
+
146
+ | Training Loss | Epoch | Step | Validation Loss |
147
+ |:-------------:|:------:|:----:|:---------------:|
148
+ | 1.6433 | 0.0074 | 1 | 2.0240 |
149
+ | 1.5696 | 0.125 | 17 | 2.0077 |
150
+ | 2.5862 | 0.25 | 34 | 1.9995 |
151
+ | 1.9231 | 0.375 | 51 | 1.9944 |
152
+ | 1.9655 | 0.5 | 68 | 1.9908 |
153
+ | 1.6909 | 0.625 | 85 | 1.9880 |
154
+ | 1.8634 | 0.75 | 102 | 1.9857 |
155
+ | 1.6684 | 0.875 | 119 | 1.9832 |
156
+ | 1.8727 | 1.0 | 136 | 1.9816 |
157
+ | 1.542 | 1.125 | 153 | 1.9806 |
158
+ | 2.5733 | 1.25 | 170 | 1.9801 |
159
+ | 1.8934 | 1.375 | 187 | 1.9797 |
160
+ | 1.9386 | 1.5 | 204 | 1.9796 |
161
+ | 1.6764 | 1.625 | 221 | 1.9796 |
162
+ | 1.8524 | 1.75 | 238 | 1.9796 |
163
+ | 1.661 | 1.875 | 255 | 1.9795 |
164
+ | 1.8697 | 2.0 | 272 | 1.9796 |
165
+
166
+
167
+ ### Framework versions
168
+
169
+ - Transformers 4.51.3
170
+ - Pytorch 2.6.0+cu124
171
+ - Datasets 3.5.1
172
+ - Tokenizers 0.21.1