TINYLLAMA-PY-CODER -BNB-4BIT-LORA_MODEL-4K

AI trained for 1 epoch on a dataset with 10% of the original size, because I used unsloth's colab notebook with GPU free. original dataset = data-oss_instruct-decontaminated_python.jsonl (I removed the other programming languages, leaving only Python). The training took 30 iterations to make a season, it would have been enough to train with two, but as this was the first successful training in the collab thanks to unsloth, to whom I am heartily grateful!

==((====))== Unsloth - 2x faster free finetuning | Num GPUs = 1
\\ /| Num examples = 967 | Num Epochs = 1
O^O/ \_/ \ Batch size per device = 2 | Gradient Accumulation steps = 16
\ / Total batch size = 32 | Total steps = 30
"-____-" Number of trainable parameters = 100,925,440
[30/30 26:26, Epoch 0/1]
Step Training Loss
1 1.737000
2 1.738000
3 1.384700
4 1.086400
5 1.009600
6 0.921000
7 0.830400
8 0.808900
9 0.774500
10 0.759900
11 0.736100
12 0.721200
13 0.733200
14 0.701000
15 0.711700
16 0.701400
17 0.689500
18 0.678800
19 0.675200
20 0.680500
21 0.685800
22 0.681200
23 0.672000
24 0.679900
25 0.675500
26 0.666600
27 0.687900
28 0.653600
29 0.672500
30 0.660900

I think one season was too little, in the future I intend to try with two seasons. I fought tirelessly to be able to train an AI on my computer without a GPU, and with only 8GB of offline memory, and all I managed was to train the AI "gpt2" 500M of similar parameters and AI, but my datasets were always small and the AI didn't progress at all, but when I see this notebook and the unsloth codes I see that they will be the way to achieve what I tried without success! Thanks unsloth!

README.md

@@ -8,6 +8,9 @@ tags:
 - unsloth
 - llama
 - trl
+- tinyllamacoder-py
+- coder-py
+- coder
 base_model: unsloth/tinyllama-bnb-4bit
 ---
 
@@ -19,4 +22,4 @@ base_model: unsloth/tinyllama-bnb-4bit
 
 This llama model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.
 
-[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)
+[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)