End of training

README.md

@@ -1,110 +1,53 @@
 ---
 library_name: transformers
 license: mit
-datasets:
-- roneneldan/TinyStories
-- Salesforce/wikitext
-- abhinand/alpaca-gpt4-sharegpt
-- shibing624/sharegpt_gpt4
-- ChristophSchuhmann/basic-math-problems-with-step-by-step-solutions
-- ajibawa-2023/SlimOrca-ShareGPT
-language:
-- en
-pipeline_tag: text-generation
+base_model: arshiaafshani/Arsh-llm
+tags:
+- generated_from_trainer
+model-index:
+- name: Arsh-llm
+  results: []
 ---
 
-# Arsh-llm: A Compact 500M Parameter Powerhouse 🚀
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
 
-**Arsh-llm** is a 500-million-parameter language model built on the Llama architecture, designed to shine in generating creative stories, coherent text, and functional code. Pretrained for 35 hours on a T4 GPU using a curated mix of small yet powerful datasets, and fine-tuned for 5 hours on conversational data, this model is a lean, mean, text-generating machine with massive potential. With a training loss between **1.2–1.9**, it’s already showing promise and is ready to level up with more training. Buckle up—this is just the beginning! 😎
+# Arsh-llm
 
-## Model Overview
+This model is a fine-tuned version of [arshiaafshani/Arsh-llm](https://huggingface.co/arshiaafshani/Arsh-llm) on an unknown dataset.
 
-- **Architecture**: Llama-based causal language model
-- **Parameters**: 500M
-- **Context Length**: 128 tokens
-- **Pretraining Duration**: \~35 hours on NVIDIA T4 GPU
-- **Fine-tuning Duration**: \~5 hours on conversational datasets
-- **Training Loss**: 1.2–1.9 (with room to improve!)
-- **Library**: Transformers (Hugging Face)
-- **License**: MIT
+## Model description
 
-## Datasets
+More information needed
 
-Arsh-llm was trained on a diverse set of datasets to ensure versatility in storytelling, text generation, and code-related tasks:
+## Intended uses & limitations
 
-- **roneneldan/TinyStories**: Short, creative stories for narrative generation.
-- **Salesforce/wikitext**: Wikipedia-based text for general knowledge and coherence.
-- **abhinand/alpaca-gpt4-sharegpt**: Instruction-based conversational data for task-oriented responses.
-- **shibing624/sharegpt_gpt4**: High-quality conversational data for chat-like interactions.
-- **ChristophSchuhmann/basic-math-problems-with-step-by-step-solutions**: Math problems with solutions to boost logical reasoning.
+More information needed
 
-Fine-tuning was performed on a structured ShareGPT chat template to enhance conversational abilities, making Arsh-llm a great starting point for dialogue-based applications.
+## Training and evaluation data
 
-## Use Cases
+More information needed
 
-Arsh-llm is a versatile model with applications in:
+## Training procedure
 
-- **Creative Writing**: Generate engaging short stories or narrative prompts.
-- **Code Generation**: Produce functional code snippets for various programming tasks.
-- **Conversational AI**: Power chatbots or assistants with natural dialogue.
-- **Educational Tools**: Assist with math problem-solving or explain concepts step-by-step.
+### Training hyperparameters
 
-> **Note**: This model is a work in progress. For production-grade performance, further pretraining on larger datasets and post-training on conversational data is recommended.
+The following hyperparameters were used during training:
+- learning_rate: 3e-05
+- train_batch_size: 4
+- eval_batch_size: 8
+- seed: 42
+- gradient_accumulation_steps: 12
+- total_train_batch_size: 48
+- optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
+- lr_scheduler_type: linear
+- lr_scheduler_warmup_steps: 2000
+- training_steps: 1200
+- mixed_precision_training: Native AMP
 
-## Getting Started
+### Framework versions
 
-To use Arsh-llm, you can load it directly from Hugging Face:
-
-```python
-from transformers import AutoModelForCausalLM, AutoTokenizer
-
-# Load model and tokenizer
-model = AutoModelForCausalLM.from_pretrained("arshiaafshani/Arsh-llm")
-tokenizer = AutoTokenizer.from_pretrained("arshiaafshani/Arsh-llm")
-
-# Example: Generate a response
-messages = [{"role": "user", "content": "Write a short story about a brave robot."}]
-input_text = tokenizer.apply_chat_template(messages, tokenize=False)
-inputs = tokenizer(input_text, return_tensors="pt")
-outputs = model.generate(**inputs, max_length=200)
-print(tokenizer.decode(outputs[0], skip_special_tokens=True))
-```
-
-## Training Details
-
-- **Pretraining**: Conducted on a T4 GPU for \~35 hours using a mix of TinyStories, WikiText, and other datasets to build a strong foundation in text and story generation.
-- **Fine-tuning**: 5 hours on ShareGPT-based conversational data with a structured chat template to enhance dialogue capabilities.
-- **Hardware**: NVIDIA T4 GPU (15GB VRAM).
-- **Training Loss**: Achieved 1.2–1.9, indicating solid performance with significant potential for improvement through extended training.
-
-## Limitations
-
-- **Current Stage**: Arsh-llm is not yet fully optimized. It performs well for its size but requires additional training to compete with larger models.
-- **Dataset Size**: Pretrained on relatively small datasets, which limits its generalization. Scaling up to larger datasets will unlock its full potential.
-- **Context Length**: Limited to 128 tokens, which may constrain performance on longer sequences.
-- **Not Production-Ready**: This model is best used as a base for further fine-tuning rather than as a standalone solution.
-
-## Future Plans
-
-The journey doesn’t end here! Arsh-llm is set to evolve with:
-
-- **Extended Pretraining**: Leveraging larger datasets for broader knowledge and better generalization.
-- **Conversational Fine-tuning**: Enhancing dialogue capabilities with advanced post-training techniques.
-- **Benchmarking**: Evaluating performance against similar models (e.g., TinyLlama, Phi-1.5) on tasks like MMLU, HumanEval, and GSM8K.
-- **Community Feedback**: Incorporating user insights to refine and improve the model.
-
-Stay tuned—Arsh-llm is on its way to becoming a legend! 🔥
-
-## License
-
-This model is licensed under the MIT License, allowing for flexible use in both research and commercial applications. Feel free to build upon, modify, or share it!
-
-## Acknowledgments
-
-- Built with ❤️ by Arshia Afshani.
-- Powered by the Hugging Face Transformers library.
-- Thanks to the open-source community for providing the amazing datasets that made this model possible.
-
----
-
-**Ready to take Arsh-llm for a spin?** Clone it, train it, and let’s make it a superstar together! 🌟 For questions, feedback, or collabs, reach out via Hugging Face or open an issue in the repo.
+- Transformers 4.52.2
+- Pytorch 2.6.0+cu124
+- Datasets 3.6.0
+- Tokenizers 0.21.1

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:dff691ae8c26f29915c826edf637c3238a0409808304a965811c84b6d215c76e
+oid sha256:d3259b87466bbb50fa26f44fff2217f3cf45b02476e18022a810cd81c84fdb1a
 size 2013432032

tokenizer.json

@@ -2,7 +2,7 @@
   "version": "1.0",
   "truncation": {
     "direction": "Right",
-    "max_length": 256,
+    "max_length": 1024,
     "strategy": "LongestFirst",
     "stride": 0
   },
@@ -97,6 +97,15 @@
       "rstrip": false,
       "normalized": false,
       "special": true
+    },
+    {
+      "id": 32768,
+      "content": "<|eot_id|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
     }
   ],
   "normalizer": {

tokenizer_config.json

@@ -79,6 +79,14 @@
       "rstrip": false,
       "single_word": false,
       "special": true
+    },
+    "32768": {
+      "content": "<|eot_id|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
     }
   },
   "bos_token": "<sos>",

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2916ab9e1755803b1193e934ee58981fcce2b041c1bb51d121df465f7d8fd9b2
+oid sha256:10d4d79d84242ba0fb3e94e4b642f47ecf74e27a3072d45c85b64945e3275f51
 size 5240