Update README.md

README.md

@@ -1,11 +1,63 @@
+---
+base_model: Xenova/llama2.c-stories110M
+inference: true
+model_type: llama
+quantized_by: mgoin
+tags:
+- nm-vllm
+- sparse
+---
+
+## llama2.c-stories110M-pruned50
+This repo contains model files for [llama2.c 110M tinystories](https://huggingface.co/Xenova/llama2.c-stories110M) optimized for [NM-vLLM](https://github.com/neuralmagic/nm-vllm), a high-throughput serving engine for compressed LLMs.
+
+This model was pruned with [SparseGPT](https://arxiv.org/abs/2301.00774), using [SparseML](https://github.com/neuralmagic/sparseml).
+
+## Inference
+Install [NM-vLLM](https://github.com/neuralmagic/nm-vllm) for fast inference and low memory-usage: 
+```bash
+pip install nm-vllm[sparse]
+```
+Run in a Python pipeline for local inference:
+```python
+from vllm import LLM, SamplingParams
+
+model = LLM("nm-testing/llama2.c-stories110M-pruned2.4", sparsity="sparse_w16a16")
+prompt = "My name is "
+formatted_prompt =  f"<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n"
+
+sampling_params = SamplingParams(max_tokens=100,temperature=0)
+outputs = model.generate(prompt, sampling_params=sampling_params)
+print(outputs[0].outputs[0].text)
+
+""""
+3 years old. My name is Sam. I love to play with my toys. I love to play with my toys.
+One day, my mom takes me to the park. She brings a big bag. She takes out a big bag. It is full of things.
+At the park, Sam sees a big box. He sees it was made from paper. He sees it is made from paper. He sees it is made from paper.
+Sam's mom takes outs
+"""
+```
+
+## Prompt template
+
+N/A
+
+## Sparsification
+For details on how this model was sparsified, see the `recipe.yaml` in this repo and follow the instructions below.
+
+Install [SparseML](https://github.com/neuralmagic/sparseml):
+```bash
+git clone https://github.com/neuralmagic/sparseml
+pip install -e "sparseml[transformers]"
+```
+
+Replace the recipe as you like and run this one-shot compression script to apply SparseGPT:
 ```python
 import sparseml.transformers
 
 original_model_name = "Xenova/llama2.c-stories110M"
+calibration_dataset = "open_platypus"
 output_directory = "output/"
-final_model_name = "nm-testing/llama2.c-stories110M-pruned2.4"
-
-dataset = "open_platypus"
 
 recipe = """
 test_stage:
@@ -20,17 +72,13 @@ test_stage:
 
 # Apply SparseGPT to the model
 sparseml.transformers.oneshot(
-    model_name_or_path=original_model_name,
-    dataset_name=dataset,
+    model=original_model_name,
+    dataset=calibration_dataset,
     recipe=recipe,
     output_dir=output_directory,
 )
+```
 
-# Upload the output model to Hugging Face Hub
-from huggingface_hub import HfApi
+## Slack
 
-HfApi().upload_folder(
-    folder_path=output_directory,
-    repo_id=final_model_name,
-)
-```
+For further support, and discussions on these models and AI in general, join [Neural Magic's Slack Community](https://join.slack.com/t/discuss-neuralmagic/shared_invite/zt-q1a1cnvo-YBoICSIw3L1dmQpjBeDurQ)