Nous Hermes 2 - Solar 10.7B - DeepSparse
This repo contains model files for Nous Hermes 2 - Solar 10.7B optimized for DeepSparse, a CPU inference runtime for sparse models.
This model was quantized and pruned with SparseGPT, using SparseML.
Inference
Install DeepSparse LLM for fast inference on CPUs:
pip install deepsparse-nightly[llm]
Run in a Python pipeline:
from deepsparse import TextGeneration
prompt = "How to make banana bread?"
formatted_prompt = f"### User:\n{prompt}\n\n### Assistant:\n"
model = TextGeneration(model_path="hf:nm-testing/Nous-Hermes-2-SOLAR-10.7B-pruned50-ds")
print(model(formatted_prompt, max_new_tokens=500).generations[0].text)
"""
To make banana bread, you will need the following ingredients:
- 3 cups of flour
- 1 teaspoon of baking soda
- 1/2 teaspoon of salt
- 1/2 cup of butter (softened)
- 1/2 cup of sugar
- 2 eggs
- 1 1/2 cups of mashed ripe bananas
- 1 teaspoon of vanilla extract
Here's how to make banana bread:
1. Preheat your oven to 350°F (175°C).
2. In a bowl, mix together the flour, baking soda, and salt.
3. In another bowl, cream together the butter and sugar until light and fluffy.
4. Add the eggs one at a time, beating well after each addition.
5. Stir in the mashed bananas and vanilla extract.
6. Gradually stir the flour mixture into the banana mixture.
7. Pour the batter into a greased loaf pan.
8. Bake for 1 hour, or until a toothpick inserted in the center comes out clean.
9. Allow the bread to cool in the pan for 10 minutes before removing it.
Enjoy your delicious banana bread!
"""
Prompt template
### User:\n
{prompt}
### Assistant:\n
Sparsification
For details on how this model was sparsified, see the recipe.yaml
in this repo and follow the instructions below.
git clone https://github.com/neuralmagic/sparseml
pip install -e "sparseml[transformers]"
python sparseml/src/sparseml/transformers/sparsification/obcq/obcq.py NousResearch/Nous-Hermes-2-SOLAR-10.7B open_platypus --recipe recipe.yaml --save True
python sparseml/src/sparseml/transformers/sparsification/obcq/export.py --task text-generation --model_path obcq_deployment
cp deployment/model.onnx deployment/model-orig.onnx
Run this kv-cache injection to speed up the model at inference by caching the Key and Value states:
import os
import onnx
from sparseml.exporters.kv_cache_injector import KeyValueCacheInjector
input_file = "deployment/model-orig.onnx"
output_file = "deployment/model.onnx"
model = onnx.load(input_file, load_external_data=False)
model = KeyValueCacheInjector(model_path=os.path.dirname(input_file)).apply(model)
onnx.save(model, output_file)
print(f"Modified model saved to: {output_file}")
Follow the instructions on our One Shot With SparseML page for a step-by-step guide for performing one-shot quantization of large language models.
Slack
For further support, and discussions on these models and AI in general, join Neural Magic's Slack Community
- Downloads last month
- 7
Model tree for nm-testing/Nous-Hermes-2-SOLAR-10.7B-pruned50-ds
Base model
upstage/SOLAR-10.7B-v1.0