nm-testing
/

llama2.c-stories110M-pruned50-compressed-tensors

Text Generation

text-generation-inference

Model card Files Files and versions Community

dtransposed commited on Apr 25, 2024

Commit

b109b5e

·

verified ·

1 Parent(s): 39acfcb

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -12,7 +12,7 @@ tags:
 This repo contains model files for [llama2.c 110M tinystories](https://huggingface.co/Xenova/llama2.c-stories110M) optimized for [NM-vLLM](https://github.com/neuralmagic/nm-vllm), a high-throughput serving engine for compressed LLMs.
 This model was pruned with [SparseGPT](https://arxiv.org/abs/2301.00774), using [SparseML](https://github.com/neuralmagic/sparseml).
 ## Inference
 Install [NM-vLLM](https://github.com/neuralmagic/nm-vllm) for fast inference and low memory-usage:
 ```bash

 This repo contains model files for [llama2.c 110M tinystories](https://huggingface.co/Xenova/llama2.c-stories110M) optimized for [NM-vLLM](https://github.com/neuralmagic/nm-vllm), a high-throughput serving engine for compressed LLMs.
 This model was pruned with [SparseGPT](https://arxiv.org/abs/2301.00774), using [SparseML](https://github.com/neuralmagic/sparseml).
+The weights for this model were saved using [compressed-tensors](https://github.com/neuralmagic/compressed-tensors/pull/30) library. The chosen compression is format bitmask-compression.
 ## Inference
 Install [NM-vLLM](https://github.com/neuralmagic/nm-vllm) for fast inference and low memory-usage:
 ```bash