FantasiaFoundry commited on
Commit
32d703a
1 Parent(s): cc01c1e

Attributions.

Browse files
Files changed (1) hide show
  1. README.md +11 -1
README.md CHANGED
@@ -7,6 +7,11 @@ tags:
7
  - quantized
8
  - text-generation-inference
9
  ---
 
 
 
 
 
10
  Simple python script (`gguf-imat.py`) to generate various GGUF-IQ-Imatrix quantizations from a Hugging Face `author/model` input, for Windows and NVIDIA hardware.
11
 
12
  This is setup for a Windows machine with 8GB of VRAM, assuming use with an NVIDIA GPU. If you want to change the the `-ngl` (number of GPU layers) amount, you can do so at **line 120**. This is only relevant during the `--imatrix` data generation. If you don't have enough VRAM you can decrease the `-ngl` amount or set it to 0 to only use your System RAM instead for all layers.
@@ -25,4 +30,9 @@ python .\gguf-imat.py
25
  ```
26
  Quantizations will be output into the created `models\{model-name}-GGUF` folder.
27
  <br><br>
28
- **If this proves useful for you, feel free to credit and share the repository.**
 
 
 
 
 
 
7
  - quantized
8
  - text-generation-inference
9
  ---
10
+
11
+ **GGUF-IQ-Imatrix-Quantization-Script:**
12
+
13
+ ![image/png](https://cdn-uploads.huggingface.co/production/uploads/65ddabb9bbffb280f4b45d8e/zQNp-muoarsSglA8cyHal.png)
14
+
15
  Simple python script (`gguf-imat.py`) to generate various GGUF-IQ-Imatrix quantizations from a Hugging Face `author/model` input, for Windows and NVIDIA hardware.
16
 
17
  This is setup for a Windows machine with 8GB of VRAM, assuming use with an NVIDIA GPU. If you want to change the the `-ngl` (number of GPU layers) amount, you can do so at **line 120**. This is only relevant during the `--imatrix` data generation. If you don't have enough VRAM you can decrease the `-ngl` amount or set it to 0 to only use your System RAM instead for all layers.
 
30
  ```
31
  Quantizations will be output into the created `models\{model-name}-GGUF` folder.
32
  <br><br>
33
+
34
+ ### **Credits:**
35
+
36
+ **If this proves useful for you, feel free to credit and share the repository.**
37
+
38
+ **Made in conjunction with [@Lewdiculous](https://huggingface.co/Lewdiculous).**