Hugging Face's logo

bartowski
/
gemma-2-9b-it-exl2

Text Generation
Transformers
conversational
Inference Endpoints
Model card Files Community
1
gemma-2-9b-it-exl2
File size: 2,970 Bytes 
7e91c42
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3720b35
 
 
 
 
7e91c42
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
 
---
license: gemma
library_name: transformers
pipeline_tag: text-generation
extra_gated_heading: Access Gemma on Hugging Face
extra_gated_prompt: >-
  To access Gemma on Hugging Face, you’re required to review and agree to
  Google’s usage license. To do this, please ensure you’re logged in to Hugging
  Face and click below. Requests are processed immediately.
extra_gated_button_content: Acknowledge license
tags:
- conversational
quantized_by: bartowski
---

## Exllama v2 Quantizations of gemma-2-9b-it

Using <a href="https://github.com/turboderp/exllamav2/releases/tag/v0.1.7">turboderp's ExLlamaV2 v0.1.7</a> for quantization.

<b>The "main" branch only contains the measurement.json, download one of the other branches for the model (see below)</b>

Each branch contains an individual bits per weight, with the main one containing only the meaurement.json for further conversions.

Original model: https://huggingface.co/google/gemma-2-9b-it

## Prompt format

```
<bos><start_of_turn>user
{prompt}<end_of_turn>
<start_of_turn>model
<end_of_turn>
<start_of_turn>model

```

Note that this model does not support a System prompt.

## Available sizes

| Branch | Bits | lm_head bits | VRAM (4k) | VRAM (16k) | VRAM (32k) | Description |
| ----- | ---- | ------- | ------ | ------ | ------ | ------------ |
| [8_0](https://huggingface.co/bartowski/gemma-2-9b-it-exl2/tree/8_0) | 8.0 | 8.0 | 11.9 GB | 15.9 GB | 21.3 GB | Maximum quality that ExLlamaV2 can produce, near unquantized performance. |
| [6_5](https://huggingface.co/bartowski/gemma-2-9b-it-exl2/tree/6_5) | 6.5 | 8.0 | 10.4 GB | 14.4 GB | 19.8 GB | Very similar to 8.0, good tradeoff of size vs performance, **recommended**. |
| [5_0](https://huggingface.co/bartowski/gemma-2-9b-it-exl2/tree/5_0) | 5.0 | 6.0 | 8.6 GB | 12.6 GB | 18.0 GB | Slightly lower quality vs 6.5, but usable on 8GB cards. |
| [4_25](https://huggingface.co/bartowski/gemma-2-9b-it-exl2/tree/4_25) | 4.25 | 6.0 | 7.9 GB | 11.9 GB | 17.3 GB | GPTQ equivalent bits per weight, slightly higher quality. |
| [3_5](https://huggingface.co/bartowski/gemma-2-9b-it-exl2/tree/3_5) | 3.5 | 6.0 | 7.1 GB | 11.1 GB | 16.9 GB | Lower quality, only use if you have to. |

## Download instructions

With git:

```shell
git clone --single-branch --branch 6_5 https://huggingface.co/bartowski/gemma-2-9b-it-exl2 gemma-2-9b-it-exl2-6_5
```

With huggingface hub (credit to TheBloke for instructions):

```shell
pip3 install huggingface-hub
```

To download a specific branch, use the `--revision` parameter. For example, to download the 6.5 bpw branch:

Linux:

```shell
huggingface-cli download bartowski/gemma-2-9b-it-exl2 --revision 6_5 --local-dir gemma-2-9b-it-exl2-6_5
```

Windows (which apparently doesn't like _ in folders sometimes?):

```shell
huggingface-cli download bartowski/gemma-2-9b-it-exl2 --revision 6_5 --local-dir gemma-2-9b-it-exl2-6.5
```

Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski