File size: 3,291 Bytes
4a5a2c3 043be07 d22b4fa 043be07 d22b4fa 043be07 d22b4fa a6b00c3 d22b4fa a6b00c3 d22b4fa a6b00c3 d22b4fa a6b00c3 4a5a2c3 d22b4fa 043be07 d22b4fa 043be07 3251ce4 ca8442b 3251ce4 67bb427 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 |
---
license: apache-2.0
datasets:
- grammarly/coedit
language:
- en
tags:
- text-generation-inference
- candle
widget:
- text: >-
Fix the grammar: When I grow up,
I start to understand what he said is
quite right.
example_title: Fluency
- text: >-
Make this text coherent: Their flight
is weak. They run quickly through
the tree canopy.
example_title: Coherence
- text: >-
Rewrite to make this easier to understand: A storm surge is what
forecasters consider a hurricane's most treacherous aspect.
example_title: Simplification
- text: >-
Paraphrase this: Do you know where I was born?
example_title: Paraphrase
- text: >-
Write this more formally: omg i love that song im
listening to it right now
example_title: Formalize
- text: >-
Write in a more neutral way: The authors' exposé on nutrition studies.
example_title: Neutralize
---
# Quantized candle weights for the CoEdIT model
Quantized weights of [CoEdIT](https://github.com/vipulraheja/coedit) for inference with [candle](https://github.com/huggingface/candle/tree/main/candle-examples/examples/quantized-t5).
## Usage
Clone [candle](https://github.com/huggingface/candle), and run the `quantized-t5` example:
```shell
$ cargo run --example quantized-t5 --release -- \
--model-id "jbochi/candle-coedit-quantized" \
--prompt "Make this text coherent: Their flight is weak. They run quickly through the tree canopy." \
--temperature 0
...
Although their flight is weak, they run quickly through the tree canopy.
```
By default, it will use CoEdIT-large (770M params, 643 MB).
To use CoEdIT-xl (3B params, 2.34 GB), specify the weight-file and config-file:
```shell
$ cargo run --example quantized-t5 --release -- \
--model-id "jbochi/candle-coedit-quantized" \
--weight-file "model-xl.gguf" \
--config-file "config-xl.json" \
--prompt "Rewrite to make this easier to understand: Note that a storm surge is what forecasters consider a hurricane's most treacherous aspect." \
--temperature 0
...
Note that a storm surge is what forecasters consider a hurricane's most dangerous part.
```
## Models available
These are all the available formats. Weight file is named `{name}_{quant}.gguf` and config-file `config-{base}.json`
| Model | Base model | Quantization | # Params | Size |
| ----- | ---------- | ------------ | ------ | ---- |
| - | [large](https://huggingface.co/grammarly/coedit-large) | None | 770M | 3.13 GB |
| model | large | 6k | 770M | 643 MB |
| model-4k | large | 4k | 770M | 441 MB |
| model-4_0 | large | 4_0 | 770M | 441 MB |
| | [xl](https://huggingface.co/grammarly/coedit-xl) | None | 3B | 11.4 GB |
| model-xl | xl | 6k | 3B | 2.34 GB |
| model-xl-4k | xl | 4k | 3B | 1.6 GB |
| model-xl-4_0 | xl | 4_0 | 3B | 1.6 GB |
| - | [xxl](https://huggingface.co/grammarly/coedit-xxl) | None | 11B | 44.5 GB |
| model-xxl | xxl | 6k | 11B | 9.14 GB |
| model-xxl-4k | xxl | 4k | 11B | WIP |
| model-xxl-4_0 | xxl | 4_0 | 11B | WIP |
## Model generation
The weights were quantized using candle:
```shell
cargo run --example tensor-tools --release -- quantize \
--quantization q6k \
/path/to/coedit-<version>/model.safetensors \
--out-file model<version>.gguf
```
|