|
|
--- |
|
|
base_model: ArliAI/QwQ-32B-ArliAI-RpR-v3 |
|
|
language: |
|
|
- en |
|
|
license: apache-2.0 |
|
|
tags: |
|
|
- llama-cpp |
|
|
- gguf-my-repo |
|
|
thumbnail: https://cdn-uploads.huggingface.co/production/uploads/6625f4a8a8d1362ebcc3851a/coilCTGeL0OUYr9PA9zna.jpeg |
|
|
--- |
|
|
|
|
|
# Triangle104/QwQ-32B-ArliAI-RpR-v3-Q3_K_M-GGUF |
|
|
This model was converted to GGUF format from [`ArliAI/QwQ-32B-ArliAI-RpR-v3`](https://huggingface.co/ArliAI/QwQ-32B-ArliAI-RpR-v3) using llama.cpp via the ggml.ai's [GGUF-my-repo](https://huggingface.co/spaces/ggml-org/gguf-my-repo) space. |
|
|
Refer to the [original model card](https://huggingface.co/ArliAI/QwQ-32B-ArliAI-RpR-v3) for more details on the model. |
|
|
|
|
|
--- |
|
|
RpR (RolePlay with Reasoning) is a new series of models from ArliAI. This series builds directly upon the successful dataset curation methodology and training methods developed for the RPMax series. |
|
|
|
|
|
RpR models use the same curated, deduplicated RP and creative writing |
|
|
dataset used for RPMax, with a focus on variety to ensure high |
|
|
creativity and minimize cross-context repetition. Users familiar with |
|
|
RPMax will recognize the unique, non-repetitive writing style unlike |
|
|
other finetuned-for-RP models. |
|
|
|
|
|
With the release of QwQ as the first high performing open-source |
|
|
reasoning model that can be easily trained, it was clear that the |
|
|
available instruct and creative writing reasoning datasets contains only |
|
|
one response per example. This is type of single response dataset used |
|
|
for training reasoning models causes degraded output quality in long |
|
|
multi-turn chats. Which is why Arli AI decided to create a real RP model |
|
|
capable of long multi-turn chat with reasoning. |
|
|
|
|
|
In order to create RpR, we first had to actually create the reasoning |
|
|
RP dataset by re-processing our existing known-good RPMax dataset into a |
|
|
reasoning dataset. This was possible by using the base QwQ Instruct |
|
|
model itself to create the reasoning process for every turn in the RPMax |
|
|
dataset conversation examples, which is then further refined in order |
|
|
to make sure the reasoning is in-line with the actual response examples |
|
|
from the dataset. |
|
|
|
|
|
Another important thing to get right is to make sure the model is |
|
|
trained on examples that present reasoning blocks in the same way as it |
|
|
encounters it during inference. Which is, never seeing the reasoning |
|
|
blocks in it's context. In order to do this, the training run was |
|
|
completed using axolotl with manual template-free segments dataset in |
|
|
order to make sure that the model is never trained to see the reasoning |
|
|
block in the context. Just like how the model will be used during |
|
|
inference time. |
|
|
|
|
|
The result of training QwQ on this dataset with this method are |
|
|
consistently coherent and interesting outputs even in long multi-turn RP |
|
|
chats. This is as far as we know the first true correctly-trained |
|
|
reasoning model trained for RP and creative writing. |
|
|
|
|
|
You can access the model at https://arliai.com and we also have a models ranking page at https://www.arliai.com/models-ranking |
|
|
|
|
|
Ask questions in our new Discord Server https://discord.com/invite/t75KbPgwhk or on our subreddit https://www.reddit.com/r/ArliAI/ |
|
|
|
|
|
--- |
|
|
## Use with llama.cpp |
|
|
Install llama.cpp through brew (works on Mac and Linux) |
|
|
|
|
|
```bash |
|
|
brew install llama.cpp |
|
|
|
|
|
``` |
|
|
Invoke the llama.cpp server or the CLI. |
|
|
|
|
|
### CLI: |
|
|
```bash |
|
|
llama-cli --hf-repo Triangle104/QwQ-32B-ArliAI-RpR-v3-Q3_K_M-GGUF --hf-file qwq-32b-arliai-rpr-v3-q3_k_m.gguf -p "The meaning to life and the universe is" |
|
|
``` |
|
|
|
|
|
### Server: |
|
|
```bash |
|
|
llama-server --hf-repo Triangle104/QwQ-32B-ArliAI-RpR-v3-Q3_K_M-GGUF --hf-file qwq-32b-arliai-rpr-v3-q3_k_m.gguf -c 2048 |
|
|
``` |
|
|
|
|
|
Note: You can also use this checkpoint directly through the [usage steps](https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#usage) listed in the Llama.cpp repo as well. |
|
|
|
|
|
Step 1: Clone llama.cpp from GitHub. |
|
|
``` |
|
|
git clone https://github.com/ggerganov/llama.cpp |
|
|
``` |
|
|
|
|
|
Step 2: Move into the llama.cpp folder and build it with `LLAMA_CURL=1` flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux). |
|
|
``` |
|
|
cd llama.cpp && LLAMA_CURL=1 make |
|
|
``` |
|
|
|
|
|
Step 3: Run inference through the main binary. |
|
|
``` |
|
|
./llama-cli --hf-repo Triangle104/QwQ-32B-ArliAI-RpR-v3-Q3_K_M-GGUF --hf-file qwq-32b-arliai-rpr-v3-q3_k_m.gguf -p "The meaning to life and the universe is" |
|
|
``` |
|
|
or |
|
|
``` |
|
|
./llama-server --hf-repo Triangle104/QwQ-32B-ArliAI-RpR-v3-Q3_K_M-GGUF --hf-file qwq-32b-arliai-rpr-v3-q3_k_m.gguf -c 2048 |
|
|
``` |
|
|
|
