File size: 3,144 Bytes
d98ec7e 88dd236 d98ec7e |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 |
---
base_model: SicariusSicariiStuff/Nano_Imp_1B
language:
- en
license: llama3.2
tags:
- llama-cpp
- gguf-my-repo
---
# Triangle104/Nano_Imp_1B-Q4_K_M-GGUF
This model was converted to GGUF format from [`SicariusSicariiStuff/Nano_Imp_1B`](https://huggingface.co/SicariusSicariiStuff/Nano_Imp_1B) using llama.cpp via the ggml.ai's [GGUF-my-repo](https://huggingface.co/spaces/ggml-org/gguf-my-repo) space.
Refer to the [original model card](https://huggingface.co/SicariusSicariiStuff/Nano_Imp_1B) for more details on the model.
---
It's the 10th of May, 2025—lots of progress is being made in the world of AI (DeepSeek, Qwen, etc...)—but still, there has yet to be a fully coherent 1B RP model. Why?
Well, at 1B size, the mere fact a model is even
coherent is some kind of a marvel—and getting it to roleplay feels like
you're asking too much from 1B parameters. Making very small yet smart
models is quite hard, making one that does RP is exceedingly hard. I
should know.
I've made the world's first 3B roleplay model—Impish_LLAMA_3B—and I thought that this was the absolute minimum size for coherency and RP capabilities. I was wrong.
One of my stated goals was to make AI accessible and available for everyone—but not everyone could run 13B or even 8B models. Some people only have mid-tier phones, should they be left behind?
A growing sentiment often says something along the lines of:
If your waifu runs on someone else's hardware—then she's not your waifu.
I'm not an expert in waifu culture, but I do agree that people should
be able to run models locally, without their data (knowingly or
unknowingly) being used for X or Y.
I thought my goal of making a roleplay model that everyone could run
would only be realized sometime in the future—when mid-tier phones got
the equivalent of a high-end Snapdragon chipset. Again I was wrong, as
this changes today.
---
## Use with llama.cpp
Install llama.cpp through brew (works on Mac and Linux)
```bash
brew install llama.cpp
```
Invoke the llama.cpp server or the CLI.
### CLI:
```bash
llama-cli --hf-repo Triangle104/Nano_Imp_1B-Q4_K_M-GGUF --hf-file nano_imp_1b-q4_k_m.gguf -p "The meaning to life and the universe is"
```
### Server:
```bash
llama-server --hf-repo Triangle104/Nano_Imp_1B-Q4_K_M-GGUF --hf-file nano_imp_1b-q4_k_m.gguf -c 2048
```
Note: You can also use this checkpoint directly through the [usage steps](https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#usage) listed in the Llama.cpp repo as well.
Step 1: Clone llama.cpp from GitHub.
```
git clone https://github.com/ggerganov/llama.cpp
```
Step 2: Move into the llama.cpp folder and build it with `LLAMA_CURL=1` flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux).
```
cd llama.cpp && LLAMA_CURL=1 make
```
Step 3: Run inference through the main binary.
```
./llama-cli --hf-repo Triangle104/Nano_Imp_1B-Q4_K_M-GGUF --hf-file nano_imp_1b-q4_k_m.gguf -p "The meaning to life and the universe is"
```
or
```
./llama-server --hf-repo Triangle104/Nano_Imp_1B-Q4_K_M-GGUF --hf-file nano_imp_1b-q4_k_m.gguf -c 2048
```
|