|
--- |
|
license: apache-2.0 |
|
tags: |
|
- moe |
|
- frankenmoe |
|
- merge |
|
- mergekit |
|
- lazymergekit |
|
- cognitivecomputations/dolphin-2_6-phi-2 |
|
- rhysjones/phi-2-orange |
|
base_model: |
|
- cognitivecomputations/dolphin-2_6-phi-2 |
|
- rhysjones/phi-2-orange |
|
--- |
|
|
|
# PhiMiX-2x2B |
|
|
|
|
|
## Code is work in progress |
|
|
|
<p align="center"> |
|
<img src="https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F11201acc-4089-416d-921b-cbd71fbf8ddb_1024x1024.jpeg" width="500" class="center"/> |
|
</p> |
|
|
|
|
|
PhiMiX-2x2B is a Mixure of Experts (MoE) made with the following models using mergekit: |
|
* [cognitivecomputations/dolphin-2_6-phi-2](https://huggingface.co/cognitivecomputations/dolphin-2_6-phi-2) |
|
* [rhysjones/phi-2-orange](https://huggingface.co/rhysjones/phi-2-orange) |
|
|
|
|
|
## ©️ Credits |
|
* [mlabonne's phixtral](https://huggingface.co/mlabonne/phixtral-4x2_8) for the PhiConfig and inference code. |
|
* [mergekit](https://github.com/cg123/mergekit) code which I tweaked (you can find the PhiConfig [here](https://github.com/cg123/mergekit/blob/508348ae34be17ea0a95d0a288a6e34491a2558a/mergekit/architecture.py#L289)) |
|
by mainly adding the config in the `moe_mixtral.py` script from `mixtral` branch. |
|
|
|
|
|
## 🧩 Configuration |
|
|
|
```yaml |
|
base_model: rhysjones/phi-2-orange |
|
gate_mode: random |
|
dtype: float16 |
|
experts: |
|
- source_model: cognitivecomputations/dolphin-2_6-phi-2 |
|
positive_prompts: [""] |
|
- source_model: rhysjones/phi-2-orange |
|
positive_prompts: [""] |
|
``` |
|
|
|
## 💻 Usage |
|
|
|
```python |
|
!pip install -qU transformers bitsandbytes accelerate |
|
|
|
from transformers import AutoTokenizer |
|
import transformers |
|
import torch |
|
|
|
model = "paulilioaica/PhiMiX-2x2B_embed" |
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model) |
|
pipeline = transformers.pipeline( |
|
"text-generation", |
|
model=model, |
|
trust_remote_code=True, |
|
model_kwargs={"torch_dtype": torch.float16, "load_in_4bit": True,}, |
|
) |
|
|
|
prompt="How many continents are there?" |
|
input = f"Instruct: <prompt>\nOutput:" |
|
outputs = pipeline(input, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95) |
|
print(outputs[0]["generated_text"]) |
|
|
|
``` |
