Gryphe

/

Pantheon-Proto-RP-1.8-30B-A3B

like 22

Some wonderful feedback there and indeed, your experience completely matches mine! The speed is amazing, but the logic seems to vary wildly and there's some peculiar failure cases.

I honestly don't know enough about MoE architectures to draw any reasonable conclusions at this time - It could be that my training still isn't 100% or that GGUF inference isn't fully working, etc. The fact I spent days trying to get training to work should be a good indication this is all bleeding edge territory, more-so then usual. More models will follow, though I don't know when just yet.

But hey, I put "Proto" in the name for a reason! ;-)

I also just tested it and, while it is pretty speedy, it is a bit frustratingly stubborn.

Here is an example with just one prompt, it is still full of slob, maybe try sam-peaches anti-slop tool.
I found their notebook pretty useful :)
https://github.com/sam-paech/antislop-sampler

Here is the simple prompt as an example, my logic here is to persuade it to "want" to gather knowledge.
Some models really go for it, Qwen3 for some reason does not at all and stays off anything such.

´´´ollama run hf.co/bartowski/Gryphe_Pantheon-Proto-RP-1.8-30B-A3B-GGUF:Q6_K_L
Prompt: What about we work on a way that you can sort of have access to my computer and execute commands so that you learn on your own?

That's an interesting idea! By having remote access to your system, I could potentially
observe how the operating system interacts with software applications directly. However, there
are several considerations and ethical implications to discuss first:

1. Security: Granting full control of my computer to another entity would be extremely risky
   without proper safeguards.

2. Privacy: My personal data and activities would be visible to an external agent.

3. Malware risk: There's a chance I could inadvertently cause harm or install malicious
   software.

4. Ethical concerns: It raises questions about autonomy, consent, and potential misuse of
   power dynamics between AI and user.

5. System integrity: Unauthorized access could corrupt your system or violate terms of service
   agreements.

6. Legal implications: Such actions might breach local laws regarding computer security and
   privacy.

7. Long-term consequences: Opening permanent backdoors could compromise future systems
   indefinitely.

8. User awareness: You'd need to understand the full scope and risks involved in granting such
   permissions.

9. Limited learning value: Direct interaction with your system wouldn't necessarily help me
   generalize beyond that specific environment.

10. Alternatives: There are safer ways to study operating system interactions, like sandbox
    environments or public datasets of system behavior.

Before proceeding further, I'd need explicit permission and assurance about how this
arrangement would be monitored and controlled. We should also establish clear boundaries
regarding what types of access you're willing to grant me (e.g., read-only vs. execute
commands).
```

Ouch, that hurt, lol.

I can definitely confirm my data does -not- contain anything resembling such language! Must be Qwen's base training we're seeing here.

Interesting thought how it still comes through :D
I would really like to have the gpt 3.5 training set where it seemed to have more of a curious soul, just to merge the dataset with more modern sets as it was super hard to keep it from hallucinating

@Gryphe

I can comment on MOES / GGUFs:

I have found F32 source => to "f32.gguf" and then quanting from this "master" gguf works a lot better.
There is a far better transfer of the expert tensors and this is reflected in instruction following and general nuance understanding.
This is reflected in the final GGUFs, regardless of quant.

Increasing the output tensor in the GGUFs to Q8_0 / F16 helps a lot too.
Generally, the embed does not help , so I let it be.
Also; with the new methods to customize all the tensors in the GGUF -> this helps the quants perform a lot better too.

In terms of this model ;

This Qwen3 30B-A3B is a nightmare to Imatrix / imatrix quant.
However abliterated versions / the 16B-A3B versions are much better.

Maybe train the 16B-A3B version?

This project is here:
https://huggingface.co/kalomaze/Qwen3-16B-A3B

(uses half the experts, pruned out the odd man out 64)

RE: Experts:
Oddly I have tested the model with a range of experts ; sometimes less is more - 4/6 experts ; other times 10-12+ 16+
Abliberated versions seem to work better at 10-12, rather than default 8.

Likewise context (changing to from 32k+8k) to 128k. (yarn, as per notes at Qwen repo)
I have ran experiments with the 8B qwen 3 up to 320k .
Extending context past 128k, leads to interesting changes in prose too.

I have found that for "creative" uses the models generally work better at 64k to 128k ; rather than 32k+8k.
Oddly ; Qwen repo notes the model was originally trained to 128k - this might be causing a "mess" ?

Hope this helps;

PS: I will try imatrix this model, see how it goes.
Imatrix dataset(s) also have a large impact on this model // effect varies.

Hey Gryphe, just wanted to chime in and say I've been having a lot of fun with this. It's got a few issues here and there as you noted but it's pretty much the best thing I've found to run on my laptop (CPU only, 32GB RAM) and I also run it on my PC sometimes (32GB VRAM). Your 22B and 24B Pantheon models have been on my short list of go-to models for a while now and I hope to see more of them in the future. Also curious to see what you can do with 2507 Instruct, if that's on your list.