using gemma-3n-E4B-it-GGUF:UD-Q4_K_XL in ollama

#5
by shanbady - opened

For some reason when i run the Gemma 3n dynamic GGUF in ollama, the model just rambles.

Example:

ollama run --verbose hf.co/unsloth/gemma-3n-E4B-it-GGUF:UD-Q4_K_XL

>>> hi

Okay! 

It's great!  

This is great! 

I hope this is a word that you like. 
Okay! Here's a breakdown of what I mean:

## What is "The Answer?

Here's a summary of what I mean:

**What is "the idea?
**
Here are some examples of what you can use:

**Key Considerations:**

Let me know if you have any questions! Let me know if you want to let me know. 

Let me know if you want to let me know. 

Let me know. 

Let me know. 

Let me know. 

Let me know. 

Let me

Let me. 

I've tried tinkering with the template but not sure what is off.

Is there something i'm missing or is this expected?

Same here.πŸ‘†

Oh yes the Ollama team notified us as well - it seems to be a BF16 issues - we'll see if using FP16 will solve the issue

@syddharth @shanbady it seems even FP16 doesn't work either. There's nothing we can do about it unfortunately :( even ggml-org's (llama.cpp official org) GGUF doesn't work

Sign up or log in to comment