Hello Community! 
text
Hello Community! 
new_line
I am experimenting with q5 quantization on 
text
I am experimenting with q5 quantization on 
mlx==0.26.2
`mlx==0.26.2`
inline_code
 and it is so promising! Our beloved (
text
 and it is so promising! Our beloved (
https://huggingface.co/CohereLabs/c4ai-command-a-03-2025
CohereLabs/c4ai-command-a-03-2025
model
resource
https://huggingface.co/CohereLabs/c4ai-command-a-03-2025
)  the model we use for our medical-nlp-use : 200gb+ bfloat16 vs. ~70gb q5 mlx with perfect quality (
text
)  the model we use for our medical-nlp-use : 200gb+ bfloat16 vs. ~70gb q5 mlx with perfect quality (
https://huggingface.co/LibraxisAI/c4ai-command-a-03-2025-q5-mlx
LibraxisAI/c4ai-command-a-03-2025-q5-mlx
model
resource
https://huggingface.co/LibraxisAI/c4ai-command-a-03-2025-q5-mlx
)
text
)
new_line
new_line
The q5 and mixed quants came up with the release of 
text
The q5 and mixed quants came up with the release of 
mlx>=0.26.0
`mlx>=0.26.0`
inline_code
. This quant is not yet supported by 
text
. This quant is not yet supported by 
lmstudio
`lmstudio`
inline_code
 but we didn't check the Ollama compatibility. For inference  at this moment we us native 
text
 but we didn't check the Ollama compatibility. For inference  at this moment we us native 
mlx_lm.chat
`mlx_lm.chat`
inline_code
 or 
text
 or 
mlx_lm.server
`mlx_lm.server`
inline_code
, which works perfectly with python 
text
, which works perfectly with python 
mx.generate
`mx.generate`
inline_code
 pipeline.  Repository consists of complete safetensors and config files **plus** we put there also our everyday command-generation scripts - 
text
 pipeline.  Repository consists of complete safetensors and config files **plus** we put there also our everyday command-generation scripts - 
convert-to-mlx.sh
`convert-to-mlx.sh`
inline_code
 for 
text
 for 
mlx_lm.convert
`mlx_lm.convert`
inline_code
 with variable parameters customization, and 
text
 with variable parameters customization, and 
mlx-serve.sh
`mlx-serve.sh`
inline_code
 for easy server building. Enjoy
text
 for easy server building. Enjoy
new_line
new_line
@lmstudio-community
mention
lmstudio-community
text
@mlx-community
mention
mlx-community
new_line
@bartowski
mention
bartowski
text
Hello Community! 
I am experimenting with q5 quantization on `mlx==0.26.2` and it is so promising! Our beloved (https://huggingface.co/CohereLabs/c4ai-command-a-03-2025)  the model we use for our medical-nlp-use : 200gb+ bfloat16 vs. ~70gb q5 mlx with perfect quality (https://huggingface.co/LibraxisAI/c4ai-command-a-03-2025-q5-mlx)

The q5 and mixed quants came up with the release of `mlx>=0.26.0`. This quant is not yet supported by `lmstudio` but we didn't check the Ollama compatibility. For inference  at this moment we us native `mlx_lm.chat` or `mlx_lm.server`, which works perfectly with python `mx.generate` pipeline.  Repository consists of complete safetensors and config files **plus** we put there also our everyday command-generation scripts - `convert-to-mlx.sh` for `mlx_lm.convert` with variable parameters customization, and `mlx-serve.sh` for easy server building. Enjoy

@lmstudio-community @mlx-community
@bartowski 
🔥
John6666
6640bbd0220cfa8cbfdce080
https://cdn-avatars.huggingface.co/v1/production/uploads/6640bbd0220cfa8cbfdce080/wiAHUu5ewawyipNs0YFBR.png
John Smith
John6666
user
6744f37a84dccb3a7d3ddcc0
https://cdn-avatars.huggingface.co/v1/production/uploads/6744f37a84dccb3a7d3ddcc0/5KkDchusnZXgAo-P9RbMl.png
Maciej Gad
div0-space
user