SmolLM3-3B GGUF Quantized Models

📌 Original Model Reference

This repository provides quantized GGUF versions of the original model:

HuggingFaceTB/SmolLM3-3B

All quantizations are derived from this base model. Please refer to the original model card for architecture, training details, and upstream updates.

🧩 Model Description

This repository contains multiple quantized GGUF versions of the SmolLM3-3B model, converted for efficient inference on CPU and mobile devices.

Quantized by: Bastion-AI
BastionAI Website: https://bastionai.github.io/
Format: GGUF (for llama.cpp and compatible)

BastionAI empowers users with advanced AI capabilities that run entirely on your device. Learn more about our privacy-first, local AI solutions at bastionai.github.io.

🚀 How to Run Locally

You can use these GGUF files with:

Example command:

./main -m SmolLM3-3B.Q4_K_M.gguf -p "Explain gravity in simple terms."

📦 Quantized GGUF Downloads

Quant Type	Disk Size	Download Link	Details
Q2_K	1.2G	Download	Standard 2-bit quant
Q3_K	1.5G	Download	Standard 3-bit quant
Q3_K_S	1.3G	Download	Small 3-bit quant
Q3_K_M	1.5G	Download	Medium 3-bit quant
Q4_0	1.7G	Download	Standard 4-bit quant
Q4_K	1.8G	Download	Standard 4-bit quant
Q4_K_S	1.7G	Download	Small 4-bit quant
Q4_K_M	1.8G	Download	Medium 4-bit quant
Q5_0	2.0G	Download	Standard 5-bit quant
Q5_K	2.1G	Download	Standard 5-bit quant
Q5_K_S	2.0G	Download	Small 5-bit quant
Q5_K_M	2.1G	Download	Medium 5-bit quant
Q6_K	2.4G	Download	Standard 6-bit quant
Q8_0	3.0G	Download	Standard 8-bit quant

🛡️ About BastionAI

BastionAI empowers users with advanced AI capabilities that run entirely on your device. Experience the full potential of AI conversation, document processing, and voice interaction without compromising your privacy or security.

Website: https://bastionai.github.io/
Products: BastionChat, BastionSDK
Mission: Democratizing AI for everyone, privacy-first, open source, cross-platform.

📖 License

This model is distributed under the Apache 2.0 license, as per the original HuggingFaceTB/SmolLM3-3B.

🙏 Credits

Original model by HuggingFaceTB
GGUF conversion and quantization by Bastion-AI
Learn more about BastionAI: https://bastionai.github.io/

🔗 Links

📣 Community & Support

📜 Citation

If you use this model, please cite the original SmolLM3-3B and BastionAI as appropriate.

@misc{smollm3,
  title={SmolLM3-3B},
  author={HuggingFaceTB},
  year={2024},
  url={https://huggingface.co/HuggingFaceTB/SmolLM3-3B}
}

Bastion-AI
/

SmolLM3-3B-GGUF