qingy2024/GPT-OS3-Beta-8B-A3B

GPT OS3 Beta 8B A3B

• Developed by: qingy2024
• Base model: AmanPriyanshu/gpt-oss-8.4b-specialized-all-pruned-moe-only-11-experts

GPT OSS Small (OS3) is a project to create usable and intelligent language models based on pruned GPT-OSS-20B variants by AmanPriyanshu. These are post trained with LoRA on the qingy2024/GPT-OS3-Dataset-v1 dataset to revert some of the "brain damage" due to the expert pruning.

Beta, Step 4163, V1 Dataset. Can generate mostly coherent output. Will use as FFT/LoRA base for the v2 dataset. Chi didn't use the right chat template :/