Text Generation
Transformers
Safetensors
English
glm4
conversational
AuriAetherwiing commited on
Commit
a66e498
·
verified ·
1 Parent(s): a8a1326

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +17 -0
README.md CHANGED
@@ -46,6 +46,23 @@ Min-P - 0.1
46
  Repetition Penalty - 1.03
47
  ```
48
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
49
  **Training config**
50
  <details><summary>See Axolotl config</summary>
51
 
 
46
  Repetition Penalty - 1.03
47
  ```
48
 
49
+ **Running on KoboldCPP and other backends**
50
+
51
+ To run GGUFs correctly, you need the most recent version of KoboldCPP, and to pass `--overridekv glm4.rope.dimension_count=int:64` to the CLI command or put `glm4.rope.dimension_count=int:64` into overridekv box in the GUI (under the Tokens tab at the very bottom).
52
+
53
+ Thanks to DaringDuck and tofumagnate for info how to apply this fix.
54
+
55
+ To run this model on vLLM, you'll need to build it from source from the git repo, GLM4 support haven't reached release yet. ExLLaMAv2 and v3 don't support GLM4 arch at the moment
56
+
57
+
58
+ **Special Thanks**
59
+
60
+ Once again, huge kudos to OwenArli for providing compute and helping with tuning along the way!
61
+ Big thanks to Artus for providing free inference for pre-release showcase of this model!
62
+ And big thanks to BeaverAI community for giving feedback and helping to figure out optimal settings!
63
+
64
+ ---
65
+
66
  **Training config**
67
  <details><summary>See Axolotl config</summary>
68