Update README.md

README.md

@@ -1,6 +1,27 @@
 ---
+license: cc-by-nc-sa-4.0
+datasets:
+- camel-ai/code
+- ehartford/wizard_vicuna_70k_unfiltered
+- anon8231489123/ShareGPT_Vicuna_unfiltered
+- teknium1/GPTeacher/roleplay-instruct-v2-final
+- teknium1/GPTeacher/codegen-isntruct
+- timdettmers/openassistant-guanaco
+- camel-ai/math
+- project-baize/baize-chatbot/medical_chat_data
+- project-baize/baize-chatbot/quora_chat_data
+- project-baize/baize-chatbot/stackoverflow_chat_data
+- camel-ai/biology
+- camel-ai/chemistry
+- camel-ai/ai_society
+- jondurbin/airoboros-gpt4-1.2
+- LongConversations
+- camel-ai/physics
+tags:
+- Composer
+- MosaicML
+- llm-foundry
 inference: false
-license: other
 ---
 
 <!-- header start -->
@@ -17,81 +38,69 @@ license: other
 </div>
 <!-- header end -->
 
-# MosaicML's MPT-30B-chat GGML
+# MosaicML's MPT-30B-Chat GGML
 
-These files are GGML format model files for [MosaicML's MPT-30B-chat](https://huggingface.co/mosaicml/mpt-30b-chat).
+These files are GGML format model files for [MosaicML's MPT-30B-Chat](https://huggingface.co/mosaicml/mpt-30b-chat).
 
-GGML files are for CPU + GPU inference using [llama.cpp](https://github.com/ggerganov/llama.cpp) and libraries and UIs which support this format, such as:
-* [text-generation-webui](https://github.com/oobabooga/text-generation-webui)
-* [KoboldCpp](https://github.com/LostRuins/koboldcpp)
-* [LoLLMS Web UI](https://github.com/ParisNeo/lollms-webui)
-* [llama-cpp-python](https://github.com/abetlen/llama-cpp-python)
-* [ctransformers](https://github.com/marella/ctransformers)
+Please note that these GGMLs are **not compatible with llama.cpp, or currently with text-generation-webui**. Please see below for a list of tools known to work with these model files.
+
+[KoboldCpp](https://github.com/LostRuins/koboldcpp_ just added GPU accelerated (OpenCL) support for MPT models, so that is the client I recommend using for these models.
 
 ## Repositories available
 
-* [4-bit GPTQ models for GPU inference](https://huggingface.co/none)
 * [2, 3, 4, 5, 6 and 8-bit GGML models for CPU+GPU inference](https://huggingface.co/TheBloke/mpt-30B-chat-GGML)
 * [Unquantised fp16 model in pytorch format, for GPU inference and for further conversions](https://huggingface.co/mosaicml/mpt-30b-chat)
 
-<!-- compatibility_ggml start -->
-## Compatibility
+## Prompt template
+
+Just type the prompt!
+```
+prompt
+```
 
-### Original llama.cpp quant methods: `q4_0, q4_1, q5_0, q5_1, q8_0`
+## A note regarding context length: 8K
 
-I have quantized these 'original' quantisation methods using an older version of llama.cpp so that they remain compatible with llama.cpp as of May 19th, commit `2d5db48`.
+The base model has an 8K context length. It is not yet confirmed if the 8K context of this model works with the quantised files.
 
-These are guaranteed to be compatbile with any UIs, tools and libraries released since late May.
+If it does, [KoboldCpp](https://github.com/LostRuins/koboldcpp) supports 8K context if you manually it to 8K by adjusting the text box above the slider:
+![.](https://i.imgur.com/tEbpeJq.png)
 
-### New k-quant methods: `q2_K, q3_K_S, q3_K_M, q3_K_L, q4_K_S, q4_K_M, q5_K_S, q6_K`
+It is currently unknown as to whether it is compatible with other clients.
 
-These new quantisation methods are compatible with llama.cpp as of June 6th, commit `2d43387`.
+If you have feedback on this, please let me know.
 
-They are now also compatible with recent releases of text-generation-webui, KoboldCpp, llama-cpp-python and ctransformers. Other tools and libraries may or may not be compatible - check their documentation if in doubt.
+<!-- compatibility_ggml start -->
+## Compatibilty
+
+These files are **not** compatible with text-generation-webui, llama.cpp, or llama-cpp-python.
+
+Currently they can be used with:
+* KoboldCpp, a powerful inference engine based on llama.cpp, with good UI and GPU accelerated support for MPT models: [KoboldCpp](https://github.com/LostRuins/koboldcpp)
+* The ctransformers Python library, which includes LangChain support: [ctransformers](https://github.com/marella/ctransformers)
+* The LoLLMS Web UI which uses ctransformers: [LoLLMS Web UI](https://github.com/ParisNeo/lollms-webui)
+* [rustformers' llm](https://github.com/rustformers/llm)
+* The example `mpt` binary provided with [ggml](https://github.com/ggerganov/ggml)
 
-## Explanation of the new k-quant methods
+As other options become available I will endeavour to update them here (do let me know in the Community tab if I've missed something!)
 
-The new methods available are:
-* GGML_TYPE_Q2_K - "type-1" 2-bit quantization in super-blocks containing 16 blocks, each block having 16 weight. Block scales and mins are quantized with 4 bits. This ends up effectively using 2.5625 bits per weight (bpw)
-* GGML_TYPE_Q3_K - "type-0" 3-bit quantization in super-blocks containing 16 blocks, each block having 16 weights. Scales are quantized with 6 bits. This end up using 3.4375 bpw.
-* GGML_TYPE_Q4_K - "type-1" 4-bit quantization in super-blocks containing 8 blocks, each block having 32 weights. Scales and mins are quantized with 6 bits. This ends up using 4.5 bpw.
-* GGML_TYPE_Q5_K - "type-1" 5-bit quantization. Same super-block structure as GGML_TYPE_Q4_K resulting in 5.5 bpw
-* GGML_TYPE_Q6_K - "type-0" 6-bit quantization. Super-blocks with 16 blocks, each block having 16 weights. Scales are quantized with 8 bits. This ends up using 6.5625 bpw
-* GGML_TYPE_Q8_K - "type-0" 8-bit quantization. Only used for quantizing intermediate results. The difference to the existing Q8_0 is that the block size is 256. All 2-6 bit dot products are implemented for this quantization type.
+## Tutorial for using LoLLMS Web UI
+
+* [Text tutorial, written by **Lucas3DCG**](https://huggingface.co/TheBloke/MPT-7B-Storywriter-GGML/discussions/2#6475d914e9b57ce0caa68888)
+* [Video tutorial, by LoLLMS Web UI's author **ParisNeo**](https://www.youtube.com/watch?v=ds_U0TDzbzI)
 
-Refer to the Provided Files table below to see what files use which methods, and how.
 <!-- compatibility_ggml end -->
 
 ## Provided files
 | Name | Quant method | Bits | Size | Max RAM required | Use case |
 | ---- | ---- | ---- | ---- | ---- | ----- |
-| mpt-30b-chat.ggmlv0.q4_0.bin | q4_0 | 4 | 16.85 GB | 19.35 GB | Original llama.cpp quant method, 4-bit. |
-| mpt-30b-chat.ggmlv0.q4_1.bin | q4_1 | 4 | 18.73 GB | 21.23 GB | Original llama.cpp quant method, 4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models. |
-| mpt-30b-chat.ggmlv0.q5_0.bin | q5_0 | 5 | 20.60 GB | 23.10 GB | Original llama.cpp quant method, 5-bit. Higher accuracy, higher resource usage and slower inference. |
-| mpt-30b-chat.ggmlv0.q5_1.bin | q5_1 | 5 | 22.47 GB | 24.97 GB | Original llama.cpp quant method, 5-bit. Even higher accuracy, resource usage and slower inference. |
-| mpt-30b-chat.ggmlv0.q8_0.bin | q8_0 | 8 | 31.83 GB | 34.33 GB | Original llama.cpp quant method, 8-bit. Almost indistinguishable from float16. High resource use and slow. Not recommended for most users. |
+| mpt-30b-chat.ggmlv0.q4_0.bin | q4_0 | 4 | 16.85 GB | 19.35 GB | 4-bit. |
+| mpt-30b-chat.ggmlv0.q4_1.bin | q4_1 | 4 | 18.73 GB | 21.23 GB | 4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models. |
+| mpt-30b-chat.ggmlv0.q5_0.bin | q5_0 | 5 | 20.60 GB | 23.10 GB | 5-bit. Higher accuracy, higher resource usage and slower inference. |
+| mpt-30b-chat.ggmlv0.q5_1.bin | q5_1 | 5 | 22.47 GB | 24.97 GB | 5-bit. Even higher accuracy, resource usage and slower inference. |
+| mpt-30b-chat.ggmlv0.q8_0.bin | q8_0 | 8 | 31.83 GB | 34.33 GB | 8-bit. Almost indistinguishable from float16. High resource use and slow. Not recommended for most users. |
 
 **Note**: the above RAM figures assume no GPU offloading. If layers are offloaded to the GPU, this will reduce RAM usage and use VRAM instead.
 
-## How to run in `llama.cpp`
-
-I use the following command line; adjust for your tastes and needs:
-
-```
-./main -t 10 -ngl 32 -m mpt-30b-chat.ggmlv3.q5_0.bin --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "### Instruction: Write a story about llamas\n### Response:"
-```
-If you're able to use full GPU offloading, you should use `-t 1` to get best performance.
-
-If not able to fully offload to GPU, you should use more cores. Change `-t 10` to the number of physical CPU cores you have, or a lower number depending on what gives best performance.
-
-Change `-ngl 32` to the number of layers to offload to GPU. Remove it if you don't have GPU acceleration.
-
-If you want to have a chat-style conversation, replace the `-p <PROMPT>` argument with `-i -ins`
-
-## How to run in `text-generation-webui`
-
-Further instructions here: [text-generation-webui/docs/llama.cpp-models.md](https://github.com/oobabooga/text-generation-webui/blob/main/docs/llama.cpp-models.md).
-
 <!-- footer start -->
 ## Discord