maddes8cht/smallcloudai-Refact-1_6B-fim-gguf

I'm constantly enhancing these model descriptions to provide you with the most relevant and comprehensive information

Refact-1_6B-fim - GGUF

• Model creator: smallcloudai
• Original model: Refact-1_6B-fim

Refact seems to be an original model so far without any descendants. It was anounced on the refact.ai website and published on Huggingface.

About GGUF format

gguf is the current file format used by the ggml library. A growing list of Software is using it and can therefore use this model. The core project making use of the ggml library is the llama.cpp project by Georgi Gerganov

Quantization variants

There is a bunch of quantized files available to cater to your specific needs. Here's how to choose the best option for you:

Legacy quants

Q4_0, Q4_1, Q5_0, Q5_1 and Q8 are legacy quantization types. Nevertheless, they are fully supported, as there are several circumstances that cause certain model not to be compatible with the modern K-quants.

Note:

Now there's a new option to use K-quants even for previously 'incompatible' models, although this involves some fallback solution that makes them not real K-quants. More details can be found in affected model descriptions. (This mainly refers to Falcon 7b and Starcoder models)

K-quants

K-quants are designed with the idea that different levels of quantization in specific parts of the model can optimize performance, file size, and memory load. So, if possible, use K-quants. With a Q6_K, you'll likely find it challenging to discern a quality difference from the original model - ask your model two times the same question and you may encounter bigger quality differences.

---

Original Model Card:

Refact-1.6B

Finally, the model we started training with our blog post is ready 🎉

After fine-tuning on generated data, it beats Replit 3b, Stability Code 3b and many other models. It almost beats StarCoder ten times the size!

Model	Size	HumanEval pass@1	HumanEval pass@10
DeciCoder-1b	1b	19.1%
Refact-1.6-fim	1.6b	32.0%	53.0%
StableCode	3b	20.2%	33.8%
ReplitCode v1	3b	21.9%
CodeGen2.5-multi	7b	28.4%	47.5%
CodeLlama	7b	33.5%	59.6%
StarCoder	15b	33.6%

Likely, it's the best model for practical use in your IDE for code completion because it's smart and fast! You can start using it right now by downloading the Refact plugin. You can host the model yourself, too, using the open source docker container.

And it's multi-language (see MultiPL-HumanEval and other metrics below) and it works as a chat (see the section below).

It Works As a Chat

The primary application of this model is code completion (infill) in multiple programming languages. But it works as a chat quite well.

HumanEval results using instruction following (chat) format, against models specialized for chat only:

Model	Size	pass@1	pass@10
Refact-1.6-fim	1.6b	38.4%	55.6%
StableCode-instruct	3b	26.9%	36.2%
OctoGeeX	6b	44.7%
CodeLlama-instruct	7b	34.8%	64.3%
CodeGen2.5-instruct	7b	36.2%	60.87
CodeLlama-instruct	13b	42.7%	71.6%
StarChat-β	15b	33.5%
OctoCoder	15b	46.2%

Example

Fill-in-the-middle uses special tokens to identify the prefix/middle/suffix part of the input and output:

# pip install -q transformers
from transformers import AutoModelForCausalLM, AutoTokenizer

checkpoint = "smallcloudai/Refact-1_6B-fim"
device = "cuda" # for GPU usage or "cpu" for CPU usage

tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint, trust_remote_code=True).to(device)

prompt = '<fim_prefix>def print_hello_world():\n    """<fim_suffix>\n    print("Hello world!")<fim_middle>'

inputs = tokenizer.encode(prompt, return_tensors="pt").to(device)
outputs = model.generate(inputs, max_length=100, temperature=0.2)
print("-"*80)
print(tokenizer.decode(outputs[0]))

Chat Format

The same model works as chat (experimental).

prompt_template = "<empty_output>SYSTEM {system}\n" \
                  "<empty_output>USER {query}\n" \
                  "<empty_output>ASSISTANT"
prompt = prompt_template.format(system="You are a programming assistant",
                                query="How do I sort a list in Python?")

Architecture

As described in more detail in the blog post, we used:

• ALiBi based attention
• LayerNorm instead of RMSNorm
• Multi Query Attention

We also used LiON, flash attention, early dropout. It's not that innovative that you can't run it, in fact you can -- see an example below.

Pretraining

For the base model, we used our own dataset that contains code with permissive licenses only, and open text datasets. Filtering is the key to success of this model:

• We only used text in English
• Only topics related to computer science
• Applied heavy deduplication

The text to code proportion was 50:50, model trained for 1.2T tokens.

We don't release the base model, because its Fill-in-the-Middle (FIM) capability likes to repeat itself too much, so its practical use is limited. But if you still want it, write us a message on Discord.

Finetuning

We tested our hypothesis that chat data should boost base model performance in FIM and regular left-to-right code completion. We found that just 15% of open code instruction-following datasets, that we filtered for quality, improves almost all metrics.

Additionally, to improve FIM, we observed common failure modes, and prepared a synthetic dataset based on The Stack dedup v1.1 to address them.

There is a distribution shift between typical code on the internet, and the code you write in your IDE. The former is likely finished, so the model tries to come up with a suggestion that makes the code complete. You are likely to have half-written code as you work on it, there is no single addition that can repair it fully.

In practice, model needs to have a tendency to stop after a couple of lines are added, and sometimes don't write anything at all. We found that just giving it empty completions, single line completions, multiline completions that end with a smaller text indent or at least a newline -- makes it much more usable. This data was used as the rest 85% of the finetune dataset.

The final model is the result of several attempts to make it work as good as possible for code completion, and to perform well on a wide range of metrics. The best attempt took 40B tokens.

Limitations and Bias

The Refact-1.6B model was trained on text in English. But it has seen a lot more languages in code comments. Its performance on non-English languages is lower, for sure.

Model Stats

• Architecture: LLAMA-like model with multi-query attention
• Objectives Fill-in-the-Middle, Chat
• Tokens context: 4096
• Pretraining tokens: 1.2T
• Finetuning tokens: 40B
• Precision: bfloat16
• GPUs 64 NVidia A5000
• Training time 28 days

License

The model is licensed under the BigScience OpenRAIL-M v1 license agreement

Citation

If you are using this model, please give a link to this page.

End of original Model File

Please consider to support my work

Coming Soon: I'm in the process of launching a sponsorship/crowdfunding campaign for my work. I'm evaluating Kickstarter, Patreon, or the new GitHub Sponsors platform, and I am hoping for some support and contribution to the continued availability of these kind of models. Your support will enable me to provide even more valuable resources and maintain the models you rely on. Your patience and ongoing support are greatly appreciated as I work to make this page an even more valuable resource for the community.