Update README.md

README.md

@@ -44,4 +44,37 @@ base_model: akameswa/mistral-7b-instruct-v0.2-bnb-16bit
 parameters:
   normalize: true
 dtype: float16
-```
+```
+
+## Inference
+```python
+from unsloth import FastLanguageModel
+import torch
+
+model, tokenizer = FastLanguageModel.from_pretrained(
+    model_name = "akameswa/mistral-7b-instruct-code-ties",
+    max_seq_length = 2048,
+)
+
+xlcost_prompt = """Below is a description of a programming task. Write a response that appropriately completes the task based on the given description.
+
+### Description:
+{}
+
+### Code:
+{}"""
+
+FastLanguageModel.for_inference(model)
+inputs = tokenizer(
+[
+    xlcost_prompt.format(
+        "Continue the fibonnaci sequence.",
+        "",
+    )
+], return_tensors = "pt").to("cuda")
+
+outputs = model.generate(**inputs, max_new_tokens = 64, use_cache = True)
+tokenizer.batch_decode(outputs)
+```
+
+* [Link to inference notebook](https://github.com/akameswa/CodeGenerationMoE/blob/main/code/inference_merge.ipynb)