README.md

metadata

license: apache-2.0
tags:
  - code-generation
  - t5
  - lora
  - peft
  - transformers
library_name: peft
base_model: t5-small
datasets: nvidia/OpenCodeReasoning
model-index:
  - name: T5-Small with LoRA on OpenCodeReasoning
    results:
      - task:
          type: text2text-generation
          name: Code Generation
        dataset:
          name: OpenCodeReasoning
          type: nvidia/OpenCodeReasoning
        metrics:
          - name: Loss
            type: loss
            value: 4.69

T5-Small with LoRA on OpenCodeReasoning

This is a LoRA fine-tuned version of T5-small on a subset of NVIDIA's OpenCodeReasoning dataset using PEFT. Improved version to be uploaded soon.

Loss Curve

Step	Train Loss	Val Loss
50	8.63	8.17
100	6.04	5.35
150	5.31	4.90
200	5.19	4.71
250	4.94	4.59
300	4.95	4.51
350	4.79	4.46
400	4.89	4.42
450	4.69	4.40

Final Train Loss: 4.69 Final Eval Loss: 4.40

Notes

Trained on subset of OpenCodeReasoning due to Colab memory limits

Use PeftModel with t5-small base

Metrics used: Loss (BLEU skipped due to output structure)

License

Apache 2.0

Example Usage

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
from peft import PeftModel, PeftConfig

config = PeftConfig.from_pretrained("ShahzebKhoso/t5-small-opencode-lora")
base_model = AutoModelForSeq2SeqLM.from_pretrained(config.base_model_name_or_path)
model = PeftModel.from_pretrained(base_model, "ShahzebKhoso/t5-small-opencode-lora")
tokenizer = AutoTokenizer.from_pretrained("ShahzebKhoso/t5-small-opencode-lora")

inputs = tokenizer("generate code: write a function to reverse a string", return_tensors="pt")
outputs = model.generate(**inputs)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
'''