---
base_model: unsloth/llama-3.2-1b-instruct-unsloth-bnb-4bit
tags:
- text-generation-inference
- transformers
- unsloth
- llama
- trl
license: apache-2.0
language:
- en
---

# Query Generation with LoRA Finetuning

This project fine-tunes a language model using supervised fine-tuning (SFT) and LoRA adapters to generate queries from documents. The model was trained on the [`prdev/qtack-gq-embeddings-unsupervised`](https://huggingface.co/datasets/prdev/qtack-gq-embeddings-unsupervised) dataset using an A100 GPU.

## Overview

- **Objective:**  
  The goal is to train a model that, given a document, generates a relevant query. Each training example is formatted with custom markers:
  - `<|document|>\n` precedes the document text.
  - `<|query|>\n` precedes the query text.
  - An EOS token is appended at the end to signal termination.

- **Text Chunking:**  
  For optimal performance, **chunk your text** into smaller, coherent pieces before providing it to the model. Long documents can lead the model to focus on specific details rather than the overall context.

- **Training Setup:**  
  The model is fine-tuned using the Unsloth framework with LoRA adapters, taking advantage of an A100 GPU for efficient training. See W&B loss curve here: https://wandb.ai/prdev/lora_model_training/panel/jp2r24xk7?nw=nwuserprdev



## Quick Usage

Below is an example code snippet to load the finetuned model and test it with a chunked document:

```python
from unsloth import FastLanguageModel
from transformers import TextStreamer

# Load the finetuned model and tokenizer from Hugging Face Hub.
model, tokenizer = FastLanguageModel.from_pretrained("prdev/query-gen", load_in_4bit=True)

# Enable faster inference if supported.
FastLanguageModel.for_inference(model)

# Example document chunk (ensure text is appropriately chunked).
document_chunk = (
    "liberal arts. 1. the academic course of instruction at a college intended to provide general knowledge "
    "and comprising the arts, humanities, natural sciences, and social sciences, as opposed to professional or technical subjects."
)

# Create the prompt using custom markers.
prompt = (
    "<|document|>\n" + document_chunk + "\n<|query|>\n"
)

# Tokenize the prompt.
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

# Set up a TextStreamer to view token-by-token generation.
streamer = TextStreamer(tokenizer, skip_prompt=True)

# Generate a query from the document.
_ = model.generate(
    input_ids=inputs["input_ids"],
    streamer=streamer,
    max_new_tokens=100,
    temperature=0.7,
    min_p=0.1,
    eos_token_id=tokenizer.eos_token_id,  # Ensures proper termination.
)
```

# Uploaded  model

- **Developed by:** prdev
- **License:** apache-2.0
- **Finetuned from model :** unsloth/llama-3.2-1b-instruct-unsloth-bnb-4bit

This llama model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.

[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)