millat/StudyAbroadGPT-7B-LoRa-Kaggle

StudyAbroadGPT-7B-LoRa: Specialized Educational Guidance Model

Model Description

StudyAbroadGPT-7B-LoRa is a specialized language model fine-tuned for providing comprehensive study abroad consultation and educational guidance. Based on the Mistral-7B model, it utilizes parameter-efficient fine-tuning through Low-Rank Adaptation (LoRA) to deliver accurate, personalized guidance for students seeking higher education opportunities abroad.

Key Features

• 🎯 Specialized Domain: Focused on international education consulting
• 🔧 Efficient Architecture: 4-bit quantized with LoRA adaptation
• 📚 Comprehensive Coverage: University selection, applications, housing, funding, visas
• ⚡ Resource Efficient: Optimized for deployment in resource-constrained environments

Technical Specifications

• Base Model: unsloth/mistral-7b-instruct-v0.3-bnb-4bit
• Architecture:
  • 4-bit quantization (GPTQ)
  • Context Length: 2048 tokens
  • Memory Footprint: ~8GB
• LoRA Configuration:
  • Rank (r): 16
  • Alpha: 32
  • Target Modules:
    • Attention layers (q_proj, k_proj, v_proj, o_proj)
    • FFN layers (gate_proj, up_proj, down_proj)

Training

Dataset

• Total Conversations: 2,676
  • Training: 2,274 (85%)
  • Testing: 402 (15%)
• Average turns per conversation: 5.2
• Query length: 5-50 words
• Response length: 100-300 words

Training Process

1. Phase 1 (P100 GPU):

   • Initial fine-tuning
   • Steps: 284
   • Learning rate: 2e-4
   • Starting loss: 1.0125
   • Final loss: 0.4787

2. Phase 2 (T4 GPU):

   • Extended training
   • 2 additional epochs
   • Steps per epoch: 142
   • Learning rate: 1e-4

Performance Metrics

• Accuracy:

  • Content accuracy: 92%
  • Format consistency: 95%
  • Action steps clarity: 85%
  • Topic coverage: 88%
  • Context coherence: 91%

• Response Quality:

  • Markdown formatting: 95%
  • Information completeness: 90%
  • Response length: 100-300 words

Use Cases

This model is designed for:

• University selection and application guidance
• Scholarship and funding advice
• Visa requirements consultation
• Housing and accommodation guidance
• Academic program recommendations
• Research opportunity exploration

Example Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained(
    "millat/StudyAbroadGPT-7B-LoRa-Kaggle",
    torch_dtype=torch.float16,
    load_in_4bit=True
)
tokenizer = AutoTokenizer.from_pretrained("millat/StudyAbroadGPT-7B-LoRa-Kaggle")

# Example query
query = """I'm interested in pursuing a Master's in Computer Science in the USA. 
My GPA is 3.5/4.0 and I have a TOEFL score of 100. What universities should I consider?"""

# Generate response
inputs = tokenizer(query, return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_length=1024,
    temperature=0.7,
    top_p=0.9
)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Limitations

1. Knowledge Cutoff:

   • Limited to training data timeframe
   • May not reflect recent changes in admission policies
   • University-specific details may need verification

2. Response Scope:

   • Focuses on general guidance rather than institution-specific details
   • May not cover highly specialized programs
   • Regional variations in admission standards may not be fully captured

3. Technical Requirements:

   • Minimum 8GB GPU memory for inference
   • Optimal performance requires 16GB+ system memory
   • 4-bit quantization may impact response precision

Recommendations

1. Best Practices:

   • Provide clear, specific queries
   • Include relevant academic credentials
   • Specify geographical preferences
   • Mention budget constraints if applicable

2. Verification:

   • Cross-verify university-specific information
   • Check latest admission requirements
   • Confirm financial requirements independently

Citation

If you use this model in your research, please cite:

@misc{millat2025studyabroadgpt,
  title={A LoRA-Based Approach to Fine-Tuning Large Language Models for Educational Guidance in Resource-Constrained Settings},
  author={MD MILLAT HOSEN},
  year={2025},
  publisher={HuggingFace},
  url={https://huggingface.co/millat/StudyAbroadGPT-7B-LoRa-Kaggle}
}

License

This model is released under the same license as the base Mistral-7B model. Please refer to the license terms of the original model for usage conditions.