README.md

---
license: apache-2.0
tags:
- pretrained
- base-model
language:
- en
- ko
- ja
pipeline_tag: text-generation
library_name: transformers
extra_gated_fields:
  Full Name: text
  Email: text
  Organization: text
---

<p align="center">
<picture>
  <img src="https://raw.githubusercontent.com/trillion-labs/.github/main/Tri-7B.png" alt="Tri-7B-Base", style="width: 80%;">
</picture>
</p>

# Tri-7B-Base

## Introduction

We present **Tri-7B-Base**, a foundation language model that serves as the pre-trained base for our Tri-7B model family. This model represents our commitment to efficient training while establishing a strong foundation for downstream fine-tuning and adaptation.

### Key Features
* **Foundation Architecture**: State-of-the-art transformer architecture optimized for efficiency
* **Multi-lingual Foundation**: Pre-trained on diverse data in Korean, English, and Japanese
* **Efficient Training**: Optimized training methodology for computational efficiency

### Model Specifications

#### Tri-7B-Base
- Type: Causal Language Model
- Training Stage: Pre-training
- Architecture: Transformer Decoder with RoPE, SwiGLU, RMSNorm
- Number of Parameters: 7.76B
- Number of Layers: 32
- Number of Attention Heads: 32
- Context Length: 4,096
- Vocab Size: 128,128

## Use Cases

As a base model, Tri-7B-Base is designed to serve as a foundation for various downstream applications:

- **Fine-tuning**: Adapt to specific domains or tasks
- **Instruction Tuning**: Create chat or assistant models
- **Domain Specialization**: Customize for specific industries or use cases
- **Research**: Explore model behaviors and capabilities
- **Language Generation**: General text completion and generation tasks

## Limitations

- **Base Model Nature**: This is a pre-trained base model without instruction tuning or alignment. For chat or assistant capabilities, consider fine-tuned variants.
- **Language Support**: The model is optimized for English, Korean, and Japanese. Usage with other languages may result in degraded performance.
- **Knowledge Cutoff**: The model's information is limited to data available up to February, 2025.
- **Generation Quality**: As a base model, outputs may require post-processing or filtering for production use cases.

## License
This model is licensed under the Apache License 2.0.

## Contact
For inquiries, please contact: [email protected]