safetensors llama region:us
---
{}
---
## T-Llama-3-8B (October 2024)
*Korean Continual Pre-trained Model based on Meta-Llama-3-8B*

### Model Description
- **Developed by:** TmaxAI
- **Shared by:** MinJae Kim (mjk00098@gmail.com)
- **License:** cc-by-nc-nd-4.0
- **Data:** 64GB of Korean Corpus (Trained About 18.8B Tokens)
- **Sequence Length:** 8k
- **Resource:** H100 80GB x 8

### Accelerating
To optimize performance and handle large-scale Korean datasets efficiently, we integrated cutting-edge optimization techniques:
- **DeepSpeed:** Advanced distributed training framework for memory and computational efficiency.
- **Gradient Checkpointing:** Reduces memory usage during backpropagation by saving only necessary states.
- **Flash_Attention_2:** Significantly accelerates attention mechanisms with reduced memory overhead.
- **BF16 Mixed Precision:** Ensures fast computations with high numerical stability.

### Evaluation (In Progress)
- **Translation (MMT)**  
  - *Fine-tuned with PEFT on a multilingual translation task by freezing PLMs, and evaluated using xCOMET scores.*  
  - **Our models** significantly *outperformed* the original PLM and achieved performance *comparable to* leading publicly available **Korean continuous pretraining models**.

<table>
  <thead>
    <tr>
      <th class="plm" rowspan="2">Llama-3-8B</th>
      <th class="direction" colspan="2">EN -> KO</th>
      <th class="direction" colspan="2">KO -> EN</th>
      <th class="direction" colspan="2">JA -> KO</th>
      <th class="direction" colspan="2">KO -> JA</th>
      <th class="direction" colspan="2">ZH -> KO</th>
      <th class="direction" colspan="2">KO -> ZH</th>
    </tr>
    <tr>
      <th class="data-source">FLoRes</th>
      <th class="data-source">AI Hub</th>
      <th class="data-source">FLoRes</th>
      <th class="data-source">AI Hub</th>
      <th class="data-source">FLoRes</th>
      <th class="data-source">AI Hub</th>
      <th class="data-source">FLoRes</th>
      <th class="data-source">AI Hub</th>
      <th class="data-source">FLoRes</th>
      <th class="data-source">AI Hub</th>
      <th class="data-source">FLoRes</th>
      <th class="data-source">AI Hub</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Tmax (Ours)</td>
      <td>89.34</td>
      <td>88.44</td>
      <td>93.40</td>
      <td>86.52</td>
      <td>88.34</td>
      <td>93.79</td>
      <td>83.36</td>
      <td>90.03</td>
      <td>86.69</td>
      <td>89.06</td>
      <td>76.04</td>
      <td>82.33</td>
    </tr>
    <tr>
      <td>Meta</td>
      <td>85.76</td>
      <td>87.04</td>
      <td>91.98</td>
      <td>82.81</td>
      <td>86.96</td>
      <td>93.44</td>
      <td>82.88</td>
      <td>88.59</td>
      <td>83.55</td>
      <td>86.79</td>
      <td>75.03</td>
      <td>80.54</td>
    </tr>
    <tr>
      <td>Beomi-Open</td>
      <td>89.25</td>
      <td>89.33</td>
      <td>93.68</td>
      <td>87.59</td>
      <td>88.04</td>
      <td>94.22</td>
      <td>85.07</td>
      <td>91.82</td>
      <td>86.79</td>
      <td>90.09</td>
      <td>78.23</td>
      <td>84.45</td>
    </tr>
    <tr>
      <td>Saltlux-Luxia</td>
      <td>86.37</td>
      <td>93.24</td>
      <td>89.86</td>
      <td>83.27</td>
      <td>82.13</td>
      <td>76.42</td>
      <td>88.42</td>
      <td>88.11</td>
      <td>93.39</td>
      <td>87.84</td>
      <td>88.91</td>
      <td>86.25</td>
    </tr>
  </tbody>
</table>
