Update README.md
Browse files
README.md
CHANGED
|
@@ -255,9 +255,6 @@ To train the model, we have developed our own training and annotation library: [
|
|
| 255 |
|
| 256 |
For the hackathon, we decided to train a model with 7 trillion parameters, since using 4-bit quantization, it is possible to run the model on domestic hardware. After analyzing the performance of a large number of LLMs, we chose [openchat-3.5-0106](https://huggingface.co/openchat/openchat-3.5-0106) due to its high performance without the need for pretraining. To minimally disturb the prior knowledge of the model that allows for this performance, we opted to use the *Low-Rank Adaptation* (LoRA) training technique.
|
| 257 |
|
| 258 |
-
The exact training configuration is available at: https://huggingface.co/somosnlp/NoticIA-7B/blob/main/openchat-3.5-0106_LoRA.yaml
|
| 259 |
-
|
| 260 |
-
|
| 261 |
#### Training Hyperparameters
|
| 262 |
|
| 263 |
|
|
|
|
| 255 |
|
| 256 |
For the hackathon, we decided to train a model with 7 trillion parameters, since using 4-bit quantization, it is possible to run the model on domestic hardware. After analyzing the performance of a large number of LLMs, we chose [openchat-3.5-0106](https://huggingface.co/openchat/openchat-3.5-0106) due to its high performance without the need for pretraining. To minimally disturb the prior knowledge of the model that allows for this performance, we opted to use the *Low-Rank Adaptation* (LoRA) training technique.
|
| 257 |
|
|
|
|
|
|
|
|
|
|
| 258 |
#### Training Hyperparameters
|
| 259 |
|
| 260 |
|