Update README.md
Browse files
README.md
CHANGED
@@ -91,13 +91,15 @@ your desired response length:
|
|
91 |
|
92 |
## Training procedure
|
93 |
[Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl) was used for training
|
94 |
-
on a 4x NVidia A40 GPU cluster.
|
|
|
|
|
|
|
|
|
95 |
it's so large because a LoRA rank of 256 was also used. The reasoning was that this
|
96 |
might have helped the model internalize any newly acquired information, making the
|
97 |
-
training process closer to a full finetune.
|
98 |
-
|
99 |
-
It's suggested to merge the adapter to the base Llama2-7B model (or other Llama2-based
|
100 |
-
models).
|
101 |
|
102 |
### Training hyperparameters
|
103 |
For the first pass these settings were used:
|
|
|
91 |
|
92 |
## Training procedure
|
93 |
[Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl) was used for training
|
94 |
+
on a 4x NVidia A40 GPU cluster.
|
95 |
+
|
96 |
+
The A40 GPU cluster has been graciously provided by [Arc Compute](https://www.arccompute.io/).
|
97 |
+
|
98 |
+
The model has been trained as an 8-bit LoRA adapter, and
|
99 |
it's so large because a LoRA rank of 256 was also used. The reasoning was that this
|
100 |
might have helped the model internalize any newly acquired information, making the
|
101 |
+
training process closer to a full finetune. It's suggested to merge the adapter to
|
102 |
+
the base Llama2-7B model (or other Llama2-based models).
|
|
|
|
|
103 |
|
104 |
### Training hyperparameters
|
105 |
For the first pass these settings were used:
|