Update README.md
Browse files
README.md
CHANGED
@@ -11,7 +11,7 @@ Be sure to use the right settings -- the given system prompt and a very low temp
|
|
11 |
|
12 |
### Model Description
|
13 |
|
14 |
-
|
15 |
|
16 |
Any AI participating in the Verus ecosystem needs to understand the system itself, and so we have open-sourced the dataset generation code and datasets that this model was trained on. More broadly, this datagen code is a demonstration of how domain-expert LLMs can be quickly trained — as a fully community-driven project, Verus is no stranger to open-source contribution. The Verus project sees the future as decentralized, people-powered, and AI-enhanced, and we are incredibly proud to release this first version of our LLM to you all (alongside the pipeline and methodology that created it).
|
17 |
|
@@ -76,7 +76,7 @@ This model is meant to help you get a good, guided introduction to Verus. It may
|
|
76 |
|
77 |
## Model, Training, And Dataset Details (TECHNICALLY HEAVY)
|
78 |
|
79 |
-
|
80 |
|
81 |
We used about 4.3 million tokens of instruct Verus data, generated with our custom [dataset generation code](https://github.com/e-p-armstrong/verustoolkit) based on [Augmentoolkit](https://github.com/e-p-armstrong/augmentoolkit/tree/master) and about 1.2 million tokens of generic instruct data. We also gave the model the "source text" used as inputs for our dataset generation code as continued pretraining data, to ensure that no knowledge slipped through the cracks. We then did a full finetune for 6 epochs on 7 A40s.
|
82 |
|
|
|
11 |
|
12 |
### Model Description
|
13 |
|
14 |
+
Llama 3 VerusGPT is a fully open-source domain-expert LLM trained to answer questions about the Verus Project. It is an AI trained to be an expert specifically on Verus, with the near-term goal of helping educate people about Verus, and the long-term goal of developing AI participants for Verus' decentralized blockchain economy.
|
15 |
|
16 |
Any AI participating in the Verus ecosystem needs to understand the system itself, and so we have open-sourced the dataset generation code and datasets that this model was trained on. More broadly, this datagen code is a demonstration of how domain-expert LLMs can be quickly trained — as a fully community-driven project, Verus is no stranger to open-source contribution. The Verus project sees the future as decentralized, people-powered, and AI-enhanced, and we are incredibly proud to release this first version of our LLM to you all (alongside the pipeline and methodology that created it).
|
17 |
|
|
|
76 |
|
77 |
## Model, Training, And Dataset Details (TECHNICALLY HEAVY)
|
78 |
|
79 |
+
Llama 3 VerusGPT is built with Meta Llama 3. We trained on the base Llama 3 8b model, using a combination of generalized assistant-style data, and special instruct tuning data focused on the Verus project. The goal was to produce a domain-expert AI that can answer basic- to intermediate-difficulty questions about the Verus protocol, community, and project; and which also understands the Verus community ethos and mission, and can communicate this to other people.
|
80 |
|
81 |
We used about 4.3 million tokens of instruct Verus data, generated with our custom [dataset generation code](https://github.com/e-p-armstrong/verustoolkit) based on [Augmentoolkit](https://github.com/e-p-armstrong/augmentoolkit/tree/master) and about 1.2 million tokens of generic instruct data. We also gave the model the "source text" used as inputs for our dataset generation code as continued pretraining data, to ensure that no knowledge slipped through the cracks. We then did a full finetune for 6 epochs on 7 A40s.
|
82 |
|