|
|
--- |
|
|
language: |
|
|
- en |
|
|
tags: |
|
|
- pytorch |
|
|
- causal-lm |
|
|
- pythia |
|
|
license: apache-2.0 |
|
|
datasets: |
|
|
- Dahoas/synthetic-instruct-gptj-pairwise |
|
|
--- |
|
|
|
|
|
This model is created by finetuning `EleutherAI/pythia-2.8b-deduped` on the `Dahoas/synthetic-instruct-gptj-pairwise` for 4 epochs. |
|
|
|
|
|
You can try a [demo](https://cloud.lambdalabs.com/demos/ml/qa-28b-8000) of the model hosted on [Lambda Cloud](https://lambdalabs.com/service/gpu-cloud). |
|
|
|
|
|
It took 8xA100 80GB five hours to train the model. We set `batch_size_per_gpu` to `2` (so global batch size is 8), and learning rate to `0.00001` (with linear decay to zero at the last trainig step). |
|
|
|
|
|
The Weights and Biases record of the training can be found [here](https://wandb.ai/chuanli11/ft-synthetic-instruct-gptj-pairwise-pythia2.8b?workspace=user-chuanli11). |
|
|
|
