|
|
--- |
|
|
license: unknown |
|
|
Batch Size: 1 |
|
|
Gradient Accumulation Steps: 8 |
|
|
Cutoff Length: 512 |
|
|
Epochs: 1 |
|
|
Learning rate: 1e-2 |
|
|
Optimizer: Adafactor |
|
|
LR Scheduler: Linear |
|
|
Warmup Steps: 64 |
|
|
Projections: Q, K, V, O, up, gate, down |
|
|
|
|
|
--- |
|
|
More of a proof of concept. Temper your expectations. |
|
|
|
|
|
Warning: May generate 18+ content. |
|
|
Trained on the dialogue from Adastra (the furry visual novel) in text-generation-webui (by oobabooga), with a modified Training_PRO extension. |
|
|
Model was loaded unquantized (BF16). |
|
|
Currently, loading IA3's works in unmodified textgen-webui (I think... Load it as you would a LoRA) only if you load the model unquantized, and not in 4 or 8 bit. |
|
|
|
|
|
Other Training parameters: |
|
|
|
|
|
Add overlapping blocks: On |
|
|
|
|
|
DEMENTOR (long form learning by FP): On //This might be the secret sauce that makes this IA3 so effective with just 1 epoch of training. |
|
|
|
|
|
Extra: |
|
|
|
|
|
Training took 9 minutes on an RTX 3090 |
|
|
|
|
|
If you are the creator of Adastra and would like this taken down, please contact me. |
|
|
|
|
|
I do not claim to have produced the training data that went into this finetune. |
