|
|
--- |
|
|
license: unknown |
|
|
--- |
|
|
Warning: May generate 18+ content |
|
|
Trained on the dialogue from Adastra (the furry visual novel) in text-generation-webui (by oobabooga), with a modified Training_PRO extension. |
|
|
Model was loaded unquantized (BF16). |
|
|
Currently, loading IA3's works in unmodified textgen-webui (I think... Load it as you would a LoRA) only if you load the model unquantized, and not |
|
|
in 4 or 8 bit. |
|
|
|
|
|
Training parameters: |
|
|
Batch Size: 1 |
|
|
Gradient Accumulation Steps: 8 |
|
|
Cutoff Length: 512 //Didn't have enough memory for more. |
|
|
Epochs: 1 |
|
|
Learning Rate: 1e-2 //yes, unusually high. maybe a quirk of IA3's? |
|
|
LR Scheduler: Linear |
|
|
Warmup Steps: 64 |
|
|
Targets: all (Q, K, V, O, up, down, gate) |
|
|
Optimizer: Adafactor |
|
|
Add overlapping blocks: On |
|
|
DEMENTOR (long form learning by FP): On //This might be the secret sauce that makes this IA3 so effective with just 1 epoch of training. |
|
|
|
|
|
Training took 9 minutes on an RTX 3090 |
|
|
If you are the creator of Adastra and would like this taken down, please contact me. |
|
|
I do not claim to have produced the training data that went into this finetune. |
