---
base_model:
- Lambent/threebird-scribe-alpha0.3-7B
- Lambent/bigbird-scribe-7B
- Lambent/aetherbird-scribe-7B
- Lambent/songbird-scribe-7B
- Lambent/codebird-scribe-7B
library_name: transformers
tags:
- mergekit
- merge
license: apache-2.0
datasets:
- Lambent/storytellers-32k
- Doctor-Shotgun/no-robots-sharegpt
- BEE-spoke-data/sarcasm-scrolls
- TheSkullery/Aether-Lite-V1.6
- vishnupriyavr/spotify-million-song-dataset
- TheSkullery/Gryphe-Opus-WritingPrompts-merged
- bigcode/the-stack-smol-xs
- bjoernp/Vezora_Tested-22k-Python-Alpaca-sharegpt-filtered
- thesven/code_bagel_35k
- practical-dreamer/RPGPT_PublicDomain-ShareGPT
- Undi95/Capybara-ShareGPT
---
# fourbirdstock

This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).

## Merge Details

| Tasks  |Version|Filter|n-shot|     Metric      |   | Value  |   |Stderr|
|--------|------:|------|-----:|-----------------|---|-------:|---|-----:|
|eq_bench|    2.1|none  |     0|eqbench          |↑  | 78.7955|±  |1.4668|
|        |       |none  |     0|percent_parseable|↑  |100.0000|±  |0.0000|


0.3 involved 3 separate tunes stock merged on overlapping datasets for long context writing, multi-turn conversation and RP, with a touch of poetry and code.
From there, each of the four threads was separately task-tuned on 2 datasets each.
Various methods of combining those via merge were tested, with this one scoring highest on EQ-Bench as an indicator.

My understanding of the Model Stock merge method is that it reduces task adaptation to a significant degree, but also significantly limits forgetting caused by training.
I have hope that the adaptation, especially over two stages, is still sufficient to aid in longer contexts and multi-turn conversations from the ancestor models, and add some individual style while retaining a fair amount of their capability.

This model's refusals are ... not nonexistent, but certainly don't rely on them.
To my knowledge it has no particular refusal behavior for simply NSFW content, but I haven't exactly exhaustively tested which OSHA violations it will aid and abet.

### Merge Method

This model was merged using the [Model Stock](https://arxiv.org/abs/2403.19522) merge method using [Lambent/threebird-scribe-alpha0.3-7B](https://huggingface.co/Lambent/threebird-scribe-alpha0.3-7B) as a base.

### Models Merged

The following models were included in the merge:
* [Lambent/bigbird-scribe-7B](https://huggingface.co/Lambent/bigbird-scribe-7B)
* [Lambent/aetherbird-scribe-7B](https://huggingface.co/Lambent/aetherbird-scribe-7B)
* [Lambent/songbird-scribe-7B](https://huggingface.co/Lambent/songbird-scribe-7B)
* [Lambent/codebird-scribe-7B](https://huggingface.co/Lambent/codebird-scribe-7B)

### Configuration

The following YAML configuration was used to produce this model:

```yaml
models:
  - model: Lambent/codebird-scribe-7B
  - model: Lambent/songbird-scribe-7B
  - model: Lambent/aetherbird-scribe-7B
  - model: Lambent/bigbird-scribe-7B
base_model: Lambent/threebird-scribe-alpha0.3-7B
merge_method: model_stock
parameters:
  filter_wise: false
tokenizer_source: union
dtype: float16


```