Hugging Face's logo

RedDragonGecko
/
MegaMaid_123b

Text Generation
Transformers
Safetensors
mistral
mergekit
Merge
conversational
text-generation-inference
Model card Files Community
RedDragonGecko commited on
Commit
30bedf9
·
verified ·
1 Parent(s): c446040

Upload 59 files

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. README.md +52 -0
  2. config.json +26 -0
  3. mergekit_config.yml +21 -0
  4. model-00001-of-00051.safetensors +3 -0
  5. model-00002-of-00051.safetensors +3 -0
  6. model-00003-of-00051.safetensors +3 -0
  7. model-00004-of-00051.safetensors +3 -0
  8. model-00005-of-00051.safetensors +3 -0
  9. model-00006-of-00051.safetensors +3 -0
  10. model-00007-of-00051.safetensors +3 -0
  11. model-00008-of-00051.safetensors +3 -0
  12. model-00009-of-00051.safetensors +3 -0
  13. model-00010-of-00051.safetensors +3 -0
  14. model-00011-of-00051.safetensors +3 -0
  15. model-00012-of-00051.safetensors +3 -0
  16. model-00013-of-00051.safetensors +3 -0
  17. model-00014-of-00051.safetensors +3 -0
  18. model-00015-of-00051.safetensors +3 -0
  19. model-00016-of-00051.safetensors +3 -0
  20. model-00017-of-00051.safetensors +3 -0
  21. model-00018-of-00051.safetensors +3 -0
  22. model-00019-of-00051.safetensors +3 -0
  23. model-00020-of-00051.safetensors +3 -0
  24. model-00021-of-00051.safetensors +3 -0
  25. model-00022-of-00051.safetensors +3 -0
  26. model-00023-of-00051.safetensors +3 -0
  27. model-00024-of-00051.safetensors +3 -0
  28. model-00025-of-00051.safetensors +3 -0
  29. model-00026-of-00051.safetensors +3 -0
  30. model-00027-of-00051.safetensors +3 -0
  31. model-00028-of-00051.safetensors +3 -0
  32. model-00029-of-00051.safetensors +3 -0
  33. model-00030-of-00051.safetensors +3 -0
  34. model-00031-of-00051.safetensors +3 -0
  35. model-00032-of-00051.safetensors +3 -0
  36. model-00033-of-00051.safetensors +3 -0
  37. model-00034-of-00051.safetensors +3 -0
  38. model-00035-of-00051.safetensors +3 -0
  39. model-00036-of-00051.safetensors +3 -0
  40. model-00037-of-00051.safetensors +3 -0
  41. model-00038-of-00051.safetensors +3 -0
  42. model-00039-of-00051.safetensors +3 -0
  43. model-00040-of-00051.safetensors +3 -0
  44. model-00041-of-00051.safetensors +3 -0
  45. model-00042-of-00051.safetensors +3 -0
  46. model-00043-of-00051.safetensors +3 -0
  47. model-00044-of-00051.safetensors +3 -0
  48. model-00045-of-00051.safetensors +3 -0
  49. model-00046-of-00051.safetensors +3 -0
  50. model-00047-of-00051.safetensors +3 -0
README.md ADDED
@@ -0,0 +1,52 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: []
3
+ library_name: transformers
4
+ tags:
5
+ - mergekit
6
+ - merge
7
+
8
+ ---
9
+ # output
10
+
11
+ This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).
12
+
13
+ ## Merge Details
14
+ ### Merge Method
15
+
16
+ This model was merged using the [Linear DELLA](https://arxiv.org/abs/2406.11617) merge method using /mnt/cb1f8c8c-0012-4778-acd5-eae4e546a0a2/models/gghfez_SmartMaid-123b as a base.
17
+
18
+ ### Models Merged
19
+
20
+ The following models were included in the merge:
21
+ * /mnt/cb1f8c8c-0012-4778-acd5-eae4e546a0a2/models/migtissera_Tess-3-Mistral-Large-2-123B
22
+ * /mnt/cb1f8c8c-0012-4778-acd5-eae4e546a0a2/models/TheDrummer_Behemoth-123B-v1
23
+ * /mnt/cb1f8c8c-0012-4778-acd5-eae4e546a0a2/models/gghfez_Writer-Large-2411-v2.1
24
+
25
+ ### Configuration
26
+
27
+ The following YAML configuration was used to produce this model:
28
+
29
+ ```yaml
30
+ models:
31
+ - model: /mnt/cb1f8c8c-0012-4778-acd5-eae4e546a0a2/models/gghfez_Writer-Large-2411-v2.1
32
+ parameters:
33
+ weight: 0.25
34
+ density: 0.4
35
+ - model: /mnt/cb1f8c8c-0012-4778-acd5-eae4e546a0a2/models/TheDrummer_Behemoth-123B-v1
36
+ parameters:
37
+ weight: 0.40
38
+ density: 0.6
39
+ - model: /mnt/cb1f8c8c-0012-4778-acd5-eae4e546a0a2/models/migtissera_Tess-3-Mistral-Large-2-123B
40
+ parameters:
41
+ weight: 0.35
42
+ density: 0.5
43
+ merge_method: della_linear
44
+ base_model: /mnt/cb1f8c8c-0012-4778-acd5-eae4e546a0a2/models/gghfez_SmartMaid-123b
45
+ parameters:
46
+ epsilon: 0.05
47
+ lambda: 1
48
+ int8_mask: true
49
+ dtype: bfloat16
50
+
51
+
52
+ ```
config.json ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "MistralForCausalLM"
4
+ ],
5
+ "attention_dropout": 0.0,
6
+ "bos_token_id": 1,
7
+ "eos_token_id": 2,
8
+ "head_dim": 128,
9
+ "hidden_act": "silu",
10
+ "hidden_size": 12288,
11
+ "initializer_range": 0.02,
12
+ "intermediate_size": 28672,
13
+ "max_position_embeddings": 131072,
14
+ "model_type": "mistral",
15
+ "num_attention_heads": 96,
16
+ "num_hidden_layers": 88,
17
+ "num_key_value_heads": 8,
18
+ "rms_norm_eps": 1e-05,
19
+ "rope_theta": 1000000.0,
20
+ "sliding_window": null,
21
+ "tie_word_embeddings": false,
22
+ "torch_dtype": "bfloat16",
23
+ "transformers_version": "4.50.0",
24
+ "use_cache": true,
25
+ "vocab_size": 32768
26
+ }
mergekit_config.yml ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ models:
2
+ - model: /mnt/cb1f8c8c-0012-4778-acd5-eae4e546a0a2/models/gghfez_Writer-Large-2411-v2.1
3
+ parameters:
4
+ weight: 0.25
5
+ density: 0.4
6
+ - model: /mnt/cb1f8c8c-0012-4778-acd5-eae4e546a0a2/models/TheDrummer_Behemoth-123B-v1
7
+ parameters:
8
+ weight: 0.40
9
+ density: 0.6
10
+ - model: /mnt/cb1f8c8c-0012-4778-acd5-eae4e546a0a2/models/migtissera_Tess-3-Mistral-Large-2-123B
11
+ parameters:
12
+ weight: 0.35
13
+ density: 0.5
14
+ merge_method: della_linear
15
+ base_model: /mnt/cb1f8c8c-0012-4778-acd5-eae4e546a0a2/models/gghfez_SmartMaid-123b
16
+ parameters:
17
+ epsilon: 0.05
18
+ lambda: 1
19
+ int8_mask: true
20
+ dtype: bfloat16
21
+
model-00001-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3aa4597e2cd829b07f81e52b621f298114affb3c0a918d2857c04fce1072a1b4
3
+ size 4378928504
model-00002-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:64b58df15f1aba3ff76e5444d9176124592320a5d1ffe0c4dee78bed341cf9d1
3
+ size 4907411088
model-00003-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:165142b14e1baa84fdf05844eb891d8d178b4389803aeeb2791a2c74f36b4b06
3
+ size 4806747904
model-00004-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2dc85659279ce042cf246c8dcc2d68d87219395eebb42c8de0657793d1787cba
3
+ size 4831938544
model-00005-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fedd2b67c8c60369cf4970d16cc98c27e6c77d057c17e807662066d85ed6a225
3
+ size 4831938552
model-00006-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:78ed1fafc1afaaba68a823710b35854ae7d8e819320d3f354c7fa31e81ccce48
3
+ size 4907411096
model-00007-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ab61befc253b6017134025ec4ebe52b954c3878e325e5ea3d9c3e9d6c4ff4f1a
3
+ size 4806747904
model-00008-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:13885583000d720241d989b8c5a4da3790d056131df80195255bafd93c110c0f
3
+ size 4831938536
model-00009-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b75d0f604e6585693fc4281264a8597b69f55b966a0cf5a0ac4c779cf88b1144
3
+ size 4831938552
model-00010-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2247a5048c441f94fb7e3a6bd109af2f548af32b47756bc2e3d85f2c6d8ed0b3
3
+ size 4907411096
model-00011-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9ef68c57882c087aadaa4eb46013df258a46afbeb1f203854eac1e47016958a0
3
+ size 4806747904
model-00012-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1313f86d015a809279334e849c193a8b7ea3224375745930c0e94c44df90aeb2
3
+ size 4831938544
model-00013-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a04c01b561e49199733c0a68069b16489985db296f2d6f11183dbabcd1888449
3
+ size 4831938552
model-00014-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:338cdac64d5d413cd198da038fb3bbe6a15f9a1832e8e0507b6ddd7eaf633e57
3
+ size 4907411088
model-00015-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5ba0bb63a165706cf95afe9688563cca07bd948bf7ba8c5dc95d0d1476492f39
3
+ size 4806747904
model-00016-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:155f00f879f90d6434aab8fff1bfb0fe34f03a3e868f3ea3a5228559c9ed50dd
3
+ size 4831938544
model-00017-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2827ffc7999d2e9c74cdefd947c93bd11043b4fae28d6e2a540d2338a1ba64b6
3
+ size 4831938552
model-00018-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:30bde48527eece68c04d5cce5539d01776c21df57d4cfd7ed810783337436443
3
+ size 4907411096
model-00019-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:93b762fabb11e134c0f416a5d09e8c855689c56979c2cf36d6b187bcceb7c040
3
+ size 4806747904
model-00020-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:de83bea8345b0583cef5a2ba2d54306a50ef0f304e54e14b3b5e02ecc6356589
3
+ size 4831938544
model-00021-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:781147524b90bcce2b60442929f0931db96a39fea518afe1eb6e8d9b59498db8
3
+ size 4831938544
model-00022-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:dbf06c10cdbf1df910aaaf387579646a974679145c410ca4ca1ab51c76d58c06
3
+ size 4907411096
model-00023-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3a6e2b6ded44df551b54a1f3fa0c0409a1e4f5091327589ed576f0c908493cd8
3
+ size 4806747904
model-00024-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d0d1b8360f4ec2844c4d3cf608f015356a7965c792364304f8323f7c60bb8479
3
+ size 4831938544
model-00025-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f71efc1050d439d855264d88e15fc9c0837ce181c52c8b0cbda6a9ea22dae694
3
+ size 4831938552
model-00026-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d930d1951b4a15708e4921a417c8dc30761f85e154e1c584715c93d680c6e279
3
+ size 4907411096
model-00027-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f31a78c417e664dbd0227f57f45a53af8052c1aa6a24fcb18d931ca3337e6a24
3
+ size 4806747896
model-00028-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bbc97b14ae0b2d607739507c34d92c69429ac32f1ca37f76fdac9461ca38324b
3
+ size 4831938544
model-00029-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c0d8f17333ed6d9485bff5485f122b9a2240457a7ec24995c91018b971ae80f1
3
+ size 4831938552
model-00030-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6da6239842a7603573df1b7ca733dffd8bad6939a5013ee6e5e250119515955f
3
+ size 4907411096
model-00031-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b08b83d7ef5c6350e45b985273c3eedfa3407e48ddd5c79ea9b17ffbfe6307c6
3
+ size 4806747904
model-00032-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fa470c58b7634f386731f636c75dce430b5f43bb8ec3e14f3163d0fee8fd9e60
3
+ size 4831938544
model-00033-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:84c5af0e616fed67c3b52f957380e74dca7c9f0b698ca739cf0a22365fa9b000
3
+ size 4831938544
model-00034-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0be2aaa4a8f3691887c965d6aa2017eefb936bea79a4755882a8bd73bdd1cea6
3
+ size 4907411096
model-00035-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3b6781c626be3e90a5cf2524ca8856ec07f5aa959d2789bad88510d7d5ab204c
3
+ size 4806747904
model-00036-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f8aaecbd3397f4bbfedfbe1447d1284e99cb27ead45834a967b5fa717a9a1950
3
+ size 4831938544
model-00037-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b7f3a1e69dbb76a5f1654d23deffb2685f95015afa53606c47695a0f132a746e
3
+ size 4831938552
model-00038-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8d1c3cee9734f44949fe3007e8d2614ba326f71c9dbd1481d2182aabc5ac04ff
3
+ size 4907411096
model-00039-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ea9d2ff1ce22f226cd994439722881dd8f9c06dddcef459a52b20b0861fd84f5
3
+ size 4806747904
model-00040-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:604cd1820e1e0d76d64e8b0cdbee953f0167e76c0f4e2b201ab1e9ae8034839d
3
+ size 4831938544
model-00041-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:37dcd15a291f675e3384f1569977bb772b8e91ec788449aff60c72854c31d470
3
+ size 4831938552
model-00042-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7ef69577beb4121ff12fd60a2dbd9de0a1be7012b8adce736702d025631772a4
3
+ size 4907411096
model-00043-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:372f9b9f1431ed20b1fe05710e697ddc75db8cfb0c0e0f3da176cb311a4b7599
3
+ size 4806747904
model-00044-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d75efba294ec0a8b26270774a80d1b4c994750685a1460afb5201a106f46a2f9
3
+ size 4831938544
model-00045-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6ab058215df7c3c82b30c67f8e1b64c02dd67113574571631845b878053d1b71
3
+ size 4831938552
model-00046-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c28c024b63b9f59c2a82d26f3cf22f382f0170f0d1e87db89701abb9196a5724
3
+ size 4907411088
model-00047-of-00051.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:07953128ea2a508184fcb928edd83f086264b6979b064eb94735c8155dd7e073
3
+ size 4806747904