File size: 3,997 Bytes
e777201 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 |
---
base_model:
- Sao10K/L3-8B-Niitama-v1
- Nitral-AI/Hathor_Tahsin-L3-8B-v0.85
- ArliAI/ArliAI-Llama-3-8B-Formax-v1.0
- nothingiisreal/L3-8B-Celeste-V1.2
library_name: transformers
tags:
- mergekit
- merge
---
1/3 of the 13B models for Horizon Himerus (will update with link later). This merge was orginally suppose to be only for that final model, but this guy is suprisangly competent.
A tad jank, but very solid for what it is.
### Quants
[OG Q8 GGUF](https://huggingface.co/kromeurus/L3-Himerus-Basis.C-13B-Q8-GGUF) by me.
Other quants are not available, yet.
### Details & Recommended Settings
(Still testing; nothing here is finalized.)
Follows intructions fairly well for RP and eRP. Dramatic as fuck at times, depending on the senario. Human dialogue and lots of it.
Rec. Settings:
```
Template: Model Default
Temperature: 1.22
Min P: 0.115
Repeat Penelty: 1.05
Repeat Penelty Tokens: 256
```
### Models Merged & Merge Theory
The following models were included in the merge:
* [Sao10K/L3-8B-Niitama-v1](https://huggingface.co/Sao10K/L3-8B-Niitama-v1)
* [Nitral-AI/Hathor_Tahsin-L3-8B-v0.85](https://huggingface.co/Nitral-AI/Hathor_Tahsin-L3-8B-v0.85)
* [ArliAI/ArliAI-Llama-3-8B-Formax-v1.0](https://huggingface.co/ArliAI/ArliAI-Llama-3-8B-Formax-v1.0)
* [nothingiisreal/L3-8B-Celeste-V1.2](https://huggingface.co/nothingiisreal/L3-8B-Celeste-V1.2)
So you're not suppose to mix models with different trained context limits, but I did it anyway. Wanted the 'human' output of Celeste v1.2 while curbing the repitition and adding
some back up from Niitama and Hathor Tahsin. Formax was included in the beginning for it's instruct following.
Took a page out of [@matchaaaaa](https://huggingface.co/matchaaaaa)'s Chaifighter Latte and took out a slice of Celeste and Nittama in the center for smoothing out layer disparity.
I realized while testing that using that 'splice' metheod, you could theoretically make a pretty big model then squish it down to streamline the layers. So, after much testing,
I came up with the following merges.
### Config
```yaml
models:
slices:
- sources:
- layer_range: [14, 20]
model: nothingiisreal/L3-8B-Celeste-V1.2
parameters:
int8_mask: true
merge_method: passthrough
dtype: bfloat16
name: celeste14-20.sl
---
models:
slices:
- sources:
- layer_range: [14, 20]
model: Sao10K/L3-8B-Niitama-v1
parameters:
int8_mask: true
merge_method: passthrough
dtype: bfloat16
name: niitama14-20.sl
---
models:
- model: celeste14-20.sl
parameters:
weight: [1, 0.75, 0.625, 0.5, 0.375, 0.25, 0]
- model: niitama14-20.sl
parameters:
weight: [0, 0.25, 0.375, 0.5, 0.625, 0.75, 1]
merge_method: dare_linear
base_model: celeste14-20.sl
dtype: bfloat16
name: celeniit14-20.sl
---
models:
slices:
- sources:
- layer_range: [0, 4]
model: Nitral-AI/Hathor_Tahsin-L3-8B-v0.85
- sources:
- layer_range: [1, 5]
model: ArliAI/ArliAI-Llama-3-8B-Formax-v1.0
- sources:
- layer_range: [4, 8]
model: Nitral-AI/Hathor_Tahsin-L3-8B-v0.85
- sources:
- layer_range: [5, 9]
model: ArliAI/ArliAI-Llama-3-8B-Formax-v1.0
- sources:
- layer_range: [8, 10]
model: Sao10K/L3-8B-Niitama-v1
- sources:
- layer_range: [6, 14]
model: nothingiisreal/L3-8B-Celeste-V1.2
- sources:
- layer_range: [0, 6]
model: celeniit14-20.sl
- sources:
- layer_range: [20, 23]
model: Sao10K/L3-8B-Niitama-v1
- sources:
- layer_range: [22, 26]
model: Nitral-AI/Hathor_Tahsin-L3-8B-v0.85
- sources:
- layer_range: [22, 28]
model: nothingiisreal/L3-8B-Celeste-V1.2
- sources:
- layer_range: [25, 27]
model: Nitral-AI/Hathor_Tahsin-L3-8B-v0.85
- sources:
- layer_range: [28, 30]
model: Sao10K/L3-8B-Niitama-v1
- sources:
- layer_range: [25, 32]
model: nothingiisreal/L3-8B-Celeste-V1.2
parameters:
int8_mask: true
merge_method: passthrough
dtype: bfloat16
```
|