D_AU - Experiments in Merging Top Models
Collection
Full precision and Q8 of merged models by me. Focus in this collection is creative, role play, story and fiction. Imatrix GGUFs addition(s) pending.
β’
50 items
β’
Updated
β’
2
INTERM STEP VERSION:
Step 1 in trying to make Tiefighter 32,768 context. This version is not usable in current form.
Step 2 however (a linear remerge of Tiefighter with this merge) is however working. GGUFs are also working... at 32768 context.
Step 2 is here: DavidAU/D_AU-Tiefighter-Plus-Giraffe-13B-32k-slerp
D_AU-Tiefighter-Giraffe-13B-32k-slerp is a merge of the following models using LazyMergekit:
slices:
- sources:
- model: KoboldAI/LLaMA2-13B-Tiefighter
layer_range: [0, 40]
- model: abacusai/Giraffe-13b-32k-v3
layer_range: [0, 40]
merge_method: slerp
base_model: abacusai/Giraffe-13b-32k-v3
parameters:
t:
- filter: self_attn
value: [0, 0.5, 0.3, 0.7, 1]
- filter: mlp
value: [1, 0.5, 0.7, 0.3, 0]
- value: 0.5
dtype: bfloat16
!pip install -qU transformers accelerate
from transformers import AutoTokenizer
import transformers
import torch
model = "DavidAU/D_AU-Tiefighter-Giraffe-13B-32k-slerp"
messages = [{"role": "user", "content": "What is a large language model?"}]
tokenizer = AutoTokenizer.from_pretrained(model)
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
pipeline = transformers.pipeline(
"text-generation",
model=model,
torch_dtype=torch.float16,
device_map="auto",
)
outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])