Upload folder using huggingface_hub

README.md

@@ -0,0 +1,137 @@
+---
+license: other
+license_name: yi-license
+license_link: https://huggingface.co/01-ai/Yi-34B/blob/main/LICENSE
+language:
+- en
+library_name: transformers
+base_model: []
+tags:
+- mergekit
+- merge
+- Yi
+- exllama
+- exllamav2
+- exl2
+---
+# RPMerge
+A merge of several Yi 34B models with a singular goal: 40K+ context, instruct-enhanced storytelling.
+
+Disappointed with some quirks of my previous kitchen sink merges (like token/instruct formats from various models showing up when they shouldn't), I've gone 'back to the basics' and picked a few Vicuna-format only models:
+
+- [DrNicefellow/ChatAllInOne-Yi-34B-200K-V1](https://huggingface.co/DrNicefellow/ChatAllInOne-Yi-34B-200K-V1) and [migtissera/Tess-34B-v1.5b](https://huggingface.co/migtissera/Tess-34B-v1.5b) both have excellent general instruction-following performance.
+
+- [cgato/Thespis-34b-v0.7](https://huggingface.co/cgato/Thespis-34b-v0.7) is trained on the "Username: {Input} / BotName: {Response}" format, to emphasize it in the merge (but not force it). It also seems to work for multi-character stories.
+
+- [Doctor-Shotgun/limarpv3-yi-llama-34b-lora](https://huggingface.co/Doctor-Shotgun/limarpv3-yi-llama-34b-lora) is trained on roleplaying data, but merged at a modest weight to not over emphasize it. This is the only non-vicuna model (being alpaca format), but it doesn't seem to interefere with the Vicuna format or adversely affect long-context perplexity
+
+- [adamo1139/yi-34b-200k-rawrr-dpo-2](https://huggingface.co/adamo1139/yi-34b-200k-rawrr-dpo-2) the base for the limarp lora, this is base Yi gently finetuned to discourage refusals.
+
+- [migtissera/Tess-M-Creative-v1.0](https://huggingface.co/migtissera/Tess-M-Creative-v1.0) and [NousResearch/Nous-Capybara-34B](https://huggingface.co/NousResearch/Nous-Capybara-34B) are both "undertrained" Yi models. I find they excel at raw completion performance (like long novel continuations) while still retaining some Vicuna instruct ability. This may be why some still prefer the original Tess 1.0/Capybara merge.
+
+I consider this a more "focused" merge that previous ones. I will investigate other models (perhaps chatML models?) for a more "factual assistant" focused merge, as well as a coding-focused merge if I can't find one to suit my needs.
+
+
+## Prompt template: Orca-Vicuna
+```
+SYSTEM: {system_message}
+USER: {prompt}
+ASSISTANT:
+```
+Raw prompting as described here is also effective: https://old.reddit.com/r/LocalLLaMA/comments/18zqy4s/the_secret_to_writing_quality_stories_with_llms/
+
+As well as a very explicit system prompt like this: https://old.reddit.com/r/LocalLLaMA/comments/1aiz6zu/roleplaying_system_prompts/koygiwa/
+
+
+## Running
+
+Chinese models with large tokenizer vocabularies like Yi need *careful* parameter tuning due to their huge logit sampling "tails." Yi in particular also runs relatively "hot" even at lower temperatures.
+
+I am a huge fan of Kalomaze's quadratic sampling (shown as "smoothing factor" where available), as described here: https://github.com/oobabooga/text-generation-webui/pull/5403
+
+Otherwise, I recommend a lower temperature with 0.1 or higher MinP, a little repetition penalty, and mirostat with a low tau, and no other samplers. See the explanation here: https://github.com/ggerganov/llama.cpp/pull/3841
+
+24GB GPUs can efficiently run Yi-34B-200K models at **40K-90K context** with exllamav2, and performant UIs like [exui](https://github.com/turboderp/exui). I go into more detail in this [post](https://old.reddit.com/r/LocalLLaMA/comments/1896igc/how_i_run_34b_models_at_75k_context_on_24gb_fast/). Empty 16GB GPUs can still run the high context with aggressive quantization.
+
+To load/train this in full-context backends like transformers, you *must* change `max_position_embeddings` in config.json to a lower value than 200,000, otherwise you will OOM! I do not recommend running high context without context-efficient backends that support flash attention + 8 bit kv cache, like exllamav2, litellm, vllm or unsloth.
+
+
+## Testing Notes
+
+Thanks to ParasiticRogue for this idea of a Vicuna-only merge, see: https://huggingface.co/brucethemoose/jondurbin_bagel-dpo-34b-v0.2-exl2-4bpw-fiction/discussions
+
+See: https://huggingface.co/brucethemoose/Yi-34B-200K-DARE-megamerge-v8#testing-notes
+
+This is a possible base for a storytelling finetune/LASER in the future, once I can bite the bullet and rent some A100s or a MI300. 
+
+I have tested this merge with with novel-style continuation (but not much chat-style roleplay), and some assistant-style responses and long context analysis. I haven't seen any refusals so far.
+
+## Merge Details
+### Merge Method
+
+This model was merged using the [DARE](https://arxiv.org/abs/2311.03099) [TIES](https://arxiv.org/abs/2306.01708) merge method using /home/alpha/Models/Raw/chargoddard_Yi-34B-200K-Llama as a base.
+
+### Models Merged
+
+The following models were included in the merge:
+* /home/alpha/Models/Raw/migtissera_Tess-34B-v1.5b
+* /home/alpha/Models/Raw/migtissera_Tess-M-Creative-v1.0
+* /home/alpha/Models/Raw/cgato_Thespis-34b-DPO-v0.7
+* /home/alpha/Models/Raw/Nous-Capybara-34B
+* /home/alpha/Models/Raw/admo_limarp
+* /home/alpha/Models/Raw/DrNicefellow_ChatAllInOne-Yi-34B-200K-V1
+
+### Configuration
+
+The following YAML configuration was used to produce this model:
+
+```yaml
+models:
+ - model: /home/alpha/Models/Raw/chargoddard_Yi-34B-200K-Llama
+ # No parameters necessary for base model
+ - model: /home/alpha/Models/Raw/migtissera_Tess-34B-v1.5b
+ #Emphasize the beginning of Vicuna format models
+ parameters:
+ weight: 0.19
+ density: 0.59
+ - model: /home/alpha/Models/Raw/Nous-Capybara-34B
+ parameters:
+ weight: 0.19
+ density: 0.55
+ # Vicuna format
+ - model: /home/alpha/Models/Raw/migtissera_Tess-M-Creative-v1.0
+ parameters:
+ weight: 0.05
+ density: 0.55
+ - model: /home/alpha/Models/Raw/DrNicefellow_ChatAllInOne-Yi-34B-200K-V1
+ parameters:
+ weight: 0.19
+ density: 0.55
+ - model: adamo1139/yi-34b-200k-rawrr-dpo-2+Doctor-Shotgun/limarpv3-yi-llama-34b-lora
+ parameters:
+ weight: 0.19
+ density: 0.48
+ - model: /home/alpha/Models/Raw/cgato_Thespis-34b-DPO-v0.7
+ parameters:
+ weight: 0.19
+ density: 0.59
+
+
+merge_method: dare_ties
+tokenizer_source: union
+base_model: /home/alpha/Models/Raw/chargoddard_Yi-34B-200K-Llama
+parameters:
+ int8_mask: true
+dtype: bfloat16
+```
+
+
+## Self Promotion
+
+I'm part of a AI startup called Holocene AI!
+
+We're new, busy, and still setting things up. But if you have any business inquiries, want a job, or just want some consultation, feel free to shoot me an email. We have expertise in RAG applications and llama/embeddings model finetuning, and absolutely *none* of the nonsense of scammy AI startups.
+
+Contact me at: [email protected]
+
+I also set up a Ko-Fi! I want to run some (personal) training/LASERing as well, at 100K context or so. If you'd like to buy me 10 minutes on an A100 (or 5 seconds on an MI300X), I'd appreciate it: https://ko-fi.com/alphaatlas

added_tokens.json

@@ -0,0 +1,4 @@
+{
+ "</s>": 64001,
+ "<s>": 64000
+}

config.json

@@ -0,0 +1,37 @@
+{
+ "_name_or_path": "/models/models/Yi-34B-200K-RPMerge",
+ "architectures": [
+ "LlamaForCausalLM"
+ ],
+ "attention_bias": false,
+ "attention_dropout": 0.0,
+ "bos_token_id": 1,
+ "eos_token_id": 2,
+ "hidden_act": "silu",
+ "hidden_size": 7168,
+ "initializer_range": 0.02,
+ "intermediate_size": 20480,
+ "max_position_embeddings": 200000,
+ "model_type": "llama",
+ "num_attention_heads": 56,
+ "num_hidden_layers": 60,
+ "num_key_value_heads": 8,
+ "pad_token_id": 0,
+ "pretraining_tp": 1,
+ "quantization_config": {
+ "bits": 4,
+ "group_size": 128,
+ "modules_to_not_convert": null,
+ "quant_method": "awq",
+ "version": "gemm",
+ "zero_point": true
+ },
+ "rms_norm_eps": 1e-05,
+ "rope_scaling": null,
+ "rope_theta": 5000000.0,
+ "tie_word_embeddings": false,
+ "torch_dtype": "float16",
+ "transformers_version": "4.37.1",
+ "use_cache": true,
+ "vocab_size": 64002
+}

generation_config.json

@@ -0,0 +1,8 @@
+{
+ "_from_model_config": true,
+ "bos_token_id": 1,
+ "do_sample": true,
+ "eos_token_id": 2,
+ "pad_token_id": 0,
+ "transformers_version": "4.37.1"
+}

model-00001-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:350292c70b4e994c6300dfa1b8d654fa77705b0e8d1db6f4948ef7808f80f66e
+size 4975402760

model-00002-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ae8f4358745bf2d5885aa2af60305d5a7330ce5c74844d3ca53ea44c0837d645
+size 4988429248

model-00003-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0c75c51bd076050b2f57d09e82deb3a19c6db1b2550b3d74e5467467840b2f33
+size 4927413864

model-00004-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1d27ae906d76ef0c8010acc27fcc916eb60464ce8c54eb9a4476cf30d68e96aa
+size 4334706272

special_tokens_map.json

@@ -0,0 +1,30 @@
+{
+ "bos_token": {
+ "content": "<|startoftext|>",
+ "lstrip": false,
+ "normalized": false,
+ "rstrip": false,
+ "single_word": false
+ },
+ "eos_token": {
+ "content": "<|endoftext|>",
+ "lstrip": false,
+ "normalized": false,
+ "rstrip": false,
+ "single_word": false
+ },
+ "pad_token": {
+ "content": "<unk>",
+ "lstrip": false,
+ "normalized": false,
+ "rstrip": false,
+ "single_word": false
+ },
+ "unk_token": {
+ "content": "<unk>",
+ "lstrip": false,
+ "normalized": false,
+ "rstrip": false,
+ "single_word": false
+ }
+}

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:386c49cf943d71aa110361135338c50e38beeff0a66593480421f37b319e1a39
+size 1033105

tokenizer_config.json

@@ -0,0 +1,59 @@
+{
+ "add_bos_token": false,
+ "add_eos_token": false,
+ "added_tokens_decoder": {
+ "0": {
+ "content": "<unk>",
+ "lstrip": false,
+ "normalized": false,
+ "rstrip": false,
+ "single_word": false,
+ "special": true
+ },
+ "1": {
+ "content": "<|startoftext|>",
+ "lstrip": false,
+ "normalized": false,
+ "rstrip": false,
+ "single_word": false,
+ "special": true
+ },
+ "2": {
+ "content": "<|endoftext|>",
+ "lstrip": false,
+ "normalized": false,
+ "rstrip": false,
+ "single_word": false,
+ "special": true
+ },
+ "64000": {
+ "content": "<s>",
+ "lstrip": false,
+ "normalized": false,
+ "rstrip": false,
+ "single_word": false,
+ "special": false
+ },
+ "64001": {
+ "content": "</s>",
+ "lstrip": false,
+ "normalized": false,
+ "rstrip": false,
+ "single_word": false,
+ "special": false
+ }
+ },
+ "bos_token": "<|startoftext|>",
+ "clean_up_tokenization_spaces": false,
+ "eos_token": "<|endoftext|>",
+ "legacy": false,
+ "model_max_length": 200000,
+ "pad_token": "<unk>",
+ "padding_side": "right",
+ "sp_model_kwargs": {},
+ "spaces_between_special_tokens": false,
+ "tokenizer_class": "LlamaTokenizer",
+ "truncation_side": "right",
+ "unk_token": "<unk>",
+ "use_default_system_prompt": false
+}