Update README.md
Browse files
README.md
CHANGED
@@ -1,3 +1,96 @@
|
|
1 |
---
|
|
|
|
|
|
|
2 |
license: llama2
|
|
|
|
|
|
|
|
|
|
|
|
|
3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
---
|
2 |
+
base_model:
|
3 |
+
- elyza/ELYZA-japanese-Llama-2-13b
|
4 |
+
- elyza/ELYZA-japanese-Llama-2-13b-instruct
|
5 |
license: llama2
|
6 |
+
language:
|
7 |
+
- ja
|
8 |
+
tags:
|
9 |
+
- mergekit
|
10 |
+
- merge
|
11 |
+
- MoE
|
12 |
---
|
13 |
+
# ELYZA-japanese-Llama-2-MoE-2x13B-v0.1
|
14 |
+
[**English description here**](#description)
|
15 |
+
|
16 |
+
|
17 |
+
## 概要
|
18 |
+
Llama-2ベースの学習済み日本語モデルである[elyza/ELYZA-japanese-Llama-2-13b](https://huggingface.co/elyza/ELYZA-japanese-Llama-2-13b)と、そのinstruction tuningモデルである[elyza/ELYZA-japanese-Llama-2-13b-instruct](https://huggingface.co/elyza/ELYZA-japanese-Llama-2-13b-instruct)
|
19 |
+
を、[mergekit](https://github.com/cg123/mergekit)を使ってMoEを行い作成したモデルです。
|
20 |
+
|
21 |
+
[GGUF版はこちら](https://huggingface.co/Aratako/ELYZA-japanese-Llama-2-MoE-2x13B-v0.1-GGUF)
|
22 |
+
|
23 |
+
以下2モデルを利用しています。
|
24 |
+
- [elyza/ELYZA-japanese-Llama-2-13b](https://huggingface.co/elyza/ELYZA-japanese-Llama-2-13b)
|
25 |
+
- [elyza/ELYZA-japanese-Llama-2-13b-instruct](https://huggingface.co/elyza/ELYZA-japanese-Llama-2-13b-instruct)
|
26 |
+
|
27 |
+
## ライセンス
|
28 |
+
元モデルの通り、Llama2ライセンスを継承します。
|
29 |
+
|
30 |
+
Llama 2 is licensed under the LLAMA 2 Community License, Copyright (c) Meta Platforms, Inc. All Rights Reserved.
|
31 |
+
|
32 |
+
## ベンチマーク
|
33 |
+
ベースとしたELYZA-japanese-Llama-2-13b-instructと本モデルの[japanese-mt-bench](https://github.com/Stability-AI/FastChat/tree/jp-stable/fastchat/llm_judge)の結果は以下の通りです。
|
34 |
+
(シングルターン)
|
35 |
+
|Model|Size|Coding|Extraction|Humanities|Math|Reasoning|Roleplay|STEM|Writing|avg_score|
|
36 |
+
|---|---|---|---|---|---|---|---|---|---|---|
|
37 |
+
| ELYZA-japanese-Llama-2-13b-instruct | 13B | **3.7** | 6.0 | **6.6** | 2.4 | 2.5 | 5.2 | 5.8 | 7.2 | **4.925** |
|
38 |
+
| This model | 2x13B | **3.7** | **6.9** | 6.3 | **3.7** | **4.4** | **6.0** | **7.0** | **7.4** | **5.675** |
|
39 |
+
|
40 |
+
![レーダーチャート](./japanese_mt_bench.png)
|
41 |
+
|
42 |
+
**ベンチマークに使用したプロンプト**
|
43 |
+
```
|
44 |
+
"""<s>[INST] <<SYS>>
|
45 |
+
あなたは誠実で優秀な日本人のアシスタントです。
|
46 |
+
<</SYS>>
|
47 |
+
|
48 |
+
{instruction} [/INST]"""
|
49 |
+
```
|
50 |
+
|
51 |
+
## Description
|
52 |
+
This model is created using MoE (Mixture of Experts) through mergekit based on [elyza/ELYZA-japanese-Llama-2-13b](https://huggingface.co/elyza/ELYZA-japanese-Llama-2-13b) and [elyza/ELYZA-japanese-Llama-2-13b-instruct](https://huggingface.co/elyza/ELYZA-japanese-Llama-2-13b-instruct).
|
53 |
+
|
54 |
+
[Click here for the GGUF version](https://huggingface.co/Aratako/ELYZA-japanese-Llama-2-MoE-2x13B-v0.1-GGUF)
|
55 |
+
|
56 |
+
It utilizes the following two models:
|
57 |
+
- [elyza/ELYZA-japanese-Llama-2-13b](https://huggingface.co/elyza/ELYZA-japanese-Llama-2-13b)
|
58 |
+
- [elyza/ELYZA-japanese-Llama-2-13b-instruct](https://huggingface.co/elyza/ELYZA-japanese-Llama-2-13b-instruct)
|
59 |
+
|
60 |
+
## License
|
61 |
+
This model inherit the Llama2 license.
|
62 |
+
|
63 |
+
Llama 2 is licensed under the LLAMA 2 Community License, Copyright (c) Meta Platforms, Inc. All Rights Reserved.
|
64 |
+
|
65 |
+
## Benchmark
|
66 |
+
The results of this model and the base ELYZA-japanese-Llama-2-13b-instruct on japanese-mt-bench are as follows.
|
67 |
+
(Single turn)
|
68 |
+
|Model|Size|Coding|Extraction|Humanities|Math|Reasoning|Roleplay|STEM|Writing|avg_score|
|
69 |
+
|---|---|---|---|---|---|---|---|---|---|---|
|
70 |
+
| ELYZA-japanese-Llama-2-13b-instruct | 13B | **3.7** | 6.0 | **6.6** | 2.4 | 2.5 | 5.2 | 5.8 | 7.2 | **4.925** |
|
71 |
+
| This model | 2x13B | **3.7** | **6.9** | 6.3 | **3.7** | **4.4** | **6.0** | **7.0** | **7.4** | **5.675** |
|
72 |
+
|
73 |
+
![レーダーチャート](./japanese_mt_bench.png)
|
74 |
+
|
75 |
+
**Prompt used for benchmark**
|
76 |
+
```
|
77 |
+
"""<s>[INST] <<SYS>>
|
78 |
+
あなたは誠実で優秀な日本人のアシスタントです。
|
79 |
+
<</SYS>>
|
80 |
+
|
81 |
+
{instruction} [/INST]"""
|
82 |
+
```
|
83 |
+
|
84 |
+
## Merge config
|
85 |
+
[mergekit_config.yml](./mergekit_moe_config.yml)
|
86 |
+
```yaml
|
87 |
+
base_model: ./ELYZA-japanese-Llama-2-13b-instruct
|
88 |
+
gate_mode: random
|
89 |
+
dtype: bfloat16
|
90 |
+
experts:
|
91 |
+
- source_model: ./ELYZA-japanese-Llama-2-13b-instruct
|
92 |
+
positive_prompts: []
|
93 |
+
- source_model: ./ELYZA-japanese-Llama-2-13b
|
94 |
+
positive_prompts: []
|
95 |
+
tokenizer_source: model:./ELYZA-japanese-Llama-2-13b-instruct
|
96 |
+
```
|