Synatra-7B-v0.3-RP / README.md
maywell's picture
Adding Evaluation Results (#2)
a994747
|
raw
history blame
3.19 kB
metadata
language:
  - ko
library_name: transformers
pipeline_tag: text-generation
license: cc-by-nc-4.0

Synatra-7B-v0.3-RP🐧

Synatra-7B-v0.3-RP

Support Me

μ‹œλ‚˜νŠΈλΌλŠ” 개인 ν”„λ‘œμ νŠΈλ‘œ, 1인의 μžμ›μœΌλ‘œ 개발되고 μžˆμŠ΅λ‹ˆλ‹€. λͺ¨λΈμ΄ λ§ˆμŒμ— λ“œμ…¨λ‹€λ©΄ μ•½κ°„μ˜ 연ꡬ비 지원은 μ–΄λ–¨κΉŒμš”? Buy me a Coffee

Wanna be a sponser? Contact me on Telegram AlzarTakkarsen

License

This model is strictly non-commercial (cc-by-nc-4.0) use only. The "Model" is completely free (ie. base model, derivates, merges/mixes) to use for non-commercial purposes as long as the the included cc-by-nc-4.0 license in any parent repository, and the non-commercial use statute remains, regardless of other models' licences. The licence can be changed after new model released. If you are to use this model for commercial purpose, Contact me.

Model Details

Base Model
mistralai/Mistral-7B-Instruct-v0.1

Trained On
A6000 48GB * 8

Instruction format

It follows ChatML format.

TODO

  • RP 기반 νŠœλ‹ λͺ¨λΈ μ œμž‘ βœ…
  • 데이터셋 μ •μ œ βœ…
  • μ–Έμ–΄ 이해λŠ₯λ ₯ κ°œμ„ 
  • 상식 보완 βœ…
  • ν† ν¬λ‚˜μ΄μ € λ³€κ²½

Model Benchmark

Ko-LLM-Leaderboard

On Benchmarking...

Implementation Code

Since, chat_template already contains insturction format above. You can use the code below.

from transformers import AutoModelForCausalLM, AutoTokenizer

device = "cuda" # the device to load the model onto

model = AutoModelForCausalLM.from_pretrained("maywell/Synatra-7B-v0.3-RP")
tokenizer = AutoTokenizer.from_pretrained("maywell/Synatra-7B-v0.3-RP")

messages = [
    {"role": "user", "content": "λ°”λ‚˜λ‚˜λŠ” μ›λž˜ ν•˜μ–€μƒ‰μ΄μ•Ό?"},
]

encodeds = tokenizer.apply_chat_template(messages, return_tensors="pt")

model_inputs = encodeds.to(device)
model.to(device)

generated_ids = model.generate(model_inputs, max_new_tokens=1000, do_sample=True)
decoded = tokenizer.batch_decode(generated_ids)
print(decoded[0])

Why It's benchmark score is lower than preview version?

Apparently, Preview model uses Alpaca Style prompt which has no pre-fix. But ChatML do.

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric Value
Avg. 57.38
ARC (25-shot) 62.2
HellaSwag (10-shot) 82.29
MMLU (5-shot) 60.8
TruthfulQA (0-shot) 52.64
Winogrande (5-shot) 76.48
GSM8K (5-shot) 21.15
DROP (3-shot) 46.06