metadata
tags:
- gptq
- 4bit
- int4
- gptqmodel
- modelcloud
- llama-3.1
- 8b
- instruct
license: llama3.1
This model has been quantized using GPTQModel.
- bits: 4
- group_size: 128
- desc_act: true
- static_groups: false
- sym: true
- lm_head: false
- damp_percent: 0.005
- true_sequential: true
- model_name_or_path: ""
- model_file_base_name: "model"
- quant_method: "gptq"
- checkpoint_format: "gptq"
- meta：
- quantizer: "gptqmodel:0.9.9-dev0"
Example:
from transformers import AutoTokenizer
from gptqmodel import GPTQModel
model_name = "ModelCloud/Meta-Llama-3.1-8B-Instruct-gptq-4bit"
prompt = [{"role": "user", "content": "I am in Shanghai, preparing to visit the natural history museum. Can you tell me the best way to"}]
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = GPTQModel.from_quantized(model_name)
input_tensor = tokenizer.apply_chat_template(prompt, add_generation_prompt=True, return_tensors="pt")
outputs = model.generate(input_ids=input_tensor.to(model.device), max_new_tokens=100)
result = tokenizer.decode(outputs[0][input_tensor.shape[1]:], skip_special_tokens=True)
print(result)
lm-eval benchmark
| Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr|
|---------------------------------------|------:|------|-----:|----------|---|-----:|---|-----:|
|arc_challenge | 1|none | 0|acc |↑ |0.4889|± |0.0146|
| | |none | 0|acc_norm |↑ |0.5265|± |0.0146|
|arc_easy | 1|none | 0|acc |↑ |0.7908|± |0.0083|
| | |none | 0|acc_norm |↑ |0.7702|± |0.0086|
|boolq | 2|none | 0|acc |↑ |0.8404|± |0.0064|
|hellaswag | 1|none | 0|acc |↑ |0.5748|± |0.0049|
| | |none | 0|acc_norm |↑ |0.7718|± |0.0042|
|lambada_openai | 1|none | 0|acc |↑ |0.7110|± |0.0063|
| | |none | 0|perplexity|↓ |4.0554|± |0.0955|
|mmlu | 1|none | |acc |↑ |0.6411|± |0.0038|
| - humanities | 1|none | |acc |↑ |0.5896|± |0.0068|
| - formal_logic | 0|none | 0|acc |↑ |0.4762|± |0.0447|
| - high_school_european_history | 0|none | 0|acc |↑ |0.7273|± |0.0348|
| - high_school_us_history | 0|none | 0|acc |↑ |0.8088|± |0.0276|
| - high_school_world_history | 0|none | 0|acc |↑ |0.8270|± |0.0246|
| - international_law | 0|none | 0|acc |↑ |0.7851|± |0.0375|
| - jurisprudence | 0|none | 0|acc |↑ |0.7593|± |0.0413|
| - logical_fallacies | 0|none | 0|acc |↑ |0.7607|± |0.0335|
| - moral_disputes | 0|none | 0|acc |↑ |0.7197|± |0.0242|
| - moral_scenarios | 0|none | 0|acc |↑ |0.4045|± |0.0164|
| - philosophy | 0|none | 0|acc |↑ |0.7074|± |0.0258|
| - prehistory | 0|none | 0|acc |↑ |0.6975|± |0.0256|
| - professional_law | 0|none | 0|acc |↑ |0.4817|± |0.0128|
| - world_religions | 0|none | 0|acc |↑ |0.7953|± |0.0309|
| - other | 1|none | |acc |↑ |0.7100|± |0.0078|
| - business_ethics | 0|none | 0|acc |↑ |0.6700|± |0.0473|
| - clinical_knowledge | 0|none | 0|acc |↑ |0.7660|± |0.0261|
| - college_medicine | 0|none | 0|acc |↑ |0.6590|± |0.0361|
| - global_facts | 0|none | 0|acc |↑ |0.3600|± |0.0482|
| - human_aging | 0|none | 0|acc |↑ |0.6547|± |0.0319|
| - management | 0|none | 0|acc |↑ |0.8447|± |0.0359|
| - marketing | 0|none | 0|acc |↑ |0.8803|± |0.0213|
| - medical_genetics | 0|none | 0|acc |↑ |0.7100|± |0.0456|
| - miscellaneous | 0|none | 0|acc |↑ |0.8161|± |0.0139|
| - nutrition | 0|none | 0|acc |↑ |0.7124|± |0.0259|
| - professional_accounting | 0|none | 0|acc |↑ |0.4787|± |0.0298|
| - professional_medicine | 0|none | 0|acc |↑ |0.7279|± |0.0270|
| - virology | 0|none | 0|acc |↑ |0.5181|± |0.0389|
| - social sciences | 1|none | |acc |↑ |0.7312|± |0.0078|
| - econometrics | 0|none | 0|acc |↑ |0.4035|± |0.0462|
| - high_school_geography | 0|none | 0|acc |↑ |0.8232|± |0.0272|
| - high_school_government_and_politics| 0|none | 0|acc |↑ |0.8653|± |0.0246|
| - high_school_macroeconomics | 0|none | 0|acc |↑ |0.6128|± |0.0247|
| - high_school_microeconomics | 0|none | 0|acc |↑ |0.7227|± |0.0291|
| - high_school_psychology | 0|none | 0|acc |↑ |0.8422|± |0.0156|
| - human_sexuality | 0|none | 0|acc |↑ |0.7634|± |0.0373|
| - professional_psychology | 0|none | 0|acc |↑ |0.6585|± |0.0192|
| - public_relations | 0|none | 0|acc |↑ |0.6182|± |0.0465|
| - security_studies | 0|none | 0|acc |↑ |0.7306|± |0.0284|
| - sociology | 0|none | 0|acc |↑ |0.8358|± |0.0262|
| - us_foreign_policy | 0|none | 0|acc |↑ |0.8600|± |0.0349|
| - stem | 1|none | |acc |↑ |0.5623|± |0.0085|
| - abstract_algebra | 0|none | 0|acc |↑ |0.3800|± |0.0488|
| - anatomy | 0|none | 0|acc |↑ |0.6222|± |0.0419|
| - astronomy | 0|none | 0|acc |↑ |0.7039|± |0.0372|
| - college_biology | 0|none | 0|acc |↑ |0.7778|± |0.0348|
| - college_chemistry | 0|none | 0|acc |↑ |0.5400|± |0.0501|
| - college_computer_science | 0|none | 0|acc |↑ |0.5300|± |0.0502|
| - college_mathematics | 0|none | 0|acc |↑ |0.3200|± |0.0469|
| - college_physics | 0|none | 0|acc |↑ |0.4608|± |0.0496|
| - computer_security | 0|none | 0|acc |↑ |0.7800|± |0.0416|
| - conceptual_physics | 0|none | 0|acc |↑ |0.5617|± |0.0324|
| - electrical_engineering | 0|none | 0|acc |↑ |0.6138|± |0.0406|
| - elementary_mathematics | 0|none | 0|acc |↑ |0.4365|± |0.0255|
| - high_school_biology | 0|none | 0|acc |↑ |0.7839|± |0.0234|
| - high_school_chemistry | 0|none | 0|acc |↑ |0.5665|± |0.0349|
| - high_school_computer_science | 0|none | 0|acc |↑ |0.6600|± |0.0476|
| - high_school_mathematics | 0|none | 0|acc |↑ |0.4407|± |0.0303|
| - high_school_physics | 0|none | 0|acc |↑ |0.4371|± |0.0405|
| - high_school_statistics | 0|none | 0|acc |↑ |0.5602|± |0.0339|
| - machine_learning | 0|none | 0|acc |↑ |0.4643|± |0.0473|
|openbookqa | 1|none | 0|acc |↑ |0.3180|± |0.0208|
| | |none | 0|acc_norm |↑ |0.4140|± |0.0220|
|piqa | 1|none | 0|acc |↑ |0.7878|± |0.0095|
| | |none | 0|acc_norm |↑ |0.7971|± |0.0094|
|rte | 1|none | 0|acc |↑ |0.6751|± |0.0282|
|truthfulqa_mc1 | 2|none | 0|acc |↑ |0.3403|± |0.0166|
|winogrande | 1|none | 0|acc |↑ |0.7206|± |0.0126|
| Groups |Version|Filter|n-shot|Metric| |Value | |Stderr|
|------------------|------:|------|------|------|---|-----:|---|-----:|
|mmlu | 1|none | |acc |↑ |0.6411|± |0.0038|
| - humanities | 1|none | |acc |↑ |0.5896|± |0.0068|
| - other | 1|none | |acc |↑ |0.7100|± |0.0078|
| - social sciences| 1|none | |acc |↑ |0.7312|± |0.0078|
| - stem | 1|none | |acc |↑ |0.5623|± |0.0085|