Hugging Face's logo

ModelCloud
/
Meta-Llama-3.1-70B-Instruct-gptq-4bit

Text Generation
Transformers
Safetensors
llama
gptq
4bit
int4
gptqmodel
modelcloud
llama-3.1
70b
instruct
conversational
text-generation-inference
Inference Endpoints
4-bit precision
Model card Files Community
5
Qubitium commited on
Commit
8936e4b
•
1 Parent(s): 0d299c8

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +93 -2
README.md CHANGED
@@ -27,7 +27,7 @@ This model has been quantized using [GPTQModel](https://github.com/ModelCloud/GP
27
  - **meta**：
28
  - **quantizer**: "gptqmodel:0.9.9-dev0"
29
 
30
- **Here is an example:**
31
  ```python
32
  from transformers import AutoTokenizer
33
  from gptqmodel import GPTQModel
@@ -45,4 +45,95 @@ outputs = model.generate(input_ids=input_tensor.to(model.device), max_new_tokens
45
  result = tokenizer.decode(outputs[0][input_tensor.shape[1]:], skip_special_tokens=True)
46
 
47
  print(result)
48
- ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
27
  - **meta**：
28
  - **quantizer**: "gptqmodel:0.9.9-dev0"
29
 
30
+ ## Example:
31
  ```python
32
  from transformers import AutoTokenizer
33
  from gptqmodel import GPTQModel
 
45
  result = tokenizer.decode(outputs[0][input_tensor.shape[1]:], skip_special_tokens=True)
46
 
47
  print(result)
48
+ ```
49
+
50
+ ## lm-eval benchmark:
51
+
52
+ | Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr|
53
+ |---------------------------------------|------:|------|-----:|----------|---|-----:|---|-----:|
54
+ |arc_challenge | 1|none | 0|acc |↑ |0.6186|± |0.0142|
55
+ | | |none | 0|acc_norm |↑ |0.6297|± |0.0141|
56
+ |arc_easy | 1|none | 0|acc |↑ |0.8628|± |0.0071|
57
+ | | |none | 0|acc_norm |↑ |0.8338|± |0.0076|
58
+ |boolq | 2|none | 0|acc |↑ |0.8761|± |0.0058|
59
+ |hellaswag | 1|none | 0|acc |↑ |0.6463|± |0.0048|
60
+ | | |none | 0|acc_norm |↑ |0.8389|± |0.0037|
61
+ |lambada_openai | 1|none | 0|acc |↑ |0.7561|± |0.0060|
62
+ | | |none | 0|perplexity|↓ |3.0311|± |0.0639|
63
+ |mmlu | 1|none | |acc |↑ |0.8100|± |0.0032|
64
+ | - humanities | 1|none | |acc |↑ |0.7981|± |0.0057|
65
+ | - formal_logic | 0|none | 0|acc |↑ |0.6349|± |0.0431|
66
+ | - high_school_european_history | 0|none | 0|acc |↑ |0.8545|± |0.0275|
67
+ | - high_school_us_history | 0|none | 0|acc |↑ |0.9412|± |0.0165|
68
+ | - high_school_world_history | 0|none | 0|acc |↑ |0.9198|± |0.0177|
69
+ | - international_law | 0|none | 0|acc |↑ |0.9008|± |0.0273|
70
+ | - jurisprudence | 0|none | 0|acc |↑ |0.8796|± |0.0315|
71
+ | - logical_fallacies | 0|none | 0|acc |↑ |0.8650|± |0.0268|
72
+ | - moral_disputes | 0|none | 0|acc |↑ |0.8266|± |0.0204|
73
+ | - moral_scenarios | 0|none | 0|acc |↑ |0.8559|± |0.0117|
74
+ | - philosophy | 0|none | 0|acc |↑ |0.8360|± |0.0210|
75
+ | - prehistory | 0|none | 0|acc |↑ |0.8827|± |0.0179|
76
+ | - professional_law | 0|none | 0|acc |↑ |0.6675|± |0.0120|
77
+ | - world_religions | 0|none | 0|acc |↑ |0.9181|± |0.0210|
78
+ | - other | 1|none | |acc |↑ |0.8304|± |0.0064|
79
+ | - business_ethics | 0|none | 0|acc |↑ |0.7900|± |0.0409|
80
+ | - clinical_knowledge | 0|none | 0|acc |↑ |0.8566|± |0.0216|
81
+ | - college_medicine | 0|none | 0|acc |↑ |0.7630|± |0.0324|
82
+ | - global_facts | 0|none | 0|acc |↑ |0.5800|± |0.0496|
83
+ | - human_aging | 0|none | 0|acc |↑ |0.8206|± |0.0257|
84
+ | - management | 0|none | 0|acc |↑ |0.8835|± |0.0318|
85
+ | - marketing | 0|none | 0|acc |↑ |0.9231|± |0.0175|
86
+ | - medical_genetics | 0|none | 0|acc |↑ |0.9400|± |0.0239|
87
+ | - miscellaneous | 0|none | 0|acc |↑ |0.9144|± |0.0100|
88
+ | - nutrition | 0|none | 0|acc |↑ |0.8660|± |0.0195|
89
+ | - professional_accounting | 0|none | 0|acc |↑ |0.6454|± |0.0285|
90
+ | - professional_medicine | 0|none | 0|acc |↑ |0.8971|± |0.0185|
91
+ | - virology | 0|none | 0|acc |↑ |0.5602|± |0.0386|
92
+ | - social sciences | 1|none | |acc |↑ |0.8736|± |0.0059|
93
+ | - econometrics | 0|none | 0|acc |↑ |0.7018|± |0.0430|
94
+ | - high_school_geography | 0|none | 0|acc |↑ |0.9242|± |0.0189|
95
+ | - high_school_government_and_politics| 0|none | 0|acc |↑ |0.9741|± |0.0115|
96
+ | - high_school_macroeconomics | 0|none | 0|acc |↑ |0.8410|± |0.0185|
97
+ | - high_school_microeconomics | 0|none | 0|acc |↑ |0.8992|± |0.0196|
98
+ | - high_school_psychology | 0|none | 0|acc |↑ |0.9229|± |0.0114|
99
+ | - human_sexuality | 0|none | 0|acc |↑ |0.8779|± |0.0287|
100
+ | - professional_psychology | 0|none | 0|acc |↑ |0.8497|± |0.0145|
101
+ | - public_relations | 0|none | 0|acc |↑ |0.7273|± |0.0427|
102
+ | - security_studies | 0|none | 0|acc |↑ |0.8163|± |0.0248|
103
+ | - sociology | 0|none | 0|acc |↑ |0.9154|± |0.0197|
104
+ | - us_foreign_policy | 0|none | 0|acc |↑ |0.9300|± |0.0256|
105
+ | - stem | 1|none | |acc |↑ |0.7456|± |0.0075|
106
+ | - abstract_algebra | 0|none | 0|acc |↑ |0.6300|± |0.0485|
107
+ | - anatomy | 0|none | 0|acc |↑ |0.7926|± |0.0350|
108
+ | - astronomy | 0|none | 0|acc |↑ |0.8947|± |0.0250|
109
+ | - college_biology | 0|none | 0|acc |↑ |0.9444|± |0.0192|
110
+ | - college_chemistry | 0|none | 0|acc |↑ |0.5800|± |0.0496|
111
+ | - college_computer_science | 0|none | 0|acc |↑ |0.6700|± |0.0473|
112
+ | - college_mathematics | 0|none | 0|acc |↑ |0.5400|± |0.0501|
113
+ | - college_physics | 0|none | 0|acc |↑ |0.6275|± |0.0481|
114
+ | - computer_security | 0|none | 0|acc |↑ |0.8200|± |0.0386|
115
+ | - conceptual_physics | 0|none | 0|acc |↑ |0.7830|± |0.0269|
116
+ | - electrical_engineering | 0|none | 0|acc |↑ |0.7862|± |0.0342|
117
+ | - elementary_mathematics | 0|none | 0|acc |↑ |0.7593|± |0.0220|
118
+ | - high_school_biology | 0|none | 0|acc |↑ |0.9194|± |0.0155|
119
+ | - high_school_chemistry | 0|none | 0|acc |↑ |0.7143|± |0.0318|
120
+ | - high_school_computer_science | 0|none | 0|acc |↑ |0.9200|± |0.0273|
121
+ | - high_school_mathematics | 0|none | 0|acc |↑ |0.5185|± |0.0305|
122
+ | - high_school_physics | 0|none | 0|acc |↑ |0.6556|± |0.0388|
123
+ | - high_school_statistics | 0|none | 0|acc |↑ |0.7361|± |0.0301|
124
+ | - machine_learning | 0|none | 0|acc |↑ |0.7054|± |0.0433|
125
+ |openbookqa | 1|none | 0|acc |↑ |0.3660|± |0.0216|
126
+ | | |none | 0|acc_norm |↑ |0.4620|± |0.0223|
127
+ |piqa | 1|none | 0|acc |↑ |0.8264|± |0.0088|
128
+ | | |none | 0|acc_norm |↑ |0.8319|± |0.0087|
129
+ |rte | 1|none | 0|acc |↑ |0.7184|± |0.0271|
130
+ |truthfulqa_mc1 | 2|none | 0|acc |↑ |0.3917|± |0.0171|
131
+ |winogrande | 1|none | 0|acc |↑ |0.7924|± |0.0114|
132
+
133
+ | Groups |Version|Filter|n-shot|Metric| |Value | |Stderr|
134
+ |------------------|------:|------|------|------|---|-----:|---|-----:|
135
+ |mmlu | 1|none | |acc |↑ |0.8100|± |0.0032|
136
+ | - humanities | 1|none | |acc |↑ |0.7981|± |0.0057|
137
+ | - other | 1|none | |acc |↑ |0.8304|± |0.0064|
138
+ | - social sciences| 1|none | |acc |↑ |0.8736|± |0.0059|
139
+ | - stem | 1|none | |acc |↑ |0.7456|± |0.0075|