File size: 9,791 Bytes
ab84b16
 
 
 
56e505d
ab84b16
 
 
 
 
057d00e
ab84b16
0964040
 
 
 
 
 
 
 
ef2cb41
0964040
 
 
 
 
 
062bc45
 
1810a8c
062bc45
 
 
 
 
 
 
 
 
 
 
 
37e8e60
 
 
 
 
1810a8c
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
062bc45
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
---
tags:
- gptq
- 4bit
- int4
- gptqmodel
- modelcloud
- llama-3.1
- 8b
- instruct
license: llama3.1
---
This model has been quantized using [GPTQModel](https://github.com/ModelCloud/GPTQModel).

- **bits**: 4
- **group_size**: 128
- **desc_act**: true
- **static_groups**: false
- **sym**: true
- **lm_head**: false
- **damp_percent**: 0.005
- **true_sequential**: true
- **model_name_or_path**: ""
- **model_file_base_name**: "model"
- **quant_method**: "gptq"
- **checkpoint_format**: "gptq"
- **meta**  - **quantizer**: "gptqmodel:0.9.9-dev0"

## Example:
```python
from transformers import AutoTokenizer
from gptqmodel import GPTQModel

model_name = "ModelCloud/Meta-Llama-3.1-8B-Instruct-gptq-4bit"

prompt = [{"role": "user", "content": "I am in Shanghai, preparing to visit the natural history museum. Can you tell me the best way to"}]

tokenizer = AutoTokenizer.from_pretrained(model_name)

model = GPTQModel.from_quantized(model_name)

input_tensor = tokenizer.apply_chat_template(prompt, add_generation_prompt=True, return_tensors="pt")
outputs = model.generate(input_ids=input_tensor.to(model.device), max_new_tokens=100)
result = tokenizer.decode(outputs[0][input_tensor.shape[1]:], skip_special_tokens=True)

print(result)
```

## lm-eval benchmark

```
|                 Tasks                 |Version|Filter|n-shot|  Metric  |   |Value |   |Stderr|
|---------------------------------------|------:|------|-----:|----------|---|-----:|---|-----:|
|arc_challenge                          |      1|none  |     0|acc       |↑  |0.4889|±  |0.0146|
|                                       |       |none  |     0|acc_norm  |↑  |0.5265|±  |0.0146|
|arc_easy                               |      1|none  |     0|acc       |↑  |0.7908|±  |0.0083|
|                                       |       |none  |     0|acc_norm  |↑  |0.7702|±  |0.0086|
|boolq                                  |      2|none  |     0|acc       |↑  |0.8404|±  |0.0064|
|hellaswag                              |      1|none  |     0|acc       |↑  |0.5748|±  |0.0049|
|                                       |       |none  |     0|acc_norm  |↑  |0.7718|±  |0.0042|
|lambada_openai                         |      1|none  |     0|acc       |↑  |0.7110|±  |0.0063|
|                                       |       |none  |     0|perplexity|↓  |4.0554|±  |0.0955|
|mmlu                                   |      1|none  |      |acc       |↑  |0.6411|±  |0.0038|
| - humanities                          |      1|none  |      |acc       |↑  |0.5896|±  |0.0068|
|  - formal_logic                       |      0|none  |     0|acc       |↑  |0.4762|±  |0.0447|
|  - high_school_european_history       |      0|none  |     0|acc       |↑  |0.7273|±  |0.0348|
|  - high_school_us_history             |      0|none  |     0|acc       |↑  |0.8088|±  |0.0276|
|  - high_school_world_history          |      0|none  |     0|acc       |↑  |0.8270|±  |0.0246|
|  - international_law                  |      0|none  |     0|acc       |↑  |0.7851|±  |0.0375|
|  - jurisprudence                      |      0|none  |     0|acc       |↑  |0.7593|±  |0.0413|
|  - logical_fallacies                  |      0|none  |     0|acc       |↑  |0.7607|±  |0.0335|
|  - moral_disputes                     |      0|none  |     0|acc       |↑  |0.7197|±  |0.0242|
|  - moral_scenarios                    |      0|none  |     0|acc       |↑  |0.4045|±  |0.0164|
|  - philosophy                         |      0|none  |     0|acc       |↑  |0.7074|±  |0.0258|
|  - prehistory                         |      0|none  |     0|acc       |↑  |0.6975|±  |0.0256|
|  - professional_law                   |      0|none  |     0|acc       |↑  |0.4817|±  |0.0128|
|  - world_religions                    |      0|none  |     0|acc       |↑  |0.7953|±  |0.0309|
| - other                               |      1|none  |      |acc       |↑  |0.7100|±  |0.0078|
|  - business_ethics                    |      0|none  |     0|acc       |↑  |0.6700|±  |0.0473|
|  - clinical_knowledge                 |      0|none  |     0|acc       |↑  |0.7660|±  |0.0261|
|  - college_medicine                   |      0|none  |     0|acc       |↑  |0.6590|±  |0.0361|
|  - global_facts                       |      0|none  |     0|acc       |↑  |0.3600|±  |0.0482|
|  - human_aging                        |      0|none  |     0|acc       |↑  |0.6547|±  |0.0319|
|  - management                         |      0|none  |     0|acc       |↑  |0.8447|±  |0.0359|
|  - marketing                          |      0|none  |     0|acc       |↑  |0.8803|±  |0.0213|
|  - medical_genetics                   |      0|none  |     0|acc       |↑  |0.7100|±  |0.0456|
|  - miscellaneous                      |      0|none  |     0|acc       |↑  |0.8161|±  |0.0139|
|  - nutrition                          |      0|none  |     0|acc       |↑  |0.7124|±  |0.0259|
|  - professional_accounting            |      0|none  |     0|acc       |↑  |0.4787|±  |0.0298|
|  - professional_medicine              |      0|none  |     0|acc       |↑  |0.7279|±  |0.0270|
|  - virology                           |      0|none  |     0|acc       |↑  |0.5181|±  |0.0389|
| - social sciences                     |      1|none  |      |acc       |↑  |0.7312|±  |0.0078|
|  - econometrics                       |      0|none  |     0|acc       |↑  |0.4035|±  |0.0462|
|  - high_school_geography              |      0|none  |     0|acc       |↑  |0.8232|±  |0.0272|
|  - high_school_government_and_politics|      0|none  |     0|acc       |↑  |0.8653|±  |0.0246|
|  - high_school_macroeconomics         |      0|none  |     0|acc       |↑  |0.6128|±  |0.0247|
|  - high_school_microeconomics         |      0|none  |     0|acc       |↑  |0.7227|±  |0.0291|
|  - high_school_psychology             |      0|none  |     0|acc       |↑  |0.8422|±  |0.0156|
|  - human_sexuality                    |      0|none  |     0|acc       |↑  |0.7634|±  |0.0373|
|  - professional_psychology            |      0|none  |     0|acc       |↑  |0.6585|±  |0.0192|
|  - public_relations                   |      0|none  |     0|acc       |↑  |0.6182|±  |0.0465|
|  - security_studies                   |      0|none  |     0|acc       |↑  |0.7306|±  |0.0284|
|  - sociology                          |      0|none  |     0|acc       |↑  |0.8358|±  |0.0262|
|  - us_foreign_policy                  |      0|none  |     0|acc       |↑  |0.8600|±  |0.0349|
| - stem                                |      1|none  |      |acc       |↑  |0.5623|±  |0.0085|
|  - abstract_algebra                   |      0|none  |     0|acc       |↑  |0.3800|±  |0.0488|
|  - anatomy                            |      0|none  |     0|acc       |↑  |0.6222|±  |0.0419|
|  - astronomy                          |      0|none  |     0|acc       |↑  |0.7039|±  |0.0372|
|  - college_biology                    |      0|none  |     0|acc       |↑  |0.7778|±  |0.0348|
|  - college_chemistry                  |      0|none  |     0|acc       |↑  |0.5400|±  |0.0501|
|  - college_computer_science           |      0|none  |     0|acc       |↑  |0.5300|±  |0.0502|
|  - college_mathematics                |      0|none  |     0|acc       |↑  |0.3200|±  |0.0469|
|  - college_physics                    |      0|none  |     0|acc       |↑  |0.4608|±  |0.0496|
|  - computer_security                  |      0|none  |     0|acc       |↑  |0.7800|±  |0.0416|
|  - conceptual_physics                 |      0|none  |     0|acc       |↑  |0.5617|±  |0.0324|
|  - electrical_engineering             |      0|none  |     0|acc       |↑  |0.6138|±  |0.0406|
|  - elementary_mathematics             |      0|none  |     0|acc       |↑  |0.4365|±  |0.0255|
|  - high_school_biology                |      0|none  |     0|acc       |↑  |0.7839|±  |0.0234|
|  - high_school_chemistry              |      0|none  |     0|acc       |↑  |0.5665|±  |0.0349|
|  - high_school_computer_science       |      0|none  |     0|acc       |↑  |0.6600|±  |0.0476|
|  - high_school_mathematics            |      0|none  |     0|acc       |↑  |0.4407|±  |0.0303|
|  - high_school_physics                |      0|none  |     0|acc       |↑  |0.4371|±  |0.0405|
|  - high_school_statistics             |      0|none  |     0|acc       |↑  |0.5602|±  |0.0339|
|  - machine_learning                   |      0|none  |     0|acc       |↑  |0.4643|±  |0.0473|
|openbookqa                             |      1|none  |     0|acc       |↑  |0.3180|±  |0.0208|
|                                       |       |none  |     0|acc_norm  |↑  |0.4140|±  |0.0220|
|piqa                                   |      1|none  |     0|acc       |↑  |0.7878|±  |0.0095|
|                                       |       |none  |     0|acc_norm  |↑  |0.7971|±  |0.0094|
|rte                                    |      1|none  |     0|acc       |↑  |0.6751|±  |0.0282|
|truthfulqa_mc1                         |      2|none  |     0|acc       |↑  |0.3403|±  |0.0166|
|winogrande                             |      1|none  |     0|acc       |↑  |0.7206|±  |0.0126|

|      Groups      |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|------------------|------:|------|------|------|---|-----:|---|-----:|
|mmlu              |      1|none  |      |acc   |↑  |0.6411|±  |0.0038|
| - humanities     |      1|none  |      |acc   |↑  |0.5896|±  |0.0068|
| - other          |      1|none  |      |acc   |↑  |0.7100|±  |0.0078|
| - social sciences|      1|none  |      |acc   |↑  |0.7312|±  |0.0078|
| - stem           |      1|none  |      |acc   |↑  |0.5623|±  |0.0085|
```