---
tags:
- gptq
- 4bit
- int4
- gptqmodel
- modelcloud
- llama-3.1
- 70b
- instruct
license: llama3.1
---
This model has been quantized using [GPTQModel](https://github.com/ModelCloud/GPTQModel).

- **bits**: 4
- **group_size**: 128
- **desc_act**: true
- **static_groups**: false
- **sym**: true
- **lm_head**: false
- **damp_percent**: 0.0025
- **true_sequential**: true
- **model_name_or_path**: ""
- **model_file_base_name**: "model"
- **quant_method**: "gptq"
- **checkpoint_format**: "gptq"
- **meta**：
  - **quantizer**: "gptqmodel:0.9.9-dev0"

## Example:
```python
from transformers import AutoTokenizer
from gptqmodel import GPTQModel

model_name = "ModelCloud/Meta-Llama-3.1-70B-Instruct-gptq-4bit"

prompt = [{"role": "user", "content": "I am in Shanghai, preparing to visit the natural history museum. Can you tell me the best way to"}]

tokenizer = AutoTokenizer.from_pretrained(model_name)

model = GPTQModel.from_quantized(model_name)

input_tensor = tokenizer.apply_chat_template(prompt, add_generation_prompt=True, return_tensors="pt")
outputs = model.generate(input_ids=input_tensor.to(model.device), max_new_tokens=100)
result = tokenizer.decode(outputs[0][input_tensor.shape[1]:], skip_special_tokens=True)

print(result)
```

## lm-eval benchmark:

|                 Tasks                 |Version|Filter|n-shot|  Metric  |   |Value |   |Stderr|                                                                                                    
|---------------------------------------|------:|------|-----:|----------|---|-----:|---|-----:|                                                                                                    
|arc_challenge                          |      1|none  |     0|acc       |↑  |0.6186|±  |0.0142|                                                                                                    
|                                       |       |none  |     0|acc_norm  |↑  |0.6297|±  |0.0141|                                                                                                    
|arc_easy                               |      1|none  |     0|acc       |↑  |0.8628|±  |0.0071|                                                                                                    
|                                       |       |none  |     0|acc_norm  |↑  |0.8338|±  |0.0076|                                                                                                    
|boolq                                  |      2|none  |     0|acc       |↑  |0.8761|±  |0.0058|                                                                                                    
|hellaswag                              |      1|none  |     0|acc       |↑  |0.6463|±  |0.0048|                                                                                                    
|                                       |       |none  |     0|acc_norm  |↑  |0.8389|±  |0.0037|                                                                                                    
|lambada_openai                         |      1|none  |     0|acc       |↑  |0.7561|±  |0.0060|                                                                                                    
|                                       |       |none  |     0|perplexity|↓  |3.0311|±  |0.0639|                                                                                                    
|mmlu                                   |      1|none  |      |acc       |↑  |0.8100|±  |0.0032|                                                                                                    
| - humanities                          |      1|none  |      |acc       |↑  |0.7981|±  |0.0057|                                                                                                    
|  - formal_logic                       |      0|none  |     0|acc       |↑  |0.6349|±  |0.0431|                                                                                                    
|  - high_school_european_history       |      0|none  |     0|acc       |↑  |0.8545|±  |0.0275|                                                                                                    
|  - high_school_us_history             |      0|none  |     0|acc       |↑  |0.9412|±  |0.0165|                                                                                                    
|  - high_school_world_history          |      0|none  |     0|acc       |↑  |0.9198|±  |0.0177|                                                                                                    
|  - international_law                  |      0|none  |     0|acc       |↑  |0.9008|±  |0.0273|                                                                                                    
|  - jurisprudence                      |      0|none  |     0|acc       |↑  |0.8796|±  |0.0315|                                                                                                    
|  - logical_fallacies                  |      0|none  |     0|acc       |↑  |0.8650|±  |0.0268|                                                                                                    
|  - moral_disputes                     |      0|none  |     0|acc       |↑  |0.8266|±  |0.0204|                                                                                                    
|  - moral_scenarios                    |      0|none  |     0|acc       |↑  |0.8559|±  |0.0117|                                                                                                    
|  - philosophy                         |      0|none  |     0|acc       |↑  |0.8360|±  |0.0210|                                                                                                    
|  - prehistory                         |      0|none  |     0|acc       |↑  |0.8827|±  |0.0179|                                                                                                    
|  - professional_law                   |      0|none  |     0|acc       |↑  |0.6675|±  |0.0120|
|  - world_religions                    |      0|none  |     0|acc       |↑  |0.9181|±  |0.0210|
| - other                               |      1|none  |      |acc       |↑  |0.8304|±  |0.0064|
|  - business_ethics                    |      0|none  |     0|acc       |↑  |0.7900|±  |0.0409|
|  - clinical_knowledge                 |      0|none  |     0|acc       |↑  |0.8566|±  |0.0216|
|  - college_medicine                   |      0|none  |     0|acc       |↑  |0.7630|±  |0.0324|
|  - global_facts                       |      0|none  |     0|acc       |↑  |0.5800|±  |0.0496|
|  - human_aging                        |      0|none  |     0|acc       |↑  |0.8206|±  |0.0257|  
|  - management                         |      0|none  |     0|acc       |↑  |0.8835|±  |0.0318|  
|  - marketing                          |      0|none  |     0|acc       |↑  |0.9231|±  |0.0175|  
|  - medical_genetics                   |      0|none  |     0|acc       |↑  |0.9400|±  |0.0239|  
|  - miscellaneous                      |      0|none  |     0|acc       |↑  |0.9144|±  |0.0100|  
|  - nutrition                          |      0|none  |     0|acc       |↑  |0.8660|±  |0.0195|  
|  - professional_accounting            |      0|none  |     0|acc       |↑  |0.6454|±  |0.0285|  
|  - professional_medicine              |      0|none  |     0|acc       |↑  |0.8971|±  |0.0185|  
|  - virology                           |      0|none  |     0|acc       |↑  |0.5602|±  |0.0386|  
| - social sciences                     |      1|none  |      |acc       |↑  |0.8736|±  |0.0059|  
|  - econometrics                       |      0|none  |     0|acc       |↑  |0.7018|±  |0.0430|  
|  - high_school_geography              |      0|none  |     0|acc       |↑  |0.9242|±  |0.0189|  
|  - high_school_government_and_politics|      0|none  |     0|acc       |↑  |0.9741|±  |0.0115|  
|  - high_school_macroeconomics         |      0|none  |     0|acc       |↑  |0.8410|±  |0.0185|  
|  - high_school_microeconomics         |      0|none  |     0|acc       |↑  |0.8992|±  |0.0196|                                                                        
|  - high_school_psychology             |      0|none  |     0|acc       |↑  |0.9229|±  |0.0114|                                                                        
|  - human_sexuality                    |      0|none  |     0|acc       |↑  |0.8779|±  |0.0287|                                                                        
|  - professional_psychology            |      0|none  |     0|acc       |↑  |0.8497|±  |0.0145|                                                                                                  
|  - public_relations                   |      0|none  |     0|acc       |↑  |0.7273|±  |0.0427|                                                                                    
|  - security_studies                   |      0|none  |     0|acc       |↑  |0.8163|±  |0.0248|                                                                                    
|  - sociology                          |      0|none  |     0|acc       |↑  |0.9154|±  |0.0197|                                                                                    
|  - us_foreign_policy                  |      0|none  |     0|acc       |↑  |0.9300|±  |0.0256|                  
| - stem                                |      1|none  |      |acc       |↑  |0.7456|±  |0.0075|                  
|  - abstract_algebra                   |      0|none  |     0|acc       |↑  |0.6300|±  |0.0485|                  
|  - anatomy                            |      0|none  |     0|acc       |↑  |0.7926|±  |0.0350|                              
|  - astronomy                          |      0|none  |     0|acc       |↑  |0.8947|±  |0.0250|                              
|  - college_biology                    |      0|none  |     0|acc       |↑  |0.9444|±  |0.0192|                  
|  - college_chemistry                  |      0|none  |     0|acc       |↑  |0.5800|±  |0.0496|                                                                                                  
|  - college_computer_science           |      0|none  |     0|acc       |↑  |0.6700|±  |0.0473|                                                                                                  
|  - college_mathematics                |      0|none  |     0|acc       |↑  |0.5400|±  |0.0501|                                                                                                  
|  - college_physics                    |      0|none  |     0|acc       |↑  |0.6275|±  |0.0481|                              
|  - computer_security                  |      0|none  |     0|acc       |↑  |0.8200|±  |0.0386|                  
|  - conceptual_physics                 |      0|none  |     0|acc       |↑  |0.7830|±  |0.0269|                              
|  - electrical_engineering             |      0|none  |     0|acc       |↑  |0.7862|±  |0.0342|                  
|  - elementary_mathematics             |      0|none  |     0|acc       |↑  |0.7593|±  |0.0220|                  
|  - high_school_biology                |      0|none  |     0|acc       |↑  |0.9194|±  |0.0155|                  
|  - high_school_chemistry              |      0|none  |     0|acc       |↑  |0.7143|±  |0.0318|                              
|  - high_school_computer_science       |      0|none  |     0|acc       |↑  |0.9200|±  |0.0273|                              
|  - high_school_mathematics            |      0|none  |     0|acc       |↑  |0.5185|±  |0.0305|                              
|  - high_school_physics                |      0|none  |     0|acc       |↑  |0.6556|±  |0.0388|                                            
|  - high_school_statistics             |      0|none  |     0|acc       |↑  |0.7361|±  |0.0301|                              
|  - machine_learning                   |      0|none  |     0|acc       |↑  |0.7054|±  |0.0433|                              
|openbookqa                             |      1|none  |     0|acc       |↑  |0.3660|±  |0.0216|                              
|                                       |       |none  |     0|acc_norm  |↑  |0.4620|±  |0.0223|                                            
|piqa                                   |      1|none  |     0|acc       |↑  |0.8264|±  |0.0088|                                            
|                                       |       |none  |     0|acc_norm  |↑  |0.8319|±  |0.0087|                                            
|rte                                    |      1|none  |     0|acc       |↑  |0.7184|±  |0.0271|                                            
|truthfulqa_mc1                         |      2|none  |     0|acc       |↑  |0.3917|±  |0.0171|                                            
|winogrande                             |      1|none  |     0|acc       |↑  |0.7924|±  |0.0114|                                            

|      Groups      |Version|Filter|n-shot|Metric|   |Value |   |Stderr|                                                                     
|------------------|------:|------|------|------|---|-----:|---|-----:|                                                                     
|mmlu              |      1|none  |      |acc   |↑  |0.8100|±  |0.0032|                                                                     
| - humanities     |      1|none  |      |acc   |↑  |0.7981|±  |0.0057|                                                                     
| - other          |      1|none  |      |acc   |↑  |0.8304|±  |0.0064|                                                                     
| - social sciences|      1|none  |      |acc   |↑  |0.8736|±  |0.0059|                                                                     
| - stem           |      1|none  |      |acc   |↑  |0.7456|±  |0.0075|