Commit
•
e04f911
1
Parent(s):
191f8fb
Update README.md
Browse files
README.md
CHANGED
|
@@ -14,7 +14,6 @@ This repository contains [`mistralai/Mistral-Small-Instruct-2409`](https://huggi
|
|
| 14 |
1. **Memory-efficiency:** The full-precision model is around 44 GB, while this xMADified model is only 12 GB, making it feasible to run on a 16 GB GPU.
|
| 15 |
|
| 16 |
2. **Accuracy:** This xMADified model preserves the quality of the full-precision model. In the table below, we present the zero-shot accuracy on popular benchmarks of this xMADified model against the [GPTQ](https://github.com/AutoGPTQ/AutoGPTQ)-quantized model (both w4g128 for a fair comparison). GPTQ fails on the difficult **MMLU** task, while the xMADai model offers significantly higher accuracy.
|
| 17 |
-
|
| 18 |
| Model | MMLU | Arc Challenge | Arc Easy | LAMBADA | WinoGrande | PIQA |
|
| 19 |
|---|---|---|---|---|---|---|
|
| 20 |
| GPTQ Mistral-Small-Instruct-2409 | 49.45 | 56.14 | 80.64 | 75.1 | 77.74 | 77.48 |
|
|
|
|
| 14 |
1. **Memory-efficiency:** The full-precision model is around 44 GB, while this xMADified model is only 12 GB, making it feasible to run on a 16 GB GPU.
|
| 15 |
|
| 16 |
2. **Accuracy:** This xMADified model preserves the quality of the full-precision model. In the table below, we present the zero-shot accuracy on popular benchmarks of this xMADified model against the [GPTQ](https://github.com/AutoGPTQ/AutoGPTQ)-quantized model (both w4g128 for a fair comparison). GPTQ fails on the difficult **MMLU** task, while the xMADai model offers significantly higher accuracy.
|
|
|
|
| 17 |
| Model | MMLU | Arc Challenge | Arc Easy | LAMBADA | WinoGrande | PIQA |
|
| 18 |
|---|---|---|---|---|---|---|
|
| 19 |
| GPTQ Mistral-Small-Instruct-2409 | 49.45 | 56.14 | 80.64 | 75.1 | 77.74 | 77.48 |
|