QTIP Quantized Models
Collection
See https://github.com/Cornell-RelaxML/qtip
•
24 items
•
Updated
YAML Metadata
Warning:
empty or missing yaml metadata in repo card
(https://huggingface.co/docs/hub/model-cards#model-card-metadata)
Perplexity (ctx 8192)
BF16:
I1005 10:12:54.337877 967332 eval_ppl.py:66] wikitext2 perplexity: 6.49992036819458
I1005 10:17:19.718732 967332 eval_ppl.py:66] c4 perplexity: 8.022844314575195
3 Bit QTIP (this model):
I1005 10:13:30.440096 967334 eval_ppl.py:66] wikitext2 perplexity: 6.804006099700928
I1005 10:19:49.873933 967334 eval_ppl.py:66] c4 perplexity: 8.314777374267578
Zeroshot Results with lm_eval 0.3.0
BF16:
|-------------|------:|--------|-----:|---|-----:|
|arc_challenge| 0|acc |0.5196|± |0.0146|
| | |acc_norm|0.5512|± |0.0145|
|arc_easy | 0|acc |0.8182|± |0.0079|
| | |acc_norm|0.7980|± |0.0082|
|boolq | 1|acc |0.8410|± |0.0064|
|piqa | 0|acc |0.8003|± |0.0093|
| | |acc_norm|0.8090|± |0.0092|
|winogrande | 0|acc |0.7380|± |0.0124|
3 Bit QTIP (this model):
|-------------|------:|--------|-----:|---|-----:|
|arc_challenge| 0|acc |0.5077|± |0.0146|
| | |acc_norm|0.5469|± |0.0145|
|arc_easy | 0|acc |0.8093|± |0.0081|
| | |acc_norm|0.7917|± |0.0083|
|boolq | 1|acc |0.8419|± |0.0064|
|piqa | 0|acc |0.7922|± |0.0095|
| | |acc_norm|0.8025|± |0.0093|
|winogrande | 0|acc |0.7198|± |0.0126|
- Downloads last month
- 21