Hugging Face's logo

d-matrix
/
gpt2

Text Generation
English
Eval Results
Model card Files Community
d-matrix commited on
Commit
1a457fd
•
1 Parent(s): c5f8250

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +16 -16
README.md CHANGED
@@ -72,7 +72,7 @@ pipe = pipeline(
72
  # device_map="auto", # enabling model parallel on multi-GPU nodes
73
  )
74
 
75
- results = pipe.evaluate(metric="d-matrix/perplexity", dataset="wikitext", dataset_version="wikitext-2-raw-v1")
76
  ```
77
 
78
  ### Evaluation results
@@ -80,26 +80,26 @@ results = pipe.evaluate(metric="d-matrix/perplexity", dataset="wikitext", datase
80
  - `perplexity` on `penn_treebank`
81
  Revision \ Configuration | **`BASELINE`** | **`BASIC`**
82
  :-- | --: | --:
83
- `distilgpt2` | 63.45854187011719 | 64.12968444824219
84
- `gpt2` | 35.771339416503906 | 35.92854309082031
85
- `gpt2-medium` | 27.062211990356445 | 27.10440444946289
86
- `gpt2-large` | 23.03119659423828 | 23.043560028076172
87
- `gpt2-xl` | 21.0135440826416 | 21.02527618408203
88
 
89
  - `perplexity` on `wikitext2`
90
  Revision \ Configuration | **`BASELINE`** | **`BASIC`**
91
  :-- | --: | --:
92
- `distilgpt2` | 46.05925369262695 | 46.44373321533203
93
- `gpt2` | 29.941965103149414 | 30.076961517333984
94
- `gpt2-medium` | 21.709003448486328 | 21.73116683959961
95
- `gpt2-large` | 19.425939559936523 | 19.428546905517578
96
- `gpt2-xl` | 17.39567756652832 | 17.396879196166992
97
 
98
  - `perplexity` on `wikitext103`
99
  Revision \ Configuration | **`BASELINE`** | **`BASIC`**
100
  :-- | --: | --:
101
- `distilgpt2` | 46.05925369262695 | 46.44373321533203
102
- `gpt2` | 29.941965103149414 |30.076961517333984
103
- `gpt2-medium` | 21.709003448486328 | 21.73116683959961
104
- `gpt2-large` | 19.425939559936523 | 19.428546905517578
105
- `gpt2-xl` | 17.39567756652832 | 17.396879196166992
 
72
  # device_map="auto", # enabling model parallel on multi-GPU nodes
73
  )
74
 
75
+ results = pipe.eval(metric="d-matrix/dmx_perplexity", dataset="wikitext", dataset_version="wikitext-2-raw-v1")
76
  ```
77
 
78
  ### Evaluation results
 
80
  - `perplexity` on `penn_treebank`
81
  Revision \ Configuration | **`BASELINE`** | **`BASIC`**
82
  :-- | --: | --:
83
+ `distilgpt2` | 63.46 | 64.13
84
+ `gpt2` | 35.77 | 35.93
85
+ `gpt2-medium` | 27.06 | 27.10
86
+ `gpt2-large` | 23.03 | 23.04
87
+ `gpt2-xl` | 21.01 | 21.02
88
 
89
  - `perplexity` on `wikitext2`
90
  Revision \ Configuration | **`BASELINE`** | **`BASIC`**
91
  :-- | --: | --:
92
+ `distilgpt2` | 46.06 | 46.44
93
+ `gpt2` | 29.94 | 30.08
94
+ `gpt2-medium` | 21.71 | 21.73
95
+ `gpt2-large` | 19.42| 19.43
96
+ `gpt2-xl` | 17.40| 17.40
97
 
98
  - `perplexity` on `wikitext103`
99
  Revision \ Configuration | **`BASELINE`** | **`BASIC`**
100
  :-- | --: | --:
101
+ `distilgpt2` | 46.06 | 46.44
102
+ `gpt2` | 29.94 |30.08
103
+ `gpt2-medium` | 21.71 | 21.73
104
+ `gpt2-large` | 19.43 | 19.43
105
+ `gpt2-xl` | 17.40 | 17.40