File size: 2,668 Bytes
93cadd1 43ca93f 93cadd1 43ca93f 93cadd1 43ca93f 93cadd1 43ca93f 93cadd1 566dd40 3a612e6 5869676 25abefe a3da97a 93cadd1 a3da97a 1b748b1 d3a4ff5 1f48029 93cadd1 5869676 9c4574d 93cadd1 1145254 51efdf2 3fe8bf9 93cadd1 23a2db5 93cadd1 23a2db5 eb23884 23a2db5 93cadd1 23a2db5 6358f4d ed9feeb 79df29a cd0c5ad 058d82f cd0c5ad 79df29a ed9feeb |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 |
---
license: apache-2.0
datasets:
- wikitext
- ptb_text_only
language:
- en
metrics:
- perplexity
pipeline_tag: text-generation
model-index:
- name: distilgpt2
results:
- task:
type: text-generation
dataset:
name: penn_treebank
type: ptb_text_only
metrics:
- name: perlexity@distilgpt2:BASELINE
type: dmx-perlexity
value: 63.45857238769531
- name: perlexity@distilgpt2:BASIC
type: dmx-perlexity
value: 64.36720275878906
- task:
type: text-generation
dataset:
name: wikitext2
type: wikitext-2-raw-v1
metrics:
- name: perlexity@distilgpt2:BASELINE
type: dmx-perlexity
value: 46.05925369262695
- name: perlexity@distilgpt2:BASIC
type: dmx-perlexity
value: 46.570838928222656
---
This is a d-Matrix functional reference of the GPT2 model family, with the following *revisions*:
- [`distilgpt2`](https://huggingface.co/distilbert/distilgpt2)
- [`gpt2`](https://huggingface.co/openai-community/gpt2)
- [`gpt2-medium`](https://huggingface.co/openai-community/gpt2-medium)
- [`gpt2-large`](https://huggingface.co/openai-community/gpt2-large)
- [`gpt2-xl`](https://huggingface.co/openai-community/gpt2-xl)
The reference provides the following functional *configurations*:
Configuration | Explanation
:-- | :--
**`BASELINE`** | a reference functionally equivalent to the original model
**`BASIC`** | all linear algebraic operands quantized to `BFP16-64`, and all other operations transformed to approximated kernel simulations
### Usage
Install d-Matrix [ML Tools](https://github.com/d-matrix-ai/dmx-mltools) first.
```sh
pip install dmx-mltools
```
The following is an example model and its evaluation.
```python
from mltools.dmx import pipeline
pipe = pipeline(
task="text-generation",
model="d-matrix/gpt2",
revision="gpt2-xl", # see above for other variants
dmx_config="BASELINE", # see above for other variants
trust_remote_code=True,
# device_map="auto", # enabling model parallel on multi-GPU nodes
)
results = pipe.evaluate(metric="d-matrix/perplexity", dataset="wikitext-2")
```
### Evaluation results
- `perplexity` on `penn_treebank`
Revision \ Configuration | **`BASELINE`** | **`BASIC`**
:-- | --: | --:
`distilgpt2` | - | - 'loss': 4.150386810302734, 'perplexity': 63.45854187011719
`gpt2` | - | -
`gpt2-medium` | - | -
`gpt2-large` | - | -
`gpt2-xl` | - | -
- `perplexity` on `wikitext2`
Revision \ Configuration | **`BASELINE`** | **`BASIC`**
:-- | --: | --:
`distilgpt2` | - | -
`gpt2` | - | -
`gpt2-medium` | - | -
`gpt2-large` | - | -
`gpt2-xl` | - | -
|