metadata
license: apache-2.0
datasets:
- wikitext
- ptb_text_only
language:
- en
metrics:
- perplexity
pipeline_tag: text-generation
model-index:
- name: distilgpt2
results:
- task:
type: text-generation
dataset:
name: penn_treebank
type: ptb_text_only
metrics:
- name: perlexity@distilgpt2:BASELINE
type: dmx-perlexity
value: 63.45857238769531
- name: perlexity@distilgpt2:BASIC
type: dmx-perlexity
value: 64.36720275878906
- task:
type: text-generation
dataset:
name: wikitext2
type: wikitext-2-raw-v1
metrics:
- name: perlexity@distilgpt2:BASELINE
type: dmx-perlexity
value: 46.05925369262695
- name: perlexity@distilgpt2:BASIC
type: dmx-perlexity
value: 46.570838928222656
This is a d-Matrix functional reference of the GPT2 model family, with the following revisions:
The reference provides the following functional configurations:
Configuration | Explanation |
---|---|
BASELINE |
a reference functionally equivalent to the original model |
BASIC |
all linear algebraic operands quantized to BFP16-64 , and all other operations transformed to approximated kernel simulations |
Usage
Install d-Matrix ML Tools first.
pip install dmx-mltools
The following is an example model and its evaluation.
from mltools.dmx import pipeline
pipe = pipeline(
task="text-generation",
model="d-matrix/gpt2",
revision="gpt2-xl", # see above for other variants
dmx_config="BASELINE", # see above for other variants
trust_remote_code=True,
# device_map="auto", # enabling model parallel on multi-GPU nodes
)
results = pipe.evaluate(metric="d-matrix/perplexity", dataset="wikitext-2")
Evaluation results
perplexity
onpenn_treebank
Revision \ Configuration BASELINE
BASIC
distilgpt2
- - 'loss': 4.150386810302734, 'perplexity': 63.45854187011719 gpt2
- - gpt2-medium
- - gpt2-large
- - gpt2-xl
- - perplexity
onwikitext2
Revision \ Configuration BASELINE
BASIC
distilgpt2
- - gpt2
- - gpt2-medium
- - gpt2-large
- - gpt2-xl
- -