README.md

metadata

license: apache-2.0
datasets:
  - wikitext
  - ptb_text_only
language:
  - en
metrics:
  - perplexity
pipeline_tag: text-generation
model-index:
  - name: distilgpt2
    results:
      - task:
          type: text-generation
        dataset:
          name: penn_treebank
          type: ptb_text_only
        metrics:
          - name: perlexity@distilgpt2:BASELINE
            type: dmx-perlexity
            value: 63.45857238769531
          - name: perlexity@distilgpt2:BASIC
            type: dmx-perlexity
            value: 64.36720275878906
      - task:
          type: text-generation
        dataset:
          name: wikitext2
          type: wikitext-2-raw-v1
        metrics:
          - name: perlexity@distilgpt2:BASELINE
            type: dmx-perlexity
            value: 46.05925369262695
          - name: perlexity@distilgpt2:BASIC
            type: dmx-perlexity
            value: 46.570838928222656

This is a d-Matrix functional reference of the GPT2 model family, with the following revisions:

• distilgpt2
• gpt2
• gpt2-medium
• gpt2-large
• gpt2-xl

The reference provides the following functional configurations:

Configuration	Explanation
BASELINE	a reference functionally equivalent to the original model
BASIC	all linear algebraic operands quantized to BFP16-64, and all other operations transformed to approximated kernel simulations

Usage

Install d-Matrix ML Tools first.

pip install dmx-mltools

The following is an example model and its evaluation.

from mltools.dmx import pipeline

pipe = pipeline(
    task="text-generation",
    model="d-matrix/gpt2",
    revision="gpt2-xl",  # see above for other variants
    dmx_config="BASELINE",  # see above for other variants
    trust_remote_code=True,
    # device_map="auto",  # enabling model parallel on multi-GPU nodes
)

results = pipe.evaluate(metric="d-matrix/perplexity", dataset="wikitext-2")

Evaluation results

• perplexity on penn_treebank

  Revision \ Configuration	BASELINE	BASIC
  distilgpt2	-	- 'loss': 4.150386810302734, 'perplexity': 63.45854187011719
  gpt2	-	-
  gpt2-medium	-	-
  gpt2-large	-	-
  gpt2-xl	-	-

• perplexity on wikitext2

  Revision \ Configuration	BASELINE	BASIC
  distilgpt2	-	-
  gpt2	-	-
  gpt2-medium	-	-
  gpt2-large	-	-
  gpt2-xl	-	-