metadata
datasets:
- JeanKaddour/minipile
language:
- en
license: mpl-2.0
RWKV Minipile
Model Specifications
- Architecture: RWKV
- Vocabulary Size: 65,536
- Embedding Size: 768
- Number of Layers: 12
- Context Length: 512
- Data Type: bfloat16
- Dataset: Minipile
- Tokens: 20,643,840 (20 Million)
The model underwent a rigorous training regimen, completing 30 epochs to optimize performance.
Inference
pip install torch numpy
python inference.py