Edit model card

Built with Axolotl

qlora-out

This model is a fine-tuned version of tavtav/Rose-20B on the Kimiko dataset.

Model description

The prompt formats used is ShareGPT/Vicuna format.

Intended uses & limitations

Per many people requests, this LoRA is intended to fix spelling from Rose 20B.

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0002
  • train_batch_size: 2
  • eval_batch_size: 2
  • seed: 42
  • gradient_accumulation_steps: 4
  • total_train_batch_size: 8
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_steps: 10
  • num_epochs: 2

Training results

Framework versions

  • Transformers 4.36.0.dev0
  • Pytorch 2.0.1+cu118
  • Datasets 2.15.0
  • Tokenizers 0.15.0

Training procedure

The following bitsandbytes quantization config was used during training:

  • quant_method: bitsandbytes
  • load_in_8bit: False
  • load_in_4bit: True
  • llm_int8_threshold: 6.0
  • llm_int8_skip_modules: None
  • llm_int8_enable_fp32_cpu_offload: False
  • llm_int8_has_fp16_weight: False
  • bnb_4bit_quant_type: nf4
  • bnb_4bit_use_double_quant: True
  • bnb_4bit_compute_dtype: bfloat16

Framework versions

  • PEFT 0.6.0
Downloads last month
6
Inference API
Unable to determine this model’s pipeline type. Check the docs .

Model tree for Chat-Error/Rose-Kimiko-20B

Base model

tavtav/Rose-20B
Adapter
(1)
this model
Adapters
4 models