Hugging Face's logo

Wenboz
/
phi3-offline-dpo-lora-noise-0.0-5e-7-thre-1.5-42

PEFT
Safetensors
phi3
alignment-handbook
trl
dpo
Generated from Trainer
custom_code
Model card Files Community
phi3-offline-dpo-lora-noise-0.0-5e-7-thre-1.5-42 / README.md
Wenboz's picture
Wenboz
End of training
ecf72a5 verified
preview code
|
raw
history blame contribute delete
3.23 kB
metadata
license: mit
library_name: peft
tags:
  - alignment-handbook
  - trl
  - dpo
  - generated_from_trainer
base_model: microsoft/Phi-3-mini-4k-instruct
model-index:
  - name: phi3-offline-dpo-lora-noise-0.0-5e-7-thre-1.5-42
    results: []

Visualize in Weights & Biases

phi3-offline-dpo-lora-noise-0.0-5e-7-thre-1.5-42

This model is a fine-tuned version of microsoft/Phi-3-mini-4k-instruct on the None dataset. It achieves the following results on the evaluation set:

  • Loss: 0.6130
  • Rewards/chosen: -0.4194
  • Rewards/rejected: -0.5933
  • Rewards/accuracies: 0.7540
  • Rewards/margins: 0.1739
  • Logps/rejected: -459.6432
  • Logps/chosen: -448.1436
  • Logits/rejected: 12.5287
  • Logits/chosen: 13.8414

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 5e-07
  • train_batch_size: 4
  • eval_batch_size: 4
  • seed: 42
  • distributed_type: multi-GPU
  • num_devices: 4
  • gradient_accumulation_steps: 4
  • total_train_batch_size: 64
  • total_eval_batch_size: 16
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_ratio: 0.1
  • num_epochs: 1

Training results

Training Loss Epoch Step Validation Loss Rewards/chosen Rewards/rejected Rewards/accuracies Rewards/margins Logps/rejected Logps/chosen Logits/rejected Logits/chosen
0.6885 0.1778 100 0.6884 -0.0158 -0.0244 0.6190 0.0086 -402.7496 -407.7805 12.8305 14.2621
0.6712 0.3556 200 0.6680 -0.0971 -0.1464 0.7937 0.0493 -414.9504 -415.9148 12.6482 14.0845
0.6339 0.5333 300 0.6389 -0.2593 -0.3712 0.7540 0.1119 -437.4307 -432.1300 12.8556 14.1744
0.6203 0.7111 400 0.6203 -0.3738 -0.5313 0.7540 0.1575 -453.4457 -443.5887 12.6256 13.9444
0.6102 0.8889 500 0.6131 -0.4150 -0.5892 0.7540 0.1743 -459.2376 -447.7001 12.5314 13.8427

Framework versions

  • PEFT 0.7.1
  • Transformers 4.42.3
  • Pytorch 2.3.0+cu121
  • Datasets 2.14.6
  • Tokenizers 0.19.1