chansung's picture
Update README.md
0c85df1
metadata
license: apache-2.0
language:
  - en
pipeline_tag: text2text-generation
tags:
  - alpaca
  - llama
  - chat
  - gpt4

This repository comes with LoRA checkpoint to make LLaMA into a chatbot like language model. The checkpoint is the output of instruction following fine-tuning process with the following settings on 8xA100(40G) DGX system.

  • Training script: borrowed from the official Alpaca-LoRA implementation
  • Training script:
python finetune.py \
    --base_model='decapoda-research/llama-30b-hf' \
    --data_path='alpaca_data_gpt4.json' \
    --num_epochs=10 \
    --cutoff_len=512 \
    --group_by_length \
    --output_dir='./gpt4-alpaca-lora-30b' \
    --lora_target_modules='[q_proj,k_proj,v_proj,o_proj]' \
    --lora_r=16 \
    --batch_size=... \
    --micro_batch_size=...

You can find how the training went from W&B report here.