--- license: llama3 base_model: Magpie-Align/Llama-3-8B-WildChat tags: - axolotl - generated_from_trainer model-index: - name: Llama-3-8B-WildChat results: [] pipeline_tag: text-generation --- # QuantFactory/Llama-3-8B-WildChat-GGUF This is quantized version of [Magpie-Align/Llama-3-8B-WildChat](https://huggingface.co/Magpie-Align/Llama-3-8B-WildChat) created using llama.cpp # Model Description [Built with Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl)
See axolotl config axolotl version: `0.4.0` ```yaml base_model: meta-llama/Meta-Llama-3-8B model_type: LlamaForCausalLM tokenizer_type: AutoTokenizer load_in_8bit: false load_in_4bit: false strict: false datasets: - path: flydust/WildChat_ShareGPT type: sharegpt conversation: llama3 dataset_prepared_path: last_run_prepared val_set_size: 0.001 output_dir: ./out_Llama-3-WildChat sequence_len: 8192 sample_packing: true eval_sample_packing: false pad_to_sequence_len: true wandb_project: SynDa wandb_entity: wandb_watch: wandb_name: Llama-3-WildChat wandb_log_model: hub_model_id: SynDa/Llama-3-8B-WildChat gradient_accumulation_steps: 8 micro_batch_size: 1 num_epochs: 2 optimizer: paged_adamw_8bit lr_scheduler: cosine learning_rate: 2e-5 train_on_inputs: false group_by_length: false bf16: auto fp16: tf32: false gradient_checkpointing: true gradient_checkpointing_kwargs: use_reentrant: false early_stopping_patience: resume_from_checkpoint: logging_steps: 1 xformers_attention: flash_attention: true warmup_steps: 100 evals_per_epoch: 3 eval_table_size: saves_per_epoch: 1 debug: deepspeed: weight_decay: 0.0 fsdp: fsdp_config: special_tokens: pad_token: <|end_of_text|> ```

# Llama-3-8B-WildChat This model is a fine-tuned version of [meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) on the None dataset. It achieves the following results on the evaluation set: - Loss: 0.8197 ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 2e-05 - train_batch_size: 1 - eval_batch_size: 1 - seed: 42 - distributed_type: multi-GPU - num_devices: 4 - gradient_accumulation_steps: 8 - total_train_batch_size: 32 - total_eval_batch_size: 4 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: cosine - lr_scheduler_warmup_steps: 100 - num_epochs: 2 ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:------:|:----:|:---------------:| | 1.1455 | 0.0003 | 1 | 1.3389 | | 0.9084 | 0.3333 | 1128 | 0.8551 | | 0.9265 | 0.6667 | 2256 | 0.8363 | | 0.9086 | 1.0 | 3384 | 0.8210 | | 0.8257 | 1.3164 | 4512 | 0.8214 | | 0.8306 | 1.6497 | 5640 | 0.8197 | | 0.8252 | 1.9831 | 6768 | 0.8197 | ### Framework versions - Transformers 4.40.2 - Pytorch 2.3.0+cu121 - Datasets 2.19.1 - Tokenizers 0.19.1