Commands to make this: | |
```bash | |
cd llama.cpp | |
./convert.py --outfile Karasu-Mixtral-8x22B-v0.1-q3_k_m --outtype f16 /workspace/llm_training/axolotl/mixtral_8x22B_training/merged_model_multiling | |
./quantize /workspace/Karasu-Mixtral-8x22B-v0.1.gguf /workspace/Karasu-Mixtral-8x22B-v0.1_q3_k_m.gguf Q3_K_M | |
./gguf-split --split --split-max-size 5G /workspace/Karasu-Mixtral-8x22B-v0.1_q3_k_m.gguf /workspace/somewhere-sensible | |
``` |