README.md

---
license: mit
datasets:
- CarrotAI/ko-instruction-dataset
- CarrotAI/Amazing-Instructions
- CarrotAI/kommlu
language:
- ko
base_model:
- Qwen/Qwen2-7B-Instruct
pipeline_tag: text-generation
tags:
- Carrot
- Korea
- mergekit
---

## Quickstart

Here provides a code snippet with `apply_chat_template` to show you how to load the tokenizer and model and how to generate contents.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
device = "cuda" # the device to load the model onto

model = AutoModelForCausalLM.from_pretrained(
 "CarrotAI/Rabbit-Ko-15B-Instruct",
 torch_dtype="auto",
 device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("CarrotAI/Rabbit-Ko-15B-Instruct")

prompt = "Give me a short introduction to large language model."
messages = [
 {"role": "system", "content": "You are a helpful assistant."},
 {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
 messages,
 tokenize=False,
 add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(device)

generated_ids = model.generate(
 model_inputs.input_ids,
 max_new_tokens=512
)
generated_ids = [
 output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
```

### Processing Long Texts
1. **Install vLLM**: You can install vLLM by running the following command.

```bash
pip install "vllm>=0.4.3"
```

Or you can install vLLM from [source](https://github.com/vllm-project/vllm/).

2. **Configure Model Settings**: After downloading the model weights, modify the `config.json` file by including the below snippet:
 ```json
 {
 "architectures": [
 "Qwen2ForCausalLM"
 ],
 // ...
 "vocab_size": 152064,

 // adding the following snippets
 "rope_scaling": {
 "factor": 4.0,
 "original_max_position_embeddings": 32768,
 "type": "yarn"
 }
 }
 ```
 This snippet enable YARN to support longer contexts.

3. **Model Deployment**: Utilize vLLM to deploy your model. For instance, you can set up an openAI-like server using the command:

 ```bash
 python -m vllm.entrypoints.openai.api_server --served-model-name Qwen2-7B-Instruct --model path/to/weights
 ```

 Then you can access the Chat API by:

 ```bash
 curl http://localhost:8000/v1/chat/completions \
 -H "Content-Type: application/json" \
 -d '{
 "model": "CarrotAI/Rabbit-Ko-15B-Instruct",
 "messages": [
 {"role": "system", "content": "You are a helpful assistant."},
 {"role": "user", "content": "Your Long Input Here."}
 ]
 }'
 ```

 For further usage instructions of vLLM, please refer to our [Github](https://github.com/QwenLM/Qwen2).


### Applications
This fine-tuned model is particularly suited for [mention applications, e.g., chatbots, question-answering systems, etc.]. Its enhanced capabilities ensure more accurate and contextually appropriate responses in these domains.

### Limitations and Considerations
While our fine-tuning process has optimized the model for specific tasks, it's important to acknowledge potential limitations. The model's performance can still vary based on the complexity of the task and the specificities of the input data. Users are encouraged to evaluate the model thoroughly in their specific context to ensure it meets their requirements.

If you liked this model, please use the card below

```
@article{RabbitKo15BInstruct,
 title={CarrotAI/Rabbit-Ko-15B-Instruct Card},
 author={CarrotAI (L, GEUN)},
 year={2024},
 url = {https://huggingface.co/CarrotAI/Rabbit-Ko-15B-Instruct}
}
```