File size: 3,651 Bytes
0476028 5fc4e3f 0476028 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 |
---
license: mit
base_model:
- unsloth/gemma-2-9b-bnb-4bit
pipeline_tag: text2text-generation
---
# Introduction
**Reverse Dictionary**
This dictionary is not a dictionary that tells you the meaning when you enter a word, but a dictionary that tells you the words corresponding to the meaning when you enter sentence.
I used [μ°λ¦¬λ§μ](https://github.com/songys/Dictionaries) dataset, which consists of a lot of information such as word, word meanings, word types, synonymsm, and example sentences.
Then, only the words and their meaning were separated to fit the model input structure.
# The process of model training
Because I worked in a colab environments, I used [Unsloth](https://github.com/unslothai/unsloth?tab=readme-ov-file), a finetuning optimization tool that is useful for small GPU resources.
I used **gemma-2-9b-bnb-4bit** model among the models supported by Unsloth. This model was 4bit quantized and trained by modifying the parameters. But during the learning process, evaluation couldn't be performed due to 'out of memory', and the entire dataset was trained.
You can find detailed code on github below.
# Result
An example inference is as follows:
![image/png](https://cdn-uploads.huggingface.co/production/uploads/66e500f67d43e55cfdf656af/DyxEwO41R66TKC3gn26zB.png)
First of all, we tested simple 10 words:
```
λΉνκΈ° - λλ ₯μΌλ‘ νλ‘ν λ¬λ₯Ό λ리거λ μ°μ κ°μ€λ₯Ό λ΄λΏλ νμ μνμ¬ μκΈ°λ μλ ₯(ζε)μ μ΄μ©νμ¬ κ³΅μ€μΌλ‘ λ μ λ μλ€λλ ν곡기
κ°λ°© - 물건μ λ£μ΄ λ€κ±°λ λ©κ³ λ€λ μ μκ² λ§λ μ©κ΅¬
κ³ μμ΄ - κ³ μμκ³Όμ νλ. μλ μν리카μ 리λΉμμ΄μΎ‘μ΄λ₯Ό κΈΈλ€μΈ κ²μΌλ‘, ν±κ³Ό μ‘κ³³λκ° νΉν λ°λ¬ν΄μ μ‘μμ μ£Όλ‘ νλ€. λ°ν±μ μμ λ‘κ² κ°μΆκ±°λ λλ¬λΌ μ μμΌλ©°, λμ μ΄λμ΄ κ³³μμλ μ λ³Ό μ μλ€. μ μλλ¬Όλ‘λ μ‘μ’
νμ¬ μ¬λ¬ νμ’
μ΄ μλ€.
μν - μΌμ ν μλ―Έλ₯Ό κ°κ³ μμ§μ΄λ λμμ 촬μνμ¬ μμ¬κΈ°λ‘ μμ¬λ§μ μ¬ννλ μ’
ν© μμ .
μλμ°¨ - μλκΈ°λ₯Ό μ₯μΉνμ¬ κ·Έ λλ ₯μΌλ‘ λ°ν΄λ₯Ό κ΅΄λ €μ μ² κΈΈμ΄λ κ°μ€λ μ μ μνμ§ μλνκ³ λ
μλ₯Ό μμ§μ΄λλ‘ λ§λ μ°¨. μΉμ©μ°¨, μΉν©μλμ°¨, νλ¬Ό μλμ°¨, νΉμ μλμ°¨ λ° μ΄λ₯μλμ°¨κ° μλ€.
λ°λλ - νμ΄κ³Όμ μλ‘ μ¬λ¬ν΄μ΄μ΄ν. λμ΄λ 3~10λ―Έν°μ΄λ©°, λ
μμ μμ€κΈ°μμ μ£½μ λͺ¨μμ μΉμ΄ λμ κΈ΄ νμνμ λ
Ήμ μμ΄ 8~10κ°κ° λμ³λκ³ , κΈ΄ μκΉμ§κ° μλ‘ κ²Ήμ³ νμ€κΈ°λ₯Ό μ΄λ£¨λ©΄μ μλλ€. μ΄μ¬λ¦μ 컀λ€λ κ½μ€κΈ°κ° λμ μ·μ λλ°μμ μκ½μ΄ μ΄μ λͺ¨μμΌλ‘ νΌκ³ , μ΄λ§€λ μμ©νλ€. μ΄λ μ§λ°©μ΄ μμ°μ§λ‘ μ°λ¦¬λλΌμμλ μ¨μ€μμ μ¬λ°°νλ€.
μ»΄ν¨ν° - μ μ νλ‘λ₯Ό μ΄μ©ν κ³ μμ μλ κ³μ°κΈ°. μ«μ κ³μ°, μλ μ μ΄, λ°μ΄ν° μ²λ¦¬, μ¬λ¬΄ κ΄λ¦¬, μΈμ΄λ μμ μ 보 μ²λ¦¬ λ°μμ κ΄λ²μνκ² μ΄μ©λλ€.
μ¬κ³Ό - μ¬κ³Όλ무μ μ΄λ§€.
μ±
- μ’
μ΄λ₯Ό μ¬λ¬ μ₯ λ¬Άμ΄ λ§¨ 물건.
νκ΅ - μΌμ ν λͺ©μ γκ΅κ³Ό κ³Όμ γμ€λΉγμ λ λ° λ²κ·μ μνμ¬ κ³μμ μΌλ‘ νμμκ² κ΅μ‘μ μ€μνλ κΈ°κ΄.
```
The result is that **7 out of 10 words** were guessed correctly, and 2 words were output as similar words.
And 10% of the dataset was used as a testset.
# References
https://github.com/teddylee777/langchain-kr/tree/main/18-FineTuning
---
**If you want to see more,**
- Github : https://github.com/hyunjin-C/gemma-sprint
- Blog : https://velog.io/@hyunjin-c/Gemma-Sprint-Gemma-2-9b-Finetuning
|