Fix eos_token and model_max_length in tokenizer_config
#20
by
AshtonIsNotHere
- opened
The eos_token should be converted from <|end_of_text|> to <|eot_id|> to correspond with the chat template and the original model's tokenizer_config. Model max length should also be changed to its correct value: 131072.