ValueError: `rope_scaling` must be a dictionary with with two fields

#131
by layor - opened

When deploying my fine tuned model on a dedicated inference endpoint on hugging face this gets triggered:

ValueError: rope_scaling must be a dictionary with with two fields, type and factor, got {'factor': 8.0, 'low_freq_factor': 1.0, 'high_freq_factor': 4.0, 'original_max_position_embeddings': 8192, 'rope_type': 'llama3'}
Application startup failed. Exiting.

I can't access the config.json file since its on the base model so no way to modify these value or to upgrade transformers (since its deploying on huggingface servers).

Any idea what to do?

Same issue!

Same here

Same here. Tried llama 3.1 and it does not work.

same here

pip install transformers -U

It works for me.

@yoo how can you run that on a dedicated inference endpoint when the endpoint can not be deployed?

Does anyone found a solution? I am trying to deploy using text generation inference.

Sign up or log in to comment