Issues loading model with ooabooga textgenwebui

#20
by Kenji776 - opened

I'm not able to get the model to load while using ooabooga textgenwebui. The Transformers returns the error

ValueError: rope_scaling must be a dictionary with two fields, type and factor, got {'factor': 8.0, 'low_freq_factor': 1.0, 'high_freq_factor': 4.0, 'original_max_position_embeddings': 8192, 'rope_type': 'llama3'}

If I use llama.cpp as the loader I get

IndexError: list index out of range

I did open up the conda environment by using cmd_windows.bat file in the textgenwebui folder and then ran

pip install -r requirements.txt --upgrade

to try and update everything, and explicitly ran

pip install --upgrade transformers
pip install --upgrade trl

I've also enabled "trust-remote-code" as I've seen that suggested but it didn't seem to make a differences. Are there any other settings or values I need to adjust or other dependencies I need to update?

image.png

image.png

image.png

same issue

working now after updating.

Kenji776 changed discussion title from Issues loading model with ooabooga and textgenwebui to Issues loading model with ooabooga textgenwebui

I found a 'fix' but I'm not sure what the side effects might be. In the config.json file of the model find the entry that says "rope_scaling" and replace it with this

"rope_scaling": {
"factor": 8.0,
"type": "dynamic"
},

I honestly do not know what these values mean, I just fed in values the loader said it wanted and it seems to work.

I found a 'fix' but I'm not sure what the side effects might be. In the config.json file of the model find the entry that says "rope_scaling" and replace it with this

"rope_scaling": {
"factor": 8.0,
"type": "dynamic"
},

I honestly do not know what these values mean, I just fed in values the loader said it wanted and it seems to work.

claude made the same suggestion it loads then crashes for me

Hi there! If you install the latest transformers version this should be fixed

Sign up or log in to comment