Exception: data did not match any variant of untagged enum PyPreTokenizerTypeWrapper at line 1411 column 3

#1
by haihua - opened

Traceback (most recent call last):
File "D:\ReactionT5v2\step1.py", line 55, in
tokenizer = AutoTokenizer.from_pretrained('sagawa/ReactionT5v2-yield')
File "C:\Users\caiha\anaconda3\envs\pt\lib\site-packages\transformers\models\auto\tokenization_auto.py", line 691, in from_pretrained
return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
File "C:\Users\caiha\anaconda3\envs\pt\lib\site-packages\transformers\tokenization_utils_base.py", line 1825, in from_pretrained
return cls._from_pretrained(
File "C:\Users\caiha\anaconda3\envs\pt\lib\site-packages\transformers\tokenization_utils_base.py", line 1988, in _from_pretrained
tokenizer = cls(*init_inputs, **init_kwargs)
File "C:\Users\caiha\anaconda3\envs\pt\lib\site-packages\transformers\models\t5\tokenization_t5_fast.py", line 133, in init
super().init(
File "C:\Users\caiha\anaconda3\envs\pt\lib\site-packages\transformers\tokenization_utils_fast.py", line 111, in init
fast_tokenizer = TokenizerFast.from_file(fast_tokenizer_file)
Exception: data did not match any variant of untagged enum PyPreTokenizerTypeWrapper at line 1411 column 3

Owner

Update tokenizers's version to 0.19.1 and try again.

sagawa changed discussion status to closed

Sign up or log in to comment