Error
Downloading tokenizer_config.json: 100%|██████████████████████████████████████████████████████████████████| 283k/283k [00:00<00:00, 1.76MB/s]
Traceback (most recent call last):
File "", line 1, in
File "", line 14, in
File "C:\Program Files\Python37\lib\site-packages\transformers\models\auto\processing_auto.py", line 269, in from_pretrained
pretrained_model_name_or_path, trust_remote_code=trust_remote_code, **kwargs
File "C:\Program Files\Python37\lib\site-packages\transformers\processing_utils.py", line 184, in from_pretrained
args = cls._get_arguments_from_pretrained(pretrained_model_name_or_path, **kwargs)
File "C:\Program Files\Python37\lib\site-packages\transformers\processing_utils.py", line 228, in _get_arguments_from_pretrained
args.append(attribute_class.from_pretrained(pretrained_model_name_or_path, **kwargs))
File "C:\Program Files\Python37\lib\site-packages\transformers\tokenization_utils_base.py", line 1835, in from_pretrained
**kwargs,
File "C:\Program Files\Python37\lib\site-packages\transformers\tokenization_utils_base.py", line 2045, in _from_pretrained
f"Non-consecutive added token '{token}' found. "
ValueError: Non-consecutive added token '<|0.02|>' found. Should have index 50365 but has index 50366 in saved vocabulary.