unhashable type 'list' after tokenizer_config.json Update
The issue was not present when using an older version of the tokenizer loaded with transformers.AutoTokenizer.from_pretrained("./models/tokenizer/"). This indicates that the recent changes in tokenizer_config.json might be the problem.
Error :
File "/home/ubuntu/Chat/back_end/prompt/prompt_template.py", line 35, in get_multi_query_prompt_template
multi_query_chat_template = self.tokenizer.apply_chat_template(self.multi_query_prompt_template, tokenize=False, add_generation_prompt=True)
File "/home/ubuntu/Chat/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1740, in apply_chat_template
compiled_template = self._compile_jinja_template(chat_template)
TypeError: unhashable type: 'list'
Expected Behavior:
The tokenizer should handle the list type appropriately without causing a crash, as it did in the previous version.
Actual Behavior:
The script crashes due to a TypeError when attempting to process a list type with the new tokenizer settings.
changes causing a crash : https://huggingface.co/NousResearch/Hermes-2-Pro-Llama-3-8B/commit/234dacf8b6ad6b072a2273ca8d2e2816197e49d4