fix: update tokenizer config to support `add_generation_prompt=True` and clarify content
#3
by
lamhieu
- opened
Here I have updated it so that for each role it will return the correct tokens id of that role more clearly, trim the spaces and support add_generation_prompt config.
I think Phi's chat template auto adds assistant at the end - but anyways I'll need to revamp the chat template
@danielhanchen yeah, I created similar PRs for the base model, please consider it!