comparing to Qwen1.5, the tokenizer is changed. Why?

by J22 - opened Apr 18

J22

Apr 18

Qwen2Tokenizer vs PreTrainedTokenizerFast.

Qwen org Apr 19

We've made some optimizations to the vocabulary for coding purposes, so there might be some discrepancies.

huybery changed discussion status to closed Apr 19

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment