Upload tokenizer.json

#17
by ArthurZ HF staff - opened
No description provided.

Per IANA registry, iw was deprecated as the code for Hebrew in 1989 and the preferred code is he

Original PR was merged into whisper here - https://github.com/openai/whisper/pull/401
HuggingFace transformers PR here - https://github.com/huggingface/transformers/pull/21310

The correct subtag:

%%
Type: language
Subtag: he
Description: Hebrew
Added: 2005-10-16
Suppress-Script: Hebr
%%
And the deprecation

%%
Type: language
Subtag: iw
Description: Hebrew
Added: 2005-10-16
Deprecated: 1989-01-01
Preferred-Value: he
Suppress-Script: Hebr
%%

ArthurZ changed pull request status to merged

Sign up or log in to comment