Spaces:

eson
/

tokenizer-arena

Running

App Files Files Community

Is there any documentation within this leaderboard?

by zhiminy - opened Jun 29

Discussion

zhiminy

Jun 29

•

edited Jun 29

I cannot locate any specification in this space...What is this leaderboard used for?

eson

Owner Jul 13

Thanks for your attention. A brief document has been added to the demo.
The leaderboard aims to evaluate tokenizer performance on different languages.

Lower oov_ratio refers to less out-of-vocabulary tokens.
Higher char/token means less words be segmented into subwords.

eson changed discussion status to closed Jul 22

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment