Hugging Face
Models
Datasets
Spaces
Posts
Docs
Solutions
Pricing
Log In
Sign Up
Spaces:
yhavinga
/
dutch-tokenizer-arena
like
6
Running
App
Files
Files
Community
1
main
dutch-tokenizer-arena
2 contributors
History:
79 commits
yhavinga
Add Llama tokenizer creation for Dutch, English, Code, Markdown and TeX.
c78da21
2 months ago
css
add compression leaderboard
2 months ago
images
update
10 months ago
js
add grok mixtral
3 months ago
patcher
add compression leaderboard
2 months ago
stats
Add Llama tokenizer creation for Dutch, English, Code, Markdown and TeX.
2 months ago
utils
Add Llama tokenizer creation for Dutch, English, Code, Markdown and TeX.
2 months ago
vocab
Add Llama tokenizer creation for Dutch, English, Code, Markdown and TeX.
2 months ago
.gitattributes
2.32 kB
add grok mixtral
3 months ago
.gitignore
171 Bytes
update
10 months ago
README.md
25.5 kB
Hack some Dutch tokenizers into it
6 months ago
app.py
565 Bytes
Add Llama tokenizer creation for Dutch, English, Code, Markdown and TeX.
2 months ago
app_compression.py
4.75 kB
Add Llama tokenizer creation for Dutch, English, Code, Markdown and TeX.
2 months ago
app_playground.py
9.81 kB
update
2 months ago
config.py
1.72 kB
Add Llama tokenizer creation for Dutch, English, Code, Markdown and TeX.
2 months ago
examples.py
3.16 kB
add compression leaderboard
2 months ago
requirements.txt
96 Bytes
Pin protobuf dep
4 months ago
util.py
8.15 kB
Hack some Dutch tokenizers into it
6 months ago