TUBELEX Statistical Language Models
N-gram models on the TUBELEX YouTube subtitle corpora. We provide modified Kneser-Ney language models of order 5 (Heafield et al., 2013), i.e. KenLM models.
The files are in LZMA-compressed ARPA format.
What is TUBELEX?
TUBELEX is a YouTube subtitle corpus currently available for Chinese, English, Indonesian, Japanese, and Spanish.
- TODO: paper link
- fastText word embeddings
- word frequencies and code
Usage
To download and use the KenLM models in Python, first install dependencies:
pip install huggingface_hub
pip install https://github.com/kpu/kenlm/archive/master.zip
You can then use e.g. the English (en) model in the following way:
import kenlm
from huggingface_hub import hf_hub_download
model_file = hf_hub_download(repo_id='naist-nlp/tubelex-kenlm', filename='tubelex-en.arpa.xz')
# Loading the model requires KenLM to be compiled with LZMA support (`HAVE_XZLIB`).
# Otherwise you fill first need to decompress the model.
model = kenlm.Model(model_file)
text = ''a sequence of words' # pre-tokenized, lower-cased, without punctuation
model.perplexity(text)