Arabic-ALBERT Base
Arabic edition of ALBERT Base pretrained language model
If you use any of these models in your work, please cite this work as:
@software{ali_safaya_2020_4718724,
author = {Ali Safaya},
title = {Arabic-ALBERT},
month = aug,
year = 2020,
publisher = {Zenodo},
version = {1.0.0},
doi = {10.5281/zenodo.4718724},
url = {https://doi.org/10.5281/zenodo.4718724}
}
Pretraining data
The models were pretrained on ~4.4 Billion words:
- Arabic version of OSCAR (unshuffled version of the corpus) - filtered from Common Crawl
- Recent dump of Arabic Wikipedia
Notes on training data:
- Our final version of corpus contains some non-Arabic words inlines, which we did not remove from sentences since that would affect some tasks like NER.
- Although non-Arabic characters were lowered as a preprocessing step, since Arabic characters do not have upper or lower case, there is no cased and uncased version of the model.
- The corpus and vocabulary set are not restricted to Modern Standard Arabic, they contain some dialectical Arabic too.
Pretraining details
- These models were trained using Google ALBERT's github repository on a single TPU v3-8 provided for free from TFRC.
- Our pretraining procedure follows training settings of bert with some changes: trained for 7M training steps with batchsize of 64, instead of 125K with batchsize of 4096.
Models
albert-base | albert-large | albert-xlarge | |
---|---|---|---|
Hidden Layers | 12 | 24 | 24 |
Attention heads | 12 | 16 | 32 |
Hidden size | 768 | 1024 | 2048 |
Results
For further details on the models performance or any other queries, please refer to Arabic-ALBERT
How to use
You can use these models by installing torch
or tensorflow
and Huggingface library transformers
. And you can use it directly by initializing it like this:
from transformers import AutoTokenizer, AutoModel
# loading the tokenizer
base_tokenizer = AutoTokenizer.from_pretrained("kuisailab/albert-base-arabic")
# loading the model
base_model = AutoModelForMaskedLM.from_pretrained("kuisailab/albert-base-arabic")
Acknowledgement
Thanks to Google for providing free TPU for the training process and for Huggingface for hosting these models on their servers 😊
- Downloads last month
- 159
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.