Create README.md

README.md

@@ -0,0 +1,75 @@
+
+# text_generation_bangla_model
+BanglaCLM dataset: 
+
+- OSCAR: 12.84GB
+
+- Wikipedia dump: 6.24GB
+
+- ProthomAlo: 3.92GB
+
+- Kalerkantho: 3.24GB
+
+
+## Model description
+
+- context size : 128
+
+
+## Training and evaluation data
+The BanglaCLM data set is divided into a training set (90%)and a validation set (10%).
+
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+
+- Batch size: 32
+
+- Initial learning rate: 5e-5
+
+- Number of warmup steps: 10000
+
+- Weight decay rate: 0.01
+
+- Tokenization algorithm: BPE
+
+- Vocabulary size of tokenizer: 50256
+
+- Total trainable params: 124,439,808
+
+- Epochs: 40
+
+- Number of training steps: 40772228
+
+- training_precision: float32
+
+
+### Training results
+
+perplexity score: 2.86.
+
+
+### Framework versions
+
+- Transformers 4.26.1
+- TensorFlow 2.11.0
+- Datasets 2.10.0
+- Tokenizers 0.13.2
+
+### Citation
+If you find this model helpful, please cite.
+```
+@INPROCEEDINGS{10303383,
+  author={Salim, Md. Shahidul and Murad, Hasan and Das, Dola and Ahmed, Faisal},
+  booktitle={2023 International Conference on Information and Communication Technology for Sustainable Development (ICICT4SD)}, 
+  title={BanglaGPT: A Generative Pretrained Transformer-Based Model for Bangla Language}, 
+  year={2023},
+  volume={},
+  number={},
+  pages={56-59},
+  doi={10.1109/ICICT4SD59951.2023.10303383}}
+
+```