---
library_name: transformers
license: mit
base_model: gpt2
tags:
- generated_from_trainer
model-index:
- name: sinhala_gpt2
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# sinhala_gpt2

This model is a fine-tuned version of [gpt2](https://huggingface.co/gpt2) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 3.4181

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-06
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 64
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 10
- num_epochs: 1

### Training results

| Training Loss | Epoch  | Step | Validation Loss |
|:-------------:|:------:|:----:|:---------------:|
| 12.5768       | 0.0737 | 20   | 11.7031         |
| 10.6016       | 0.1475 | 40   | 10.1428         |
| 9.5592        | 0.2212 | 60   | 8.4000          |
| 7.7086        | 0.2949 | 80   | 6.1398          |
| 6.1288        | 0.3687 | 100  | 5.1259          |
| 5.2551        | 0.4424 | 120  | 4.4283          |
| 4.7127        | 0.5161 | 140  | 4.0241          |
| 4.3572        | 0.5899 | 160  | 3.7673          |
| 4.1243        | 0.6636 | 180  | 3.6012          |
| 3.9714        | 0.7373 | 200  | 3.5126          |
| 3.8867        | 0.8111 | 220  | 3.4489          |
| 3.8334        | 0.8848 | 240  | 3.4256          |
| 3.8204        | 0.9585 | 260  | 3.4181          |


### Framework versions

- Transformers 4.44.2
- Pytorch 2.4.0+cu121
- Datasets 2.21.0
- Tokenizers 0.19.1