judy93536/distilroberta-base-rb156k-ep40

distilroberta-base-rb156k

This model is a fine-tuned version of distilroberta-base on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.2509

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 7.2115e-05
• train_batch_size: 64
• eval_batch_size: 64
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.12
• num_epochs: 40
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
1.779	1.0	8994	1.6188
1.6848	2.0	17988	1.5515
1.6376	3.0	26982	1.5219
1.6208	4.0	35976	1.5021
1.6025	5.0	44970	1.4949
1.5731	6.0	53964	1.4753
1.5512	7.0	62958	1.4579
1.5305	8.0	71952	1.4457
1.5159	9.0	80946	1.4308
1.4967	10.0	89940	1.4212
1.4761	11.0	98934	1.4081
1.4607	12.0	107928	1.3979
1.4512	13.0	116922	1.3896
1.4394	14.0	125916	1.3816
1.431	15.0	134910	1.3755
1.4226	16.0	143904	1.3639
1.4032	17.0	152898	1.3608
1.3992	18.0	161892	1.3544
1.3909	19.0	170886	1.3446
1.3809	20.0	179880	1.3428
1.3701	21.0	188874	1.3320
1.3574	22.0	197868	1.3276
1.3586	23.0	206862	1.3216
1.3521	24.0	215856	1.3180
1.3371	25.0	224850	1.3134
1.3334	26.0	233844	1.3083
1.3261	27.0	242838	1.3042
1.316	28.0	251832	1.2971
1.3061	29.0	260826	1.2931
1.3057	30.0	269820	1.2903
1.2997	31.0	278814	1.2802
1.288	32.0	287808	1.2784
1.2876	33.0	296802	1.2737
1.2778	34.0	305796	1.2703
1.2695	35.0	314790	1.2636
1.268	36.0	323784	1.2597
1.2606	37.0	332778	1.2583
1.254	38.0	341772	1.2547
1.2516	39.0	350766	1.2491
1.2506	40.0	359760	1.2501

Framework versions

• Transformers 4.35.0
• Pytorch 2.1.0+cu118
• Datasets 2.14.6
• Tokenizers 0.14.1