alunapr's picture
Update README.md
22e1a91 verified
metadata
license: mit
library_name: peft
tags:
  - generated_from_trainer
base_model: roberta-large
metrics:
  - accuracy
model-index:
  - name: lora-roberta-large-finetuned-reduced_captures
    results: []
language:
  - en

lora-roberta-large-finetuned-reduced_captures

This model is a fine-tuned version of roberta-large on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 0.2458
  • Accuracy: 0.9345

Tests metrics:

  • loss: 0.23798689246177673
  • accuracy: 0.9321285694578563

Model description

Captures prediction based on LoRA-adapted RoBERTa-large. This includes labels 0-12 but excluding 9 (criminality).

Intended uses & limitations

Impero Safeguarding & Wellbeing

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0003
  • train_batch_size: 10
  • eval_batch_size: 10
  • seed: 42
  • distributed_type: multi-GPU
  • gradient_accumulation_steps: 4
  • total_train_batch_size: 40
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • num_epochs: 10

Training results

Training Loss Epoch Step Validation Loss Accuracy
0.3924 0.9996 616 0.3862 0.8906
0.4385 1.9992 1232 0.3267 0.9048
0.388 2.9988 1848 0.2849 0.9175
0.3256 4.0 2465 0.2728 0.9207
0.2718 4.9996 3081 0.2939 0.9170
0.2877 5.9992 3697 0.2522 0.9267
0.233 6.9988 4313 0.2624 0.9260
0.1832 8.0 4930 0.2512 0.9317
0.2399 8.9996 5546 0.2458 0.9345
0.1506 9.9959 6160 0.2390 0.9336

Framework versions

  • PEFT 0.12.0
  • Transformers 4.40.2
  • Pytorch 2.3.0+cu121
  • Datasets 2.19.1
  • Tokenizers 0.19.0