File size: 1,390 Bytes
dbdac9b |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
---
license: mit
---
# Model
The Pandemic PACT Advanced Categorisation Engine (PPACE) is a fine-tuned 8B LLM designed for automatically classifying research abstracts from funded biomedical projects according to WHO-aligned research priorities. Developed as part of the GLOPID-R Pandemic PACT project, PPACE assists in tracking and analysing research funding and clinical evidence for a wide range of diseases with outbreak potential.
The model leverages a human-annotated dataset expanded with rationales generated by a larger LLM. These rationales provide explanations for the chosen labels, enhancing the model's interpretability and accuracy.
# Usage
Todo
# Model Details
PPACE is fine-tuned using Low-Rank Adaptation (LoRA) to ensure efficient training while maintaining high performance. The fine-tuning process involves training the model for 4 epochs on a dataset of 5142 projects, using 8 A100 GPUs with a batch size of 1 per GPU and 4 gradient accumulation steps.
## Hyperparameters
| Hyperparameter | Value |
|---------------------------|--------|
| Total Batch Size | 2 |
| Gradient Accumulation Steps | 4 |
| Learning Rate | 2e-4 |
| LR Scheduler | Linear |
| Epochs | 2 |
| LoRA Rank | 128 |
| LoRA α | 256 |
| LoRA Dropout | 0.05 |
|