---
license: apache-2.0
datasets:
- joshuachou/SkinCAP
- HemanthKumarK/SKINgpt
language:
- en
pipeline_tag: image-to-text
tags:
- biology
- skin
- skin disease
- cancer
- medical
---
# Model Card for PaliGemma Dermatology Model

## Model Details

### Model Description

This model, based on the PaliGemma-3B architecture, has been fine-tuned for dermatology-related image and text processing tasks. The model is designed to assist in the identification of various skin conditions using a combination of image analysis and natural language processing.


- **Developed by:** Bruce_Wayne
- **Model type:** vision model
- **Finetuned from model:** https://huggingface.co/google/paligemma-3b-pt-224
- **LoRa Adaptors used:** Yes
- **Intended use:** Medical image analysis, specifically for dermatology
**
## Uses
### Direct Use

The model can be directly used for analyzing dermatology images, providing insights into potential skin conditions.


## Bias, Risks, and Limitations

**Skin Tone Bias:** The model may have been trained on a dataset that does not adequately represent all skin tones, potentially leading to biased results.
**Geographic Bias:** The model's performance may vary depending on the prevalence of certain conditions in different geographic regions.

## How to Get Started with the Model

```python

from transformers import AutoProcessor, PaliGemmaForConditionalGeneration

model_id = "brucewayne0459/paligemma_derm"
processor = AutoProcessor.from_pretrained(model_id)
model = PaliGemmaForConditionalGeneration.from_pretrained(model_id)
```
## Training Details

### Training Data

The model was fine-tuned on a dataset of dermatological images combined with disease names

### Training Procedure

The model was fine-tuned using LoRA (Low-Rank Adaptation) for more efficient training. Mixed precision (bfloat16) was used to speed up training and reduce memory usage.

#### Training Hyperparameters

- **Training regime:** Mixed precision (bfloat16)
- **Epochs:** 10
- **Learning rate:** 2e-5
- **Batch size:** 6
- **Gradient accumulation steps:** 4


## Evaluation

### Testing Data, Factors & Metrics

#### Testing Data

The model was evaluated on a separate validation set of dermatological images and Disease Names, distinct from the training data.

#### Metrics
- **Validation Loss:** The loss was tracked throughout the training process to evaluate model performance.
- **Accuracy:** The primary metric for assessing model predictions.
### Results

The model achieved a final validation loss of approximately 0.2214, indicating reasonable performance in predicting skin conditions based on the dataset used.

#### Summary


## Environmental Impact


- **Hardware Type:** 1 x L4 GPU
- **Hours used:** ~22 HOURS
- **Cloud Provider:** LIGHTNING AI
- **Compute Region:** USA
- **Carbon Emitted:** 0.9 kg eq. CO2

## Technical Specifications

### Model Architecture and Objective

- **Architecture:** Vision-Language model based on PaliGemma-3B
- **Objective:** To classify and diagnose dermatological conditions from images and text

### Compute Infrastructure

#### Hardware

- **GPU:** 1xL4 GPU
## Model Card Authors 
Bruce_Wayne