license: apache-2.0
Model Card for PaliGemma Dermatology Model
Model Details
Model Description
This model, based on the PaliGemma-3B architecture, has been fine-tuned for dermatology-related image and text processing tasks. The model is designed to assist in the identification of various skin conditions using a combination of image analysis and natural language processing.
- Developed by: Bruce_Wayne
- Funded by [optional]: Jhonny and koti
- Model type: vision model
- Finetuned from model [optional]: https://huggingface.co/google/paligemma-3b-pt-224
- LoRa Adaptors used: Yes
- Intended use: Medical image analysis, specifically for dermatology **
Uses
Direct Use
The model can be directly used for analyzing dermatology images, providing insights into potential skin conditions.
Bias, Risks, and Limitations
Skin Tone Bias: The model may have been trained on a dataset that does not adequately represent all skin tones, potentially leading to biased results. Geographic Bias: The model's performance may vary depending on the prevalence of certain conditions in different geographic regions.
How to Get Started with the Model
** python
from transformers import AutoProcessor, PaliGemmaForConditionalGeneration
model_id = "brucewayne0459/paligemma_derm" processor = AutoProcessor.from_pretrained(model_id) model = PaliGemmaForConditionalGeneration.from_pretrained(model_id) **
Training Details
Training Data
The model was fine-tuned on a dataset of dermatological images combined with disease names
Training Procedure
The model was fine-tuned using LoRA (Low-Rank Adaptation) for more efficient training. Mixed precision (bfloat16) was used to speed up training and reduce memory usage.
Training Hyperparameters
- Training regime: Mixed precision (bfloat16)
- Epochs: 10
- Learning rate: 2e-5
- Batch size: 6
- Gradient accumulation steps: 4
Evaluation
Testing Data, Factors & Metrics
Testing Data
The model was evaluated on a separate validation set of dermatological images and Disease Names, distinct from the training data.
Metrics
- Validation Loss: The loss was tracked throughout the training process to evaluate model performance.
- Accuracy: The primary metric for assessing model predictions.
Results
The model achieved a final validation loss of approximately 0.2214, indicating reasonable performance in predicting skin conditions based on the dataset used.
Summary
Environmental Impact
- Hardware Type: 1 x L4 GPU
- Hours used: ~22 HOURS
- Cloud Provider: LIGHTNING AI
- Compute Region: USA
- Carbon Emitted: 0.9 kg eq. CO2
Technical Specifications
Model Architecture and Objective
- Architecture: Vision-Language model based on PaliGemma-3B
- Objective: To classify and diagnose dermatological conditions from images and text
Compute Infrastructure
Hardware
- GPU: 1xL4 GPU
Model Card Authors
Bruce_Wayne