import gradio as gr
from transformers import pipeline
import torch
title = "Extractive QA Biomedicine"
description = """
Taking into account the existence of masked language models trained on Spanish Biomedical corpus, the objective of this project is to use them to generate extractice QA models for Biomedicine and compare their effectiveness with general masked language models.
The models were trained on the SQUAD_ES Dataset (automatic translation of the Stanford Question Answering Dataset into Spanish). SQUAD v2 version was chosen in order to include questions that cannot be answered based on a provided context.
The models were evaluated on https://huggingface.co/datasets/hackathon-pln-es/biomed_squad_es_v2 , a subset of the SQUAD_ES dev dataset containing questions related to the Biomedical domain.
"""
article = """
Results
| Model |
Base Model Domain |
exact |
f1 |
HasAns_exact |
HasAns_f1 |
NoAns_exact |
NoAns_f1 |
|---|
| hackathon-pln-es/roberta-base-bne-squad2-es |
General |
67.6341 |
75.6988 |
53.7367 |
70.0526 |
81.2174 |
81.2174 |
| hackathon-pln-es/roberta-base-biomedical-clinical-es-squad2-es |
Biomedical |
66.8426 |
75.2346 |
53.0249 |
70.0031 |
80.3478 |
80.3478 |
| hackathon-pln-es/roberta-base-biomedical-es-squad2-es |
Biomedical |
67.6341 |
74.5612 |
47.6868 |
61.7012 |
87.1304 |
87.1304 |
| hackathon-pln-es/biomedtra-small-es-squad2-es |
Biomedical |
29.6394 |
36.317 |
32.2064 |
45.716 |
27.1304 |
27.1304 |
Conclusion and Future Work
If F1 score is considered, the results show that there may be no advantage in using domain-specific masked language models to generate Biomedical QA models. In any case, close results are observed for the biomedical roberta-based models in comparison with the general roberta-based model.
However, if only unanswerable questions are taken into account, the model with the best F1 metric is hackathon-pln-es/roberta-base-biomedical-es-squad2-es.
As future work, the following experiments could be carried out:
- Use Biomedical masked-language models that were not generated from scratch from a Biomedical corpus but have been adapted from a general model, so as not to lose words and features of Spanish that are also present in Biomedical questions and articles.
- Create a Biomedical training dataset with SQUAD v2 format.
- Generate a new and bigger validation dataset based on questions and contexts generated directly in Spanish and not translated as in SQUAD_Es v2.
- Ensamble different models.
Team
Santiago Maximo
"""
device = 0 if torch.cuda.is_available() else -1
MODEL_NAMES = ["hackathon-pln-es/roberta-base-bne-squad2-es",
"hackathon-pln-es/roberta-base-biomedical-clinical-es-squad2-es",
"hackathon-pln-es/roberta-base-biomedical-es-squad2-es",
"hackathon-pln-es/biomedtra-small-es-squad2-es"]
examples = [
[MODEL_NAMES[3], "¿Qué entidades están incluidas en el sistema de salud federal?", "En el sistema federal de salud (incluyendo el VA, el Servicio de Salud de la India y el NIH), los farmacéuticos de atención ambulatoria reciben plena autoridad prescriptiva independiente. En algunos estados como Carolina del Norte y Nuevo México estos médicos farmacéuticos reciben una autoridad prescriptiva y diagnóstica colaborativa. En 2011 la Junta de Especialidades Farmacéuticas aprobó la práctica de farmacia ambulatoria como una certificación de junta separada. La designación oficial para los farmacéuticos que pasan el examen de certificación de la farmacia ambulatoria será la Junta Certificada de Ambulatorio Farmacéutica y estos farmacéuticos llevarán las iniciales BCACP"],
[MODEL_NAMES[3], "¿Qué cidippido se utiliza como descripción de los ctenóforos en la mayoría de los libros de texto?","Para un filo con relativamente pocas especies, los ctenóforos tienen una amplia gama de planes corporales. Las especies costeras necesitan ser lo suficientemente duras para soportar las olas y remolcar partículas de sedimentos, mientras que algunas especies oceánicas son tan frágiles que es muy difícil capturarlas intactas para su estudio. Además, las especies oceánicas no conservan bien, y son conocidas principalmente por fotografías y notas de observadores. Por lo tanto, la mayor atención se ha concentrado recientemente en tres géneros costeros: Pleurobrachia, Beroe y Mnemiopsis. Al menos dos libros de texto basan sus descripciones de ctenóforos en los cidipépidos Pleurobrachia."]
]
def getanswer(model_name, question, context):
question_answerer = pipeline("question-answering", model=model_name, device=device)
response = question_answerer({
'question': question,
'context': context
})
text_hilight_output = [
(context[:response['start']], None),
(context[response['start']:response['end']], 'Answer'),
(context[response['end']:], None)
]
return response['answer'], response['score']
face = gr.Interface(
fn=getanswer,
inputs=[
gr.inputs.Radio(
label="Pick a QA Model",
choices=MODEL_NAMES,
),
gr.inputs.Textbox(lines=1, placeholder="Question Here… "),
gr.inputs.Textbox(lines=10, placeholder="Context Here… ")
],
outputs=[
gr.outputs.Textbox(label="Answer"),
gr.outputs.Label(num_top_classes=1, label='Score'),
],
layout="vertical",
title=title,
examples=examples,
description=description,
article=article,
allow_flagging ="never"
)
face.launch()