File size: 3,668 Bytes
07ed740 49982d2 07ed740 49982d2 c653011 07ed740 97e7fe6 87e115d 97e7fe6 87e115d 97e7fe6 07ed740 87e115d 07ed740 7f5acb6 97e7fe6 334df3f 97e7fe6 87e115d 07ed740 97e7fe6 07ed740 983f956 07ed740 49982d2 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 |
---
license: apache-2.0
base_model: bert-base-multilingual-cased
model-index:
- name: >-
bert-base-multilingual-cased-finetuned-openalex-topic-classification-title-abstract
results: []
pipeline_tag: text-classification
widget:
- text: "Cleavage of Structural Proteins during the Assembly of the Head of Bacteriophage T4"
---
# bert-base-multilingual-cased-finetuned-openalex-topic-classification-title-abstract
This model is a fine-tuned version of [bert-base-multilingual-cased](https://huggingface.co/bert-base-multilingual-cased) on a labeled dataset provided by CWTS:
[CWTS Labeled Data]
This is NOT the full model being used to tag [OpenAlex](https://openalex.org/) works with a topic. For that, check out the following github repo:
[OpenAlex Topic Classification](https://github.com/ourresearch/openalex-topic-classification)
That repository will also contain information about text preprocessing, modeling, testing, and deployment.
## Model description
The model was trained using the following input data format (so it is recommended the data be in this format as well):
"\<TITLE\> {insert-processed-title-here}\n\<ABSTRACT\> {insert-processed-abstract-here}"
The quickest way to use this model in Python is with the following code (assuming you have the transformers library installed):
```
from transformers import pipeline
title = "{insert-processed-title-here}"
abstract = "{insert-processed-abstract-here}"
classifier = \
pipeline(model="OpenAlex/bert-base-multilingual-cased-finetuned-openalex-topic-classification-title-abstract", top_k=10)
classifier(f"""<TITLE> {title}\n<ABSTRACT> {abstract}""")
```
## Intended uses & limitations
The model is intended to be used as part of a larger model that also incorporates journal information and citation features. However, this model is good if you want to use it for quickly generating a topic based only on a title/abstract.
Since this model was fine-tuned on a BERT model, all of the biases seen in that model will most likely show up in this model as well.
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- optimizer: {'name': 'Adam', 'weight_decay': None, 'clipnorm': None, 'global_clipnorm': None, 'clipvalue': None, 'use_ema': False, 'ema_momentum': 0.99, 'ema_overwrite_frequency': None, 'jit_compile': True, 'is_legacy_optimizer': False, 'learning_rate': {'module': 'transformers.optimization_tf', 'class_name': 'WarmUp', 'config': {'initial_learning_rate': 6e-05, 'decay_schedule_fn': {'module': 'keras.optimizers.schedules', 'class_name': 'PolynomialDecay', 'config': {'initial_learning_rate': 6e-05, 'decay_steps': 335420, 'end_learning_rate': 0.0, 'power': 1.0, 'cycle': False, 'name': None}, 'registered_name': None}, 'warmup_steps': 500, 'power': 1.0, 'name': None}, 'registered_name': 'WarmUp'}, 'beta_1': 0.9, 'beta_2': 0.999, 'epsilon': 1e-08, 'amsgrad': False}
- training_precision: float32
### Training results
| Train Loss | Validation Loss | Train Accuracy | Epoch |
|:----------:|:---------------:|:--------------:|:-----:|
| 4.8075 | 3.6686 | 0.3839 | 0 |
| 3.4867 | 3.3360 | 0.4337 | 1 |
| 3.1865 | 3.2005 | 0.4556 | 2 |
| 2.9969 | 3.1379 | 0.4675 | 3 |
| 2.8489 | 3.0900 | 0.4746 | 4 |
| 2.7212 | 3.0744 | 0.4799 | 5 |
| 2.6035 | 3.0660 | 0.4831 | 6 |
| 2.4942 | 3.0737 | 0.4846 | 7 |
### Framework versions
- Transformers 4.35.2
- TensorFlow 2.13.0
- Datasets 2.15.0
- Tokenizers 0.15.0 |