---
license: apache-2.0
base_model: bert-base-multilingual-cased
model-index:
- name: >-
    bert-base-multilingual-cased-finetuned-openalex-topic-classification-title-abstract
  results: []
pipeline_tag: text-classification
widget:
- text: "Cleavage of Structural Proteins during the Assembly of the Head of Bacteriophage T4"
---


# bert-base-multilingual-cased-finetuned-openalex-topic-classification-title-abstract

This model is a fine-tuned version of [bert-base-multilingual-cased](https://huggingface.co/bert-base-multilingual-cased) on a labeled dataset provided by CWTS:
[CWTS Labeled Data]

This is NOT the full model being used to tag [OpenAlex](https://openalex.org/) works with a topic. For that, check out the following github repo:
[OpenAlex Topic Classification](https://github.com/ourresearch/openalex-topic-classification)

That repository will also contain information about text preprocessing, modeling, testing, and deployment.

## Model description

The model was trained using the following input data format (so it is recommended the data be in this format as well):

"\<TITLE\> {insert-processed-title-here}\n\<ABSTRACT\> {insert-processed-abstract-here}"

The quickest way to use this model in Python is with the following code (assuming you have the transformers library installed):

```
from transformers import pipeline

title = "{insert-processed-title-here}"
abstract = "{insert-processed-abstract-here}"

classifier = \
    pipeline(model="OpenAlex/bert-base-multilingual-cased-finetuned-openalex-topic-classification-title-abstract", top_k=10)

classifier(f"""<TITLE> {title}\n<ABSTRACT> {abstract}""")

```

## Intended uses & limitations

The model is intended to be used as part of a larger model that also incorporates journal information and citation features. However, this model is good if you want to use it for quickly generating a topic based only on a title/abstract.

Since this model was fine-tuned on a BERT model, all of the biases seen in that model will most likely show up in this model as well.

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- optimizer: {'name': 'Adam', 'weight_decay': None, 'clipnorm': None, 'global_clipnorm': None, 'clipvalue': None, 'use_ema': False, 'ema_momentum': 0.99, 'ema_overwrite_frequency': None, 'jit_compile': True, 'is_legacy_optimizer': False, 'learning_rate': {'module': 'transformers.optimization_tf', 'class_name': 'WarmUp', 'config': {'initial_learning_rate': 6e-05, 'decay_schedule_fn': {'module': 'keras.optimizers.schedules', 'class_name': 'PolynomialDecay', 'config': {'initial_learning_rate': 6e-05, 'decay_steps': 335420, 'end_learning_rate': 0.0, 'power': 1.0, 'cycle': False, 'name': None}, 'registered_name': None}, 'warmup_steps': 500, 'power': 1.0, 'name': None}, 'registered_name': 'WarmUp'}, 'beta_1': 0.9, 'beta_2': 0.999, 'epsilon': 1e-08, 'amsgrad': False}
- training_precision: float32

### Training results

| Train Loss | Validation Loss | Train Accuracy | Epoch |
|:----------:|:---------------:|:--------------:|:-----:|
| 4.8075     | 3.6686          | 0.3839         | 0     |
| 3.4867     | 3.3360          | 0.4337         | 1     |
| 3.1865     | 3.2005          | 0.4556         | 2     |
| 2.9969     | 3.1379          | 0.4675         | 3     |
| 2.8489     | 3.0900          | 0.4746         | 4     |
| 2.7212     | 3.0744          | 0.4799         | 5     |
| 2.6035     | 3.0660          | 0.4831         | 6     |
| 2.4942     | 3.0737          | 0.4846         | 7     |


### Framework versions

- Transformers 4.35.2
- TensorFlow 2.13.0
- Datasets 2.15.0
- Tokenizers 0.15.0