File size: 1,162 Bytes
9d6e32f 26b70be 1021173 26b70be 4ed2f43 26b70be |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
---
license: mit
---
**Note: please check [DeepKPG](https://github.com/uclanlp/DeepKPG#scibart) for using this model in huggingface, including setting up the newly trained tokenizer.**
Paper: [Pre-trained Language Models for Keyphrase Generation: A Thorough Empirical Study](https://arxiv.org/abs/2212.10233)
```
@article{https://doi.org/10.48550/arxiv.2212.10233,
doi = {10.48550/ARXIV.2212.10233},
url = {https://arxiv.org/abs/2212.10233},
author = {Wu, Di and Ahmad, Wasi Uddin and Chang, Kai-Wei},
keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
title = {Pre-trained Language Models for Keyphrase Generation: A Thorough Empirical Study},
publisher = {arXiv},
year = {2022},
copyright = {Creative Commons Attribution 4.0 International}
}
```
Pre-training Corpus: [S2ORC (titles and abstracts)](https://github.com/allenai/s2orc)
Pre-training Details:
- Pre-trained **from scratch** with a science vocabulary
- Batch size: 2048
- Total steps: 250k
- Learning rate: 3e-4
- LR schedule: polynomial with 10k warmup steps
- Masking ratio: 30%, Poisson lambda = 3.5 |