scibart-base / README.md
xiaowu0162's picture
Update README.md
1021173
metadata
license: mit

Note: please check DeepKPG for using this model in huggingface, including setting up the newly trained tokenizer.

Paper: Pre-trained Language Models for Keyphrase Generation: A Thorough Empirical Study

@article{https://doi.org/10.48550/arxiv.2212.10233,
  doi = {10.48550/ARXIV.2212.10233},
  url = {https://arxiv.org/abs/2212.10233},
  author = {Wu, Di and Ahmad, Wasi Uddin and Chang, Kai-Wei},
  keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
  title = {Pre-trained Language Models for Keyphrase Generation: A Thorough Empirical Study},
  publisher = {arXiv},
  year = {2022}, 
  copyright = {Creative Commons Attribution 4.0 International}
}

Pre-training Corpus: S2ORC (titles and abstracts)

Pre-training Details:

  • Pre-trained from scratch with a science vocabulary
  • Batch size: 2048
  • Total steps: 250k
  • Learning rate: 3e-4
  • LR schedule: polynomial with 10k warmup steps
  • Masking ratio: 30%, Poisson lambda = 3.5