File size: 2,936 Bytes
6552579 da21945 17a41a8 da21945 c13efb5 5bd2e88 d7a5829 c13efb5 d7a5829 c13efb5 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 |
---
datasets:
- mhan/Shot2Story-20K
- mhan/shot2story
language:
- en
metrics:
- bleu
pipeline_tag: visual-question-answering
---
# Shot2Story: A New Benchmark for Comprehensive Understanding of Multi-shot Videos
- **Repository:** [Shot2Story](https://github.com/bytedance/Shot2Story)
- **Paper:** [2312.10300](https://arxiv.org/abs/2312.10300)
- **Point of Contact:** mailto:[Mingfei Han](hmf282@gmail.com)
## Training Dataset
**Please download the multi-shot videos [here](https://1drv.ms/f/s!Ap3OKt6-X52NgXoG4-64N9WZDenS?e=oIHfkZ).**
We are excited to release a new video-text benchmark for multi-shot video understanding. This release contains a 134k version of our dataset. It includes detailed long summaries (human annotated + GPTV generated) for 134k videos and shot captions (human annotated) for 188k video shots. Please check the dataset [here](https://huggingface.co/datasets/mhan/Shot2Story-134K).
## Models
We are releasing the checkpoints trained with our [Shot2Story-20K](https://huggingface.co/datasets/mhan/Shot2Story-20K) and [Shot2Story-134K](https://huggingface.co/datasets/mhan/Shot2Story-134K).
- **{20k,134k}-version/sum_shot_best_epoch.pth:** Model tuned on our multi-shot summary data. Used in the config files `ckpt`.
- **{20k,134k}-version/shot_av_best_epoch.pth:** Model trained on our single-shot caption data. Used in the config files `ckpt`.
- **transnetv2-pytorch-weights.pth:** Checkpoint used for automatic shot detection method, which is used in the Bot demo. Please following the original license of the TransNetv2.
- **BLIP.cache.tar:** Cached checkpoints for training, testing and offline demos. This is only to ease the usage case that servers can't access huggingface. Please be restriected the original license to the different models.
## License <a name="license"></a>
Our text annotations are licensed under a [Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) License](https://creativecommons.org/licenses/by-nc-sa/4.0/). They are available strictly for non-commercial research.
Users must refer to [HD-VILA-100M](https://github.com/microsoft/XPretrain/blob/main/hd-vila-100m/README.md) for original video access. By downloading our annotations, you agree to these terms. Respect for video copyright holders is paramount. Ensure your use of the videos aligns with the original source's terms.
---
## Citation <a name="citation"></a>
If you find our work useful for your research, please consider citing the paper
```
@misc{han2023shot2story20k,
title={Shot2Story20K: A New Benchmark for Comprehensive Understanding of Multi-shot Videos},
author={Mingfei Han and Linjie Yang and Xiaojun Chang and Heng Wang},
year={2023},
eprint={2312.10300},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
``` |