|
--- |
|
license: mit |
|
sdk: gradio |
|
emoji: 😻 |
|
colorTo: green |
|
pinned: true |
|
--- |
|
<div align="center"> |
|
<img src="./.asset/grounding_dino_logo.png" width="30%"> |
|
</div> |
|
|
|
# :sauropod: Grounding DINO |
|
|
|
[](https://paperswithcode.com/sota/zero-shot-object-detection-on-mscoco?p=grounding-dino-marrying-dino-with-grounded) [](https://paperswithcode.com/sota/zero-shot-object-detection-on-odinw?p=grounding-dino-marrying-dino-with-grounded) \ |
|
[](https://paperswithcode.com/sota/object-detection-on-coco-minival?p=grounding-dino-marrying-dino-with-grounded) [](https://paperswithcode.com/sota/object-detection-on-coco?p=grounding-dino-marrying-dino-with-grounded) |
|
|
|
|
|
**[IDEA-CVR, IDEA-Research](https://github.com/IDEA-Research)** |
|
|
|
[Shilong Liu](http://www.lsl.zone/), [Zhaoyang Zeng](https://scholar.google.com/citations?user=U_cvvUwAAAAJ&hl=zh-CN&oi=ao), [Tianhe Ren](https://rentainhe.github.io/), [Feng Li](https://scholar.google.com/citations?user=ybRe9GcAAAAJ&hl=zh-CN), [Hao Zhang](https://scholar.google.com/citations?user=B8hPxMQAAAAJ&hl=zh-CN), [Jie Yang](https://github.com/yangjie-cv), [Chunyuan Li](https://scholar.google.com/citations?user=Zd7WmXUAAAAJ&hl=zh-CN&oi=ao), [Jianwei Yang](https://jwyang.github.io/), [Hang Su](https://scholar.google.com/citations?hl=en&user=dxN1_X0AAAAJ&view_op=list_works&sortby=pubdate), [Jun Zhu](https://scholar.google.com/citations?hl=en&user=axsP38wAAAAJ), [Lei Zhang](https://www.leizhang.org/)<sup>:email:</sup>. |
|
|
|
|
|
[[`Paper`](https://arxiv.org/abs/2303.05499)] [[`Demo`](https://huggingface.co/spaces/ShilongLiu/Grounding_DINO_demo)] [[`BibTex`](#black_nib-citation)] |
|
|
|
|
|
PyTorch implementation and pretrained models for Grounding DINO. For details, see the paper **[Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://arxiv.org/abs/2303.05499)**. |
|
|
|
## :sun_with_face: Helpful Tutorial |
|
|
|
- :grapes: [[Read our arXiv Paper](https://arxiv.org/abs/2303.05499)] |
|
- :apple: [[Watch our simple introduction video on YouTube](https://youtu.be/wxWDt5UiwY8)] |
|
- :blossom: [[Try the Colab Demo](https://colab.research.google.com/github/roboflow-ai/notebooks/blob/main/notebooks/zero-shot-object-detection-with-grounding-dino.ipynb)] |
|
- :sunflower: [[Try our Official Huggingface Demo](https://huggingface.co/spaces/ShilongLiu/Grounding_DINO_demo)] |
|
- :maple_leaf: [[Watch the Step by Step Tutorial about GroundingDINO by Roboflow AI](https://youtu.be/cMa77r3YrDk)] |
|
- :mushroom: [[GroundingDINO: Automated Dataset Annotation and Evaluation by Roboflow AI](https://youtu.be/C4NqaRBz_Kw)] |
|
- :hibiscus: [[Accelerate Image Annotation with SAM and GroundingDINO by Roboflow AI](https://youtu.be/oEQYStnF2l8)] |
|
- :white_flower: [[Autodistill: Train YOLOv8 with ZERO Annotations based on Grounding-DINO and Grounded-SAM by Roboflow AI](https://github.com/autodistill/autodistill)] |
|
|
|
<!-- Grounding DINO Methods | |
|
[](https://arxiv.org/abs/2303.05499) |
|
[](https://youtu.be/wxWDt5UiwY8) --> |
|
|
|
<!-- Grounding DINO Demos | |
|
[](https://colab.research.google.com/github/roboflow-ai/notebooks/blob/main/notebooks/zero-shot-object-detection-with-grounding-dino.ipynb) --> |
|
<!-- [](https://youtu.be/cMa77r3YrDk) |
|
[](https://huggingface.co/spaces/ShilongLiu/Grounding_DINO_demo) |
|
[](https://youtu.be/oEQYStnF2l8) |
|
[](https://youtu.be/C4NqaRBz_Kw) --> |
|
|
|
## :sparkles: Highlight Projects |
|
|
|
- [Semantic-SAM: a universal image segmentation model to enable segment and recognize anything at any desired granularity.](https://github.com/UX-Decoder/Semantic-SAM), |
|
- [DetGPT: Detect What You Need via Reasoning](https://github.com/OptimalScale/DetGPT) |
|
- [Grounded-SAM: Marrying Grounding DINO with Segment Anything](https://github.com/IDEA-Research/Grounded-Segment-Anything) |
|
- [Grounding DINO with Stable Diffusion](demo/image_editing_with_groundingdino_stablediffusion.ipynb) |
|
- [Grounding DINO with GLIGEN for Controllable Image Editing](demo/image_editing_with_groundingdino_gligen.ipynb) |
|
- [OpenSeeD: A Simple and Strong Openset Segmentation Model](https://github.com/IDEA-Research/OpenSeeD) |
|
- [SEEM: Segment Everything Everywhere All at Once](https://github.com/UX-Decoder/Segment-Everything-Everywhere-All-At-Once) |
|
- [X-GPT: Conversational Visual Agent supported by X-Decoder](https://github.com/microsoft/X-Decoder/tree/xgpt) |
|
- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://github.com/gligen/GLIGEN) |
|
- [LLaVA: Large Language and Vision Assistant](https://github.com/haotian-liu/LLaVA) |
|
|
|
<!-- Extensions | [Grounding DINO with Segment Anything](https://github.com/IDEA-Research/Grounded-Segment-Anything); [Grounding DINO with Stable Diffusion](demo/image_editing_with_groundingdino_stablediffusion.ipynb); [Grounding DINO with GLIGEN](demo/image_editing_with_groundingdino_gligen.ipynb) --> |
|
|
|
|
|
|
|
<!-- Official PyTorch implementation of [Grounding DINO](https://arxiv.org/abs/2303.05499), a stronger open-set object detector. Code is available now! --> |
|
|
|
|
|
## :bulb: Highlight |
|
|
|
- **Open-Set Detection.** Detect **everything** with language! |
|
- **High Performancce.** COCO zero-shot **52.5 AP** (training without COCO data!). COCO fine-tune **63.0 AP**. |
|
- **Flexible.** Collaboration with Stable Diffusion for Image Editting. |
|
|
|
|
|
|
|
|
|
## :fire: News |
|
- **`2023/07/18`**: We release [Semantic-SAM](https://github.com/UX-Decoder/Semantic-SAM), a universal image segmentation model to enable segment and recognize anything at any desired granularity. **Code** and **checkpoint** are available! |
|
- **`2023/06/17`**: We provide an example to evaluate Grounding DINO on COCO zero-shot performance. |
|
- **`2023/04/15`**: Refer to [CV in the Wild Readings](https://github.com/Computer-Vision-in-the-Wild/CVinW_Readings) for those who are interested in open-set recognition! |
|
- **`2023/04/08`**: We release [demos](demo/image_editing_with_groundingdino_gligen.ipynb) to combine [Grounding DINO](https://arxiv.org/abs/2303.05499) with [GLIGEN](https://github.com/gligen/GLIGEN) for more controllable image editings. |
|
- **`2023/04/08`**: We release [demos](demo/image_editing_with_groundingdino_stablediffusion.ipynb) to combine [Grounding DINO](https://arxiv.org/abs/2303.05499) with [Stable Diffusion](https://github.com/Stability-AI/StableDiffusion) for image editings. |
|
- **`2023/04/06`**: We build a new demo by marrying GroundingDINO with [Segment-Anything](https://github.com/facebookresearch/segment-anything) named **[Grounded-Segment-Anything](https://github.com/IDEA-Research/Grounded-Segment-Anything)** aims to support segmentation in GroundingDINO. |
|
- **`2023/03/28`**: A YouTube [video](https://youtu.be/cMa77r3YrDk) about Grounding DINO and basic object detection prompt engineering. [[SkalskiP](https://github.com/SkalskiP)] |
|
- **`2023/03/28`**: Add a [demo](https://huggingface.co/spaces/ShilongLiu/Grounding_DINO_demo) on Hugging Face Space! |
|
- **`2023/03/27`**: Support CPU-only mode. Now the model can run on machines without GPUs. |
|
- **`2023/03/25`**: A [demo](https://colab.research.google.com/github/roboflow-ai/notebooks/blob/main/notebooks/zero-shot-object-detection-with-grounding-dino.ipynb) for Grounding DINO is available at Colab. [[SkalskiP](https://github.com/SkalskiP)] |
|
- **`2023/03/22`**: Code is available Now! |
|
|
|
<details open> |
|
<summary><font size="4"> |
|
Description |
|
</font></summary> |
|
<a href="https://arxiv.org/abs/2303.05499">Paper</a> introduction. |
|
<img src=".asset/hero_figure.png" alt="ODinW" width="100%"> |
|
Marrying <a href="https://github.com/IDEA-Research/GroundingDINO">Grounding DINO</a> and <a href="https://github.com/gligen/GLIGEN">GLIGEN</a> |
|
<img src="https://huggingface.co/ShilongLiu/GroundingDINO/resolve/main/GD_GLIGEN.png" alt="gd_gligen" width="100%"> |
|
</details> |
|
|
|
## :star: Explanations/Tips for Grounding DINO Inputs and Outputs |
|
- Grounding DINO accepts an `(image, text)` pair as inputs. |
|
- It outputs `900` (by default) object boxes. Each box has similarity scores across all input words. (as shown in Figures below.) |
|
- We defaultly choose the boxes whose highest similarities are higher than a `box_threshold`. |
|
- We extract the words whose similarities are higher than the `text_threshold` as predicted labels. |
|
- If you want to obtain objects of specific phrases, like the `dogs` in the sentence `two dogs with a stick.`, you can select the boxes with highest text similarities with `dogs` as final outputs. |
|
- Note that each word can be split to **more than one** tokens with different tokenlizers. The number of words in a sentence may not equal to the number of text tokens. |
|
- We suggest separating different category names with `.` for Grounding DINO. |
|
 |
|
 |
|
|
|
|
|
## :medal_military: Results |
|
|
|
<details open> |
|
<summary><font size="4"> |
|
COCO Object Detection Results |
|
</font></summary> |
|
<img src=".asset/COCO.png" alt="COCO" width="100%"> |
|
</details> |
|
|
|
<details open> |
|
<summary><font size="4"> |
|
ODinW Object Detection Results |
|
</font></summary> |
|
<img src=".asset/ODinW.png" alt="ODinW" width="100%"> |
|
</details> |
|
|
|
<details open> |
|
<summary><font size="4"> |
|
Marrying Grounding DINO with <a href="https://github.com/Stability-AI/StableDiffusion">Stable Diffusion</a> for Image Editing |
|
</font></summary> |
|
See our example <a href="https://github.com/IDEA-Research/GroundingDINO/blob/main/demo/image_editing_with_groundingdino_stablediffusion.ipynb">notebook</a> for more details. |
|
<img src=".asset/GD_SD.png" alt="GD_SD" width="100%"> |
|
</details> |
|
|
|
|
|
<details open> |
|
<summary><font size="4"> |
|
Marrying Grounding DINO with <a href="https://github.com/gligen/GLIGEN">GLIGEN</a> for more Detailed Image Editing. |
|
</font></summary> |
|
See our example <a href="https://github.com/IDEA-Research/GroundingDINO/blob/main/demo/image_editing_with_groundingdino_gligen.ipynb">notebook</a> for more details. |
|
<img src=".asset/GD_GLIGEN.png" alt="GD_GLIGEN" width="100%"> |
|
</details> |
|
|
|
## :sauropod: Model: Grounding DINO |
|
|
|
Includes: a text backbone, an image backbone, a feature enhancer, a language-guided query selection, and a cross-modality decoder. |
|
|
|
 |
|
|
|
|
|
## :hearts: Acknowledgement |
|
|
|
Our model is related to [DINO](https://github.com/IDEA-Research/DINO) and [GLIP](https://github.com/microsoft/GLIP). Thanks for their great work! |
|
|
|
We also thank great previous work including DETR, Deformable DETR, SMCA, Conditional DETR, Anchor DETR, Dynamic DETR, DAB-DETR, DN-DETR, etc. More related work are available at [Awesome Detection Transformer](https://github.com/IDEACVR/awesome-detection-transformer). A new toolbox [detrex](https://github.com/IDEA-Research/detrex) is available as well. |
|
|
|
Thanks [Stable Diffusion](https://github.com/Stability-AI/StableDiffusion) and [GLIGEN](https://github.com/gligen/GLIGEN) for their awesome models. |
|
|
|
|
|
## :black_nib: Citation |
|
|
|
If you find our work helpful for your research, please consider citing the following BibTeX entry. |
|
|
|
```bibtex |
|
@article{liu2023grounding, |
|
title={Grounding dino: Marrying dino with grounded pre-training for open-set object detection}, |
|
author={Liu, Shilong and Zeng, Zhaoyang and Ren, Tianhe and Li, Feng and Zhang, Hao and Yang, Jie and Li, Chunyuan and Yang, Jianwei and Su, Hang and Zhu, Jun and others}, |
|
journal={arXiv preprint arXiv:2303.05499}, |
|
year={2023} |
|
} |
|
``` |
