A GPT-4V Level MLLM for Single Image, Multi Image and Video on Your Phone

<h1>A GPT-4V Level MLLM for Single Image, Multi Image and Video on Your Phone</h1>

[GitHub](https://github.com/Ucas-HaoranWei/GOT-OCR2.0/tree/main)


## Usage
Inference using Huggingface transformers on NVIDIA GPUs. Requirements tested on python 3.10：
```
torch==2.0.1
torchvision==0.15.2
transformers==4.37.2
megfile==3.1.2
```


```python
from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('ucaslcl/GOT-OCR2_0', trust_remote_code=True)
model = AutoModel.from_pretrained('ucaslcl/GOT-OCR2_0', trust_remote_code=True, low_cpu_mem_usage=True, device_map='cuda', use_safetensors=True, pad_token_id=tokenizer.eos_token_id)
model = model.eval().cuda()


# input your test image
image_file = 'xxx.jpg'

# plain texts OCR
model.chat(tokenizer, image_file, ocr_type='ocr')

# format texts OCR:
model.chat(tokenizer, image_file, ocr_type='format')

# fine-grained OCR:
model.chat(tokenizer, image_file, ocr_type='ocr', ocr_box='')
model.chat(tokenizer, image_file, ocr_type='format', ocr_box='')
model.chat(tokenizer, image_file, ocr_type='ocr', ocr_color='')
model.chat(tokenizer, image_file, ocr_type='format', ocr_color='')

# multi-crop OCR:
res = model.chat_crop(tokenizer, image_file = image_file)

# render the formatted OCR results:
model.chat(tokenizer, image_file, ocr_type='format', ocr_box='', ocr_color='', render=True, save_render_file = '/data/code/a2hf/chat_plus.html')

print(res)


```