Qwen

/

Qwen2-72B-Instruct-AWQ

like 16

Compared to Qwen/Qwen2-72B-Instruct-GPTQ-Int4 this repo seems to be missing the tokenizer.json file.

Also using TGI with this model works but only outputs gibberish like: 'mm theF. to toF1mer.0 to to t...'

Reproduce like this:
docker run --gpus all -it --rm --shm-size 1g -p 80:80 ghcr.io/huggingface/text-generation-inference:2.0.4 --model-id Qwen/Qwen2-72B-Instruct-AWQ --quantize awq --num-shard 4