INT4 LLMs for vLLM - a neuralmagic Collection

neuralmagic 's Collections

FP8 LLMs for vLLM

Llama-3.1 Quantization

INT8 LLMs for vLLM

INT4 LLMs for vLLM

Sparse Foundational Llama 2 Models

Compression Papers

DeepSparse Sparse LLMs

Sparse Finetuning MPT

Compressed LLMs from the Community

INT4 LLMs for vLLM

updated 7 days ago

Accurate INT4 quantized models by Neural Magic, ready for use with vLLM!

neuralmagic/Meta-Llama-3.1-405B-Instruct-quantized.w4a16

Text Generation • Updated 24 days ago • 6.51k • 7
neuralmagic/Meta-Llama-3.1-70B-Instruct-quantized.w4a16

Text Generation • Updated 24 days ago • 60.9k • 13
neuralmagic/Meta-Llama-3.1-8B-Instruct-quantized.w4a16

Text Generation • Updated 24 days ago • 6.68k • 17
neuralmagic/Mistral-Nemo-Instruct-2407-quantized.w4a16

Text Generation • Updated 21 days ago • 112 • 1
neuralmagic/gemma-2-9b-it-quantized.w4a16

Text Generation • Updated 21 days ago • 83 • 1
neuralmagic/gemma-2-2b-it-quantized.w4a16

Text Generation • Updated 21 days ago • 28
neuralmagic/Phi-3-medium-128k-instruct-quantized.w4a16

Text Generation • Updated 1 day ago • 2.81k • 3
neuralmagic/Phi-3-mini-128k-instruct-quantized.w4a16

Text Generation • Updated 21 days ago • 145 • 1
neuralmagic/Qwen2-0.5B-Instruct-quantized.w4a16

Text Generation • Updated Jul 18 • 347
neuralmagic/Qwen2-1.5B-Instruct-quantized.w4a16

Text Generation • Updated Jul 18 • 37
neuralmagic/Qwen2-72B-Instruct-quantized.w4a16

Text Generation • Updated Jul 18 • 246 • 4
neuralmagic/Qwen2-7B-Instruct-quantized.w4a16

Text Generation • Updated Jul 18 • 1.22k
neuralmagic/Meta-Llama-3-8B-Instruct-quantized.w4a16

Text Generation • Updated Jul 18 • 381 • 2
neuralmagic/Meta-Llama-3-70B-Instruct-quantized.w4a16

Text Generation • Updated 8 days ago • 375k • 2
neuralmagic/Llama-2-7b-chat-quantized.w4a16

Text Generation • Updated Jul 18 • 43
neuralmagic/Mistral-7B-Instruct-v0.3-quantized.w4a16

Text Generation • Updated Jul 18 • 63