INT8 LLMs for vLLM - a neuralmagic Collection

neuralmagic 's Collections

FP8 LLMs for vLLM

Llama-3.1 Quantization

INT8 LLMs for vLLM

INT4 LLMs for vLLM

Sparse Foundational Llama 2 Models

Compression Papers

DeepSparse Sparse LLMs

Sparse Finetuning MPT

Compressed LLMs from the Community

INT8 LLMs for vLLM

updated 8 days ago

Accurate INT8 quantized models by Neural Magic, ready for use with vLLM!

neuralmagic/Meta-Llama-3.1-70B-Instruct-quantized.w8a8

Text Generation • Updated 21 days ago • 2.37k • 5
neuralmagic/Meta-Llama-3.1-8B-Instruct-quantized.w8a8

Text Generation • Updated 24 days ago • 1.12k • 8
neuralmagic/Meta-Llama-3.1-405B-Instruct-quantized.w8a8

Text Generation • Updated 15 days ago • 112 • 1
neuralmagic/Phi-3-medium-128k-instruct-quantized.w8a8

Text Generation • Updated 15 days ago • 125 • 1
neuralmagic/Phi-3-mini-128k-instruct-quantized.w8a8

Text Generation • Updated Jul 18 • 107
neuralmagic/gemma-2-9b-it-quantized.w8a8

Text Generation • Updated 15 days ago • 20 • 1
neuralmagic/Meta-Llama-3-70B-Instruct-quantized.w8a16

Text Generation • Updated Jul 18 • 87 • 2
neuralmagic/Qwen2-72B-Instruct-quantized.w8a16

Text Generation • Updated Jul 18 • 43 • 1
neuralmagic/Llama-2-7b-chat-quantized.w8a16

Text Generation • Updated Jul 18 • 166
neuralmagic/Meta-Llama-3-8B-Instruct-quantized.w8a16

Text Generation • Updated Jul 18 • 483 • 1
neuralmagic/Qwen2-0.5B-Instruct-quantized.w8a16

Text Generation • Updated Jul 18 • 22
neuralmagic/Qwen2-1.5B-Instruct-quantized.w8a16

Text Generation • Updated Jul 18 • 3
neuralmagic/Qwen2-7B-Instruct-quantized.w8a16

Text Generation • Updated Jul 18 • 32
neuralmagic/Mistral-7B-Instruct-v0.3-quantized.w8a16

Text Generation • Updated Jul 18 • 133
neuralmagic/Phi-3-mini-128k-instruct-quantized.w8a16

Text Generation • Updated 22 days ago • 25
neuralmagic/Phi-3-medium-128k-instruct-quantized.w8a16

Text Generation • Updated 22 days ago • 160 • 1
neuralmagic/Meta-Llama-3-8B-Instruct-quantized.w8a8

Text Generation • Updated Jul 18 • 218 • 2
neuralmagic/Llama-2-7b-chat-quantized.w8a8

Text Generation • Updated Jul 18 • 142 • 1
neuralmagic/Qwen2-0.5B-Instruct-quantized.w8a8

Text Generation • Updated Jul 18 • 10
neuralmagic/Qwen2-1.5B-Instruct-quantized.w8a8

Text Generation • Updated Jul 18 • 1.4k
neuralmagic/Qwen2-7B-Instruct-quantized.w8a8

Text Generation • Updated Jul 18 • 16
neuralmagic/Qwen2-72B-Instruct-quantized.w8a8

Text Generation • Updated Jul 18 • 20 • 1
neuralmagic/Meta-Llama-3-70B-Instruct-quantized.w8a8

Text Generation • Updated Jul 18 • 47
neuralmagic/Mistral-7B-Instruct-v0.3-quantized.w8a8

Text Generation • Updated Jul 18 • 28
neuralmagic/Meta-Llama-3.1-8B-Instruct-quantized.w8a16

Text Generation • Updated 24 days ago • 4.49k • 7
neuralmagic/Meta-Llama-3.1-70B-Instruct-quantized.w8a16

Text Generation • Updated 24 days ago • 597
neuralmagic/Meta-Llama-3.1-8B-quantized.w8a16

Text Generation • Updated 24 days ago • 95 • 1
neuralmagic/Meta-Llama-3.1-8B-quantized.w8a8

Text Generation • Updated 16 days ago • 289 • 1
neuralmagic/starcoder2-7b-quantized.w8a16

Text Generation • Updated Aug 2 • 8
neuralmagic/starcoder2-15b-quantized.w8a16

Text Generation • Updated Aug 2 • 10
neuralmagic/starcoder2-3b-quantized.w8a16

Text Generation • Updated Aug 2 • 5
neuralmagic/starcoder2-15b-quantized.w8a8

Text Generation • Updated Aug 2 • 3
neuralmagic/starcoder2-7b-quantized.w8a8

Text Generation • Updated Aug 2 • 2
neuralmagic/starcoder2-3b-quantized.w8a8

Text Generation • Updated Aug 2 • 3
neuralmagic/gemma-2-2b-it-quantized.w8a16

Text Generation • Updated 21 days ago • 294 • 1
neuralmagic/Phi-3-small-128k-instruct-quantized.w8a16

Text Generation • Updated 22 days ago • 116
neuralmagic/SmolLM-1.7B-Instruct-quantized.w8a16

Text Generation • Updated 15 days ago • 13
neuralmagic/gemma-2-2b-quantized.w8a16

Text Generation • Updated 21 days ago • 39
neuralmagic/gemma-2-9b-it-quantized.w8a16

Text Generation • Updated 15 days ago • 81 • 1
neuralmagic/gemma-2-2b-it-quantized.w8a8

Text Generation • Updated 21 days ago • 7
neuralmagic/Meta-Llama-3.1-405B-Instruct-quantized.w8a16

Text Generation • Updated 16 days ago • 69 • 2
neuralmagic/SmolLM-360M-Instruct-quantized.w8a8

Text Generation • Updated 14 days ago • 7
neuralmagic/SmolLM-135M-Instruct-quantized.w8a8

Text Generation • Updated 14 days ago • 8