---
license: cc-by-nc-4.0
language:
- en
pipeline_tag: text-generation
tags:
- text-generation-inference
- meta
- llama
- llama-3.1
- autoawq
- awq
---

# Model Information

The Meta Llama 3.1 collection of multilingual large language models (LLMs) is a collection of pretrained and instruction tuned generative models in 8B, 70B and 405B sizes (text in/text out). The Llama 3.1 instruction tuned text only models (8B, 70B, 405B) are optimized for multilingual dialogue use cases and outperform many of the available open source and closed chat models on common industry benchmarks.

# Quantized Model

This repo contains the meta-llama/Meta-Llama-3.1-8B quantized down to INT4 with AutoAWQ using GEMM kernels performing zero-point quantization with a group size of 128.

# Usage Requirements

The model inference requires 7.29 GB of VRAM to load the model checkpoint. This model has been tested to run on a T4 GPU but can be run on a slightly less powerful machine if needed.

# Running The Model

Please be sure to install the following packages before running the sample code.

```terminal
pip install -q --upgrade transformers accelerate
pip install autoawq
```
Below is an example of how this model can be run. Simply replace the 'prompt' with an input of your choosing and adjust the maximum token size as needed.

```python
import torch
from transformers import AutoTokenizer, AwqConfig, AutoModelForCausalLM, pipeline
from huggingface_hub import login

#This code below is if you are using a hugging face token on google colab.
from google.colab import userdata
my_token = userdata.get("HF_TOKEN")
login(my_token)

model_name = "clearcash/Llama-3.1-8B-AWQ-INT4"

quantization_config = AwqConfig(
bits=4,
fuse_max_seq_len=512, # Note: Update this as per your use-case
do_fuse=True,
)

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype=torch.float16,
low_cpu_mem_usage=True,
device_map="auto",
quantization_config=quantization_config
)

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype=torch.float16,
low_cpu_mem_usage=True,
device_map="auto",
quantization_config=quantization_config
)

text_generator = pipeline(
'text-generation',
model = model,
tokenizer = tokenizer,
max_new_tokens=456,
)

def get_response(prompt):
response = text_generator(prompt)
gen_text = response[0]['generated_text']
return gen_text

prompt = "generate python code to make a bar graph"
llama_response = get_response(prompt)
print(llama_response)
```

# Final Statements

I hope this model proves useful to you in your projects. Please reach out if you would like to use this model for commercial purposes. Good luck and happy coding!

### Acknowledgements

The example above uses code from the following sources:

- hugging-quants/Meta-Llama-3.1-405B-Instruct-AWQ-INT4
- Author: hugging-quants
- Source: https://huggingface.co/hugging-quants/Meta-Llama-3.1-405B-Instruct-AWQ-INT4
- License: Llama 3.1 Community License Agreement