File size: 3,243 Bytes
4a173cf 2f5fa2d 37ea4ce ba675c7 2827054 841af07 ba675c7 ab6c6ec ba675c7 e7ebe1f ba675c7 f0d22b9 ba675c7 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 |
---
license: cc-by-nc-4.0
language:
- en
pipeline_tag: text-generation
tags:
- text-generation-inference
- meta
- llama
- llama-3.1
- autoawq
- awq
- '3.1'
- 8B
- base
---
## If You Would Like A Slightly More Robust Version Of This Model Please Check It Out [Here!](https://huggingface.co/UCLA-EMC/Meta-Llama-3.1-8B-Instruct-AWQ-INT4-32-2.17B)
# Model Information
The Meta Llama 3.1 collection of multilingual large language models (LLMs) is a collection of pretrained and instruction tuned generative models in 8B, 70B and 405B sizes (text in/text out). The Llama 3.1 instruction tuned text only models (8B, 70B, 405B) are optimized for multilingual dialogue use cases and outperform many of the available open source and closed chat models on common industry benchmarks.
# Quantized Model
This is the quantizied version of the BASE model of LLama-3.1-8B. This repo contains the meta-llama/Meta-Llama-3.1-8B quantized down to INT4 with AutoAWQ using GEMM kernels performing zero-point quantization with a group size of 128.
# Usage Requirements
The model inference requires 7.29 GB of VRAM to load the model checkpoint. This model has been tested to run on a T4 GPU but can be run on a slightly less powerful machine if needed.
# Running The Model
Please be sure to install the following packages before running the sample code.
```terminal
pip install -q --upgrade transformers accelerate
pip install autoawq
```
Below is an example of how this model can be run. Simply replace the 'prompt' with an input of your choosing and adjust the maximum token size as needed.
```python
import torch
from transformers import AutoTokenizer, AwqConfig, AutoModelForCausalLM, pipeline
from huggingface_hub import login
#This code below is if you are using a hugging face token on google colab.
from google.colab import userdata
my_token = userdata.get("HF_TOKEN")
login(my_token)
model_name = "UCLA-EMC/Meta-Llama-3.1-8B-AWQ-INT4"
quantization_config = AwqConfig(
bits=4,
fuse_max_seq_len=512, # Note: Update this as per your use-case
do_fuse=True,
)
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype=torch.float16,
low_cpu_mem_usage=True,
device_map="auto",
quantization_config=quantization_config
)
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype=torch.float16,
low_cpu_mem_usage=True,
device_map="auto",
quantization_config=quantization_config
)
text_generator = pipeline(
'text-generation',
model = model,
tokenizer = tokenizer,
max_new_tokens=456,
)
def get_response(prompt):
response = text_generator(prompt)
gen_text = response[0]['generated_text']
return gen_text
prompt = "generate python code to make a bar graph"
llama_response = get_response(prompt)
print(llama_response)
```
# Final Statements
I hope this model proves useful to you in your projects. Good luck and happy coding!
### Acknowledgements
The example above uses code from the following sources:
- hugging-quants/Meta-Llama-3.1-405B-Instruct-AWQ-INT4
- Author: hugging-quants
- Source: https://huggingface.co/hugging-quants/Meta-Llama-3.1-405B-Instruct-AWQ-INT4
- License: Llama 3.1 Community License Agreement |