File size: 2,318 Bytes
5a99b54
 
 
 
 
 
 
 
 
 
 
cbbe0bc
 
 
 
d417c1f
cbbe0bc
 
 
5a99b54
 
 
 
 
 
a2c5955
5a99b54
 
 
 
 
 
a2c5955
5a99b54
 
 
 
 
 
 
 
 
 
 
cbbe0bc
 
 
5a99b54
 
cbbe0bc
5a99b54
cbbe0bc
 
 
 
 
d417c1f
5a99b54
 
 
 
 
 
 
d417c1f
 
 
 
5a99b54
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
d417c1f
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
---
title: Alpaca LoRa 7B
language: en
license: other
tags:
- alpaca
- lora
- llama
- peft
---

# Alpaca LoRa 7B

This repository contains a LLaMA-7B fine-tuned model on the [Standford Alpaca](https://github.com/tatsu-lab/stanford_alpaca) cleaned version dataset.

⚠️ **I used [LLaMA-7B-hf](decapoda-research/llama-7b-hf) as a base model, so this model is for Research purpose only (See the [license](https://huggingface.co/decapoda-research/llama-7b-hf/blob/main/LICENSE))**

# Usage

## Creating prompt

The model was trained on the following kind of prompt:

```python
def generate_prompt(instruction: str, input_ctxt: str = None) -> str:
    if input_ctxt:
        return f"""Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
{instruction}

### Input:
{input_ctxt}

### Response:"""
    else:
        return f"""Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{instruction}

### Response:"""
```

## Using the model

```python
import torch
from transformers import GenerationConfig, LlamaTokenizer, LlamaForCausalLM

tokenizer = LlamaTokenizer.from_pretrained("chainyo/alpaca-lora-7b")
model = LlamaForCausalLM.from_pretrained(
    "chainyo/alpaca-lora-7b",
    load_in_8bit=True,
    torch_dtype=torch.float16,
    device_map="auto",
)
generation_config = GenerationConfig(
    temperature=0.2,
    top_p=0.75,
    top_k=40,
    num_beams=4,
    max_new_tokens=128,
)

model.eval()
if torch.__version__ >= "2":
    model = torch.compile(model)

instruction = "What is the meaning of life?"
input_ctxt = None  # For some tasks, you can provide an input context to help the model generate a better response.

prompt = generate_prompt(instruction, input_ctxt)
input_ids = tokenizer(prompt, return_tensors="pt").input_ids
input_ids = input_ids.to(model.device)

with torch.no_grad():
    outputs = model.generate(
        input_ids=input_ids,
        generation_config=generation_config,
        return_dict_in_generate=True,
        output_scores=True,
    )

response = tokenizer.decode(outputs.sequences[0], skip_special_tokens=True)
print(response)

>>> The meaning of life is to live a life of meaning.
```