File size: 10,012 Bytes
4ba1c97
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
---
pipeline_tag: text-generation
inference: false
license: apache-2.0
# datasets:
# metrics:
# - code_eval
library_name: transformers
tags:
- language
- granite-3.0
model-index:
- name: granite-3.0-3b-a800m-instruct
  results:
  - task:
      type: text-generation
    dataset:
        type: human-exams
        name: MMLU
    metrics:
    - name: pass@1
      type: pass@1
      value: 
      veriefied: false
  - task:
      type: text-generation
    dataset:
        type: human-exams
        name: MMLU-Pro
    metrics:
    - name: pass@1
      type: pass@1
      value: 
      veriefied: false
  - task:
      type: text-generation
    dataset:
        type: human-exams
        name: AGI-Eval
    metrics:
    - name: pass@1
      type: pass@1
      value: 
      veriefied: false
  - task:
      type: text-generation
    dataset:
        type: commonsense
        name: WinoGrande
    metrics:
    - name: pass@1
      type: pass@1
      value: 
      veriefied: false
  - task:
      type: text-generation
    dataset:
        type: commonsense
        name: OBQA 
    metrics:
    - name: pass@1
      type: pass@1
      value: 
      veriefied: false
  - task:
      type: text-generation
    dataset:
        type: commonsense
        name: SIQA
    metrics:
    - name: pass@1
      type: pass@1
      value: 
      veriefied: false
  - task:
      type: text-generation
    dataset:
        type: commonsense
        name: PIQA
    metrics:
    - name: pass@1
      type: pass@1
      value: 
      veriefied: false
  - task:
      type: text-generation
    dataset:
        type: commonsense
        name: Hellaswag
    metrics:
    - name: pass@1
      type: pass@1
      value: 
      veriefied: false
  - task:
      type: text-generation
    dataset:
        type: commonsense
        name: TruthfulQA
    metrics:
    - name: pass@1
      type: pass@1
      value: 
      veriefied: false
  - task:
      type: text-generation
    dataset:
        type: reading-comprehension
        name: BoolQ
    metrics:
    - name: pass@1
      type: pass@1
      value: 
      veriefied: false
  - task:
      type: text-generation
    dataset:
        type: reading-comprehension
        name: SQuAD v2
    metrics:
    - name: pass@1
      type: pass@1
      value: 
      veriefied: false
  - task:
      type: text-generation
    dataset:
        type: reasoning
        name: ARC-C
    metrics:
    - name: pass@1
      type: pass@1
      value: 
      veriefied: false
  - task:
      type: text-generation
    dataset:
        type: reasoning
        name: GPQA
    metrics:
    - name: pass@1
      type: pass@1
      value: 
      veriefied: false
  - task:
      type: text-generation
    dataset:
        type: reasoning
        name: BBH
    metrics:
    - name: pass@1
      type: pass@1
      value: 
      veriefied: false
  - task:
      type: text-generation
    dataset:
        type: code
        name: HumanEval
    metrics:
    - name: pass@1
      type: pass@1
      value: 
      veriefied: false
  - task:
      type: text-generation
    dataset:
        type: code
        name: MBPP
    metrics:
    - name: pass@1
      type: pass@1
      value: 
      veriefied: false     
  - task:
      type: text-generation
    dataset:
        type: math
        name: GSM8K
    metrics:
    - name: pass@1
      type: pass@1
      value: 
      veriefied: false 
  - task:
      type: text-generation
    dataset:
        type: math
        name: MATH
    metrics:
    - name: pass@1
      type: pass@1
      value: 
      veriefied: false   
  - task:
      type: text-generation
    dataset:
        type: multilingual
        name: MGSM
    metrics:
    - name: pass@1
      type: pass@1
      value: 
      veriefied: false
---

<!-- ![image/png](https://cdn-uploads.huggingface.co/production/uploads/62cd5057674cdb524450093d/1hzxoPwqkBJXshKVVe6_9.png) -->

# Granite-3.0-3B-A800M-Instruct

## Model Summary
**Granite-3.0-3B-A800M-Instruct** is a lightweight and open-source 3B parameter model fine tuned from *Granite-3.0-3B-A800M-Base-4K* on a combination of open-source and proprietary instruction data with a **permissively licensed**. This language model is designed to excel in instruction following tasks such as summarization, problem-solving, text translation, reasoning, code tasks, funcion-calling, and more.

- **Developers:** IBM Research
- **GitHub Repository:** [ibm-granite/granite-language-models](https://github.com/ibm-granite/granite-language-models)
- **Website**: [Granite Docs](https://www.ibm.com/granite/docs/)
- **Paper:** [Granite Language Models](https://) <!--     TO DO: Update github repo link when it is ready -->
- **Release Date**: October 21st, 2024
- **License:** [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0).

## Supported Languages
English, German, Spanish, French, Japanese, Portuguese, Arabic, Czech, Italian, Korean, Dutch, Chinese (Simplified) 

## Usage
### Intended use
The model is designed to respond to general instructions and can be used to build AI assistants for multiple domains, including bussiness applications.

### Capabilities
* Summarization
* Text classification
* Text extraction
* Question-answering
* Retrieval Augmented Generation (RAG)
* Code related 
* Function-calling
* Multilingual dialog use cases

### Generation
This is a simple example of how to use **Granite-3.0-3B-A800M-Instruct** model.

Install the following libraries:

```shell
pip install torch torchvision torchaudio
pip install accelerate
pip install transformers
```
Then, copy the snippet from the section that is relevant for your usecase.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

device = "auto"
model_path = "ibm-granite/granite-3.0-3b-a800m-instruct"
tokenizer = AutoTokenizer.from_pretrained(model_path)
# drop device_map if running on CPU
model = AutoModelForCausalLM.from_pretrained(model_path, device_map=device)
model.eval()
# change input text as desired
chat = [
    { "role": "user", "content": "Please list one IBM Research laboratory located in the United States. You should only output its name and location." },
]
chat = tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True)
# tokenize the text
input_tokens = tokenizer(chat, return_tensors="pt").to(device)
# generate output tokens
output = model.generate(**input_tokens, 
                        max_new_tokens=100)
# decode output tokens into text
output = tokenizer.batch_decode(output)
# print output
print(output)
```

<!-- TO DO: function-calling-example
 -->

<!-- ['<|start_of_role|>user<|end_of_role|>Please list one IBM Research laboratory located in the United States. You should only output its name and location.<|end_of_text|>\n<|start_of_role|>assistant<|end_of_role|>1. IBM Research - Almaden, San Jose, California<|end_of_text|>'] -->

## Model Architeture
**Granite-3.0-3B-A800M-Instruct** is based on a decoder-only sparse Mixture of Experts(MoE) transformer architecture. Core components of this architecture are: Fine-grained Experts, Dropless Token Routing, and Load Balancing Loss.

| Model                        | 2B Dense | 8B Dense | 1B MoE   | 3B MoE       |
| :--------                    | :--------| :--------| :--------| :--------    |
| Embedding size               | 2048     | 4096     | 1024     | **1536**     |
| Number of layers             | 40       | 40       | 24       | **32**       |
| Attention head size          | 64       | 128      | 64       | **64**       |
| Number of attention heads    | 32       | 32       | 16       | **24**       |
| Number of KV heads           | 8        | 8        | 8        | **8**        |
| MLP hidden size              | 8192     | 12800    | 512      | **512**      |
| MLP activation               | SwiGLU   | SwiGLU   | SwiGLU   | **SwiGLU**   |
| Number of Experts            | —        | —        | 32       | **40**       |
| MoE TopK                     | —        | —        | 8        | **8**        |
| Initialization std           | 0.1      | 0.1      | 0.1      | **0.1**      |
| Sequence Length              | 4096     | 4096     | 4096     | **4096**     |
| Position Embedding           | RoPE     | RoPE     | RoPE     | **RoPE**     |
| # Paremeters                 | 2.5B     | 8.1B     | 1.3B     | **3.3B**     |
| # Active Parameters          | 2.5B     | 8.1B     | 400M     | **800M**     |
| # Training tokens            | 12T      | 12T      | 10T      | **10T**      |

<!-- TO DO: To be completed once the paper is ready, we may changed title to Supervised Finetuning -->
## Training Data
This model is trained on a mix of open-source and proprietary datasets.
<!-- ### Instruction Datasets
* Language Instruction Datasets: We include high-quality datasets such as [TO DO: List of datasets]
* Synthetic Instruction Datasets: [TO DO: paragraph about synthetic data]
### Processing
* [TO DO: Data annotation with MagPie pipeline: quality, duplicates] -->

## Infrastructure
We train the Granite Language models using IBM's super computing cluster, Blue Vela, which is outfitted with NVIDIA H100 GPUs. This cluster provides a scalable and efficient infrastructure for training our models over thousands of GPUs.

## Ethical Considerations and Limitations
Granite instruct models are primarily finetuned using instruction-response pairs mostly in English, but also in German, Spanish, French, Japanese, Portuguese, Arabic, Czech, Italian, Korean, Dutch, and Chinese (Simplified). As this model has been exposed to multilingual data, it can handle multilingual dialog use cases with a limited performance in non-English tasks. In such case, introducing a small number of examples (few-shot) can help the model in generating more accurate outputs. The model also inherits ethical considerations and limitations from its base model. For more information, please refer to *[Granite-3.0-3B-A800M-Base-4K](https://huggingface.co/ibm-granite/granite-3.0-3b-a800m-base)* model card.