Updated Readme

README.md

@@ -1,201 +1,240 @@
 ---
 library_name: transformers
-tags: []
+tags:
+- 8bit
+- bnb
+- bitsandbytes
+- llama
+- llama-2
+- facebook
+- meta
+- 7b
+- quantized
+license: llama2
+pipeline_tag: text-generation
 ---
 
-# Model Card for Model ID
+# Model Card for alokabhishek/Llama-2-7b-chat-hf-bnb-8bit
 
 <!-- Provide a quick summary of what the model is/does. -->
 
-
+This repo contains 8-bit quantized (using bitsandbytes) model of Meta's meta-llama/Llama-2-7b-chat-hf
 
 ## Model Details
 
-### Model Description
-
-<!-- Provide a longer summary of what this model is. -->
+- Model creator: [Meta](https://huggingface.co/meta-llama)
+- Original model: [Llama-2-7b-chat-hf](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf)
 
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+### About 4 bit quantization using bitsandbytes
 
-### Model Sources [optional]
 
-<!-- Provide the basic links for the model. -->
+QLoRA: Efficient Finetuning of Quantized LLMs: [arXiv - QLoRA: Efficient Finetuning of Quantized LLMs](https://arxiv.org/abs/2305.14314)
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+Hugging Face Blog post on 4-bit quantization using bitsandbytes: [Making LLMs even more accessible with bitsandbytes, 4-bit quantization and QLoRA](https://huggingface.co/blog/4bit-transformers-bitsandbytes)
 
-## Uses
+bitsandbytes github repo: [bitsandbytes github repo](https://github.com/TimDettmers/bitsandbytes)
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+# How to Get Started with the Model
 
-### Direct Use
+Use the code below to get started with the model.
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+## How to run from Python code
 
-[More Information Needed]
+#### First install the package
+```shell
+pip install --quiet bitsandbytes
+pip install --quiet --upgrade transformers # Install latest version of transformers
+pip install --quiet --upgrade accelerate
+pip install --quiet sentencepiece
+pip install -q -U torch huggingface_hub
+pip install flash-attn --no-build-isolation
+```
 
-### Downstream Use [optional]
+#### Import 
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+```python
+import torch
+import os
+from torch import bfloat16
+from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline, BitsAndBytesConfig, LlamaForCausalLM
+```
 
-[More Information Needed]
+#### Use a pipeline as a high-level helper
 
-### Out-of-Scope Use
+```python
+model_id_llama = "alokabhishek/Llama-2-7b-chat-hf-bnb-8bit"
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+tokenizer_llama = AutoTokenizer.from_pretrained(model_id_llama, use_fast=True)
 
-[More Information Needed]
+model_llama = AutoModelForCausalLM.from_pretrained(
+    model_id_llama,
+    device_map="auto"
+)
 
-## Bias, Risks, and Limitations
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+pipe_llama = pipeline(model=model_llama, tokenizer=tokenizer_llama, task='text-generation')
 
-[More Information Needed]
+prompt_llama = "Tell me a funny joke about Large Language Models meeting a Blackhole in an intergalactic Bar."
 
-### Recommendations
+output_llama = pipe_llama(prompt_llama, max_new_tokens=512)
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+print(output_llama[0]["generated_text"])
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+```
 
-## How to Get Started with the Model
 
-Use the code below to get started with the model.
 
-[More Information Needed]
+## Uses
 
-## Training Details
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
 
-### Training Data
+### Direct Use
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
 
 [More Information Needed]
 
-### Training Procedure 
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+### Downstream Use [optional]
 
-#### Preprocessing [optional]
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
 
 [More Information Needed]
 
+### Out-of-Scope Use
 
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
 
 [More Information Needed]
 
-## Evaluation
-
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
+## Bias, Risks, and Limitations
 
-<!-- This should link to a Dataset Card if possible. -->
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
 
 [More Information Needed]
 
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
 
-#### Metrics
 
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+## Model Card Authors [optional]
 
 [More Information Needed]
 
-### Results
+## Model Card Contact
 
 [More Information Needed]
 
-#### Summary
+# **Llama 2**
+Llama 2 is a collection of pretrained and fine-tuned generative text models ranging in scale from 7 billion to 70 billion parameters. This is the repository for the 7B fine-tuned model, optimized for dialogue use cases and converted for the Hugging Face Transformers format. Links to other models can be found in the index at the bottom.
+
+## Model Details
+*Note: Use of this model is governed by the Meta license. In order to download the model weights and tokenizer, please visit the [website](https://ai.meta.com/resources/models-and-libraries/llama-downloads/) and accept our License before requesting access here.*
 
+Meta developed and publicly released the Llama 2 family of large language models (LLMs), a collection of pretrained and fine-tuned generative text models ranging in scale from 7 billion to 70 billion parameters. Our fine-tuned LLMs, called Llama-2-Chat, are optimized for dialogue use cases. Llama-2-Chat models outperform open-source chat models on most benchmarks we tested, and in our human evaluations for helpfulness and safety, are on par with some popular closed-source models like ChatGPT and PaLM.
 
+**Model Developers** Meta
 
-## Model Examination [optional]
+**Variations** Llama 2 comes in a range of parameter sizes — 7B, 13B, and 70B — as well as pretrained and fine-tuned variations.
 
-<!-- Relevant interpretability work for the model goes here -->
+**Input** Models input text only.
 
-[More Information Needed]
+**Output** Models generate text only.
 
-## Environmental Impact
+**Model Architecture** Llama 2 is an auto-regressive language model that uses an optimized transformer architecture. The tuned versions use supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) to align to human preferences for helpfulness and safety.
 
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
 
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+||Training Data|Params|Content Length|GQA|Tokens|LR|
+|---|---|---|---|---|---|---|
+|Llama 2|*A new mix of publicly available online data*|7B|4k|&#10007;|2.0T|3.0 x 10<sup>-4</sup>|
+|Llama 2|*A new mix of publicly available online data*|13B|4k|&#10007;|2.0T|3.0 x 10<sup>-4</sup>|
+|Llama 2|*A new mix of publicly available online data*|70B|4k|&#10004;|2.0T|1.5 x 10<sup>-4</sup>|
 
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
+*Llama 2 family of models.* Token counts refer to pretraining data only. All models are trained with a global batch-size of 4M tokens. Bigger models -  70B -- use Grouped-Query Attention (GQA) for improved inference scalability.
 
-## Technical Specifications [optional]
+**Model Dates** Llama 2 was trained between January 2023 and July 2023.
 
-### Model Architecture and Objective
+**Status** This is a static model trained on an offline dataset. Future versions of the tuned models will be released as we improve model safety with community feedback.
 
-[More Information Needed]
+**License** A custom commercial license is available at: [https://ai.meta.com/resources/models-and-libraries/llama-downloads/](https://ai.meta.com/resources/models-and-libraries/llama-downloads/)
 
-### Compute Infrastructure
+**Research Paper** ["Llama-2: Open Foundation and Fine-tuned Chat Models"](arxiv.org/abs/2307.09288)
 
-[More Information Needed]
+## Intended Use
+**Intended Use Cases** Llama 2 is intended for commercial and research use in English. Tuned models are intended for assistant-like chat, whereas pretrained models can be adapted for a variety of natural language generation tasks.
 
-#### Hardware
+To get the expected features and performance for the chat versions, a specific formatting needs to be followed, including the `INST` and `<<SYS>>` tags, `BOS` and `EOS` tokens, and the whitespaces and breaklines in between (we recommend calling `strip()` on inputs to avoid double-spaces). See our reference code in github for details: [`chat_completion`](https://github.com/facebookresearch/llama/blob/main/llama/generation.py#L212).
 
-[More Information Needed]
+**Out-of-scope Uses** Use in any manner that violates applicable laws or regulations (including trade compliance laws).Use in languages other than English. Use in any other way that is prohibited by the Acceptable Use Policy and Licensing Agreement for Llama 2.
 
-#### Software
+## Hardware and Software
+**Training Factors** We used custom training libraries, Meta's Research Super Cluster, and production clusters for pretraining. Fine-tuning, annotation, and evaluation were also performed on third-party cloud compute.
 
-[More Information Needed]
+**Carbon Footprint** Pretraining utilized a cumulative 3.3M GPU hours of computation on hardware of type A100-80GB (TDP of 350-400W). Estimated total emissions were 539 tCO2eq, 100% of which were offset by Meta’s sustainability program.
 
-## Citation [optional]
+||Time (GPU hours)|Power Consumption (W)|Carbon Emitted(tCO<sub>2</sub>eq)|
+|---|---|---|---|
+|Llama 2 7B|184320|400|31.22|
+|Llama 2 13B|368640|400|62.44|
+|Llama 2 70B|1720320|400|291.42|
+|Total|3311616||539.00|
 
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+**CO<sub>2</sub> emissions during pretraining.** Time: total GPU time required for training each model. Power Consumption: peak power capacity per GPU device for the GPUs used adjusted for power usage efficiency. 100% of the emissions are directly offset by Meta's sustainability program, and because we are openly releasing these models, the pretraining costs do not need to be incurred by others.
 
-**BibTeX:**
+## Training Data
+**Overview** Llama 2 was pretrained on 2 trillion tokens of data from publicly available sources. The fine-tuning data includes publicly available instruction datasets, as well as over one million new human-annotated examples. Neither the pretraining nor the fine-tuning datasets include Meta user data.
 
-[More Information Needed]
+**Data Freshness** The pretraining data has a cutoff of September 2022, but some tuning data is more recent, up to July 2023.
 
-**APA:**
+## Evaluation Results
 
-[More Information Needed]
+In this section, we report the results for the Llama 1 and Llama 2 models on standard academic benchmarks.For all the evaluations, we use our internal evaluations library.
 
-## Glossary [optional]
+|Model|Size|Code|Commonsense Reasoning|World Knowledge|Reading Comprehension|Math|MMLU|BBH|AGI Eval|
+|---|---|---|---|---|---|---|---|---|---|
+|Llama 1|7B|14.1|60.8|46.2|58.5|6.95|35.1|30.3|23.9|
+|Llama 1|13B|18.9|66.1|52.6|62.3|10.9|46.9|37.0|33.9|
+|Llama 1|33B|26.0|70.0|58.4|67.6|21.4|57.8|39.8|41.7|
+|Llama 1|65B|30.7|70.7|60.5|68.6|30.8|63.4|43.5|47.6|
+|Llama 2|7B|16.8|63.9|48.9|61.3|14.6|45.3|32.6|29.3|
+|Llama 2|13B|24.5|66.9|55.4|65.8|28.7|54.8|39.4|39.1|
+|Llama 2|70B|**37.5**|**71.9**|**63.6**|**69.4**|**35.2**|**68.9**|**51.2**|**54.2**|
 
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+**Overall performance on grouped academic benchmarks.** *Code:* We report the average pass@1 scores of our models on HumanEval and MBPP. *Commonsense Reasoning:* We report the average of PIQA, SIQA, HellaSwag, WinoGrande, ARC easy and challenge, OpenBookQA, and CommonsenseQA. We report 7-shot results for CommonSenseQA and 0-shot results for all other benchmarks. *World Knowledge:* We evaluate the 5-shot performance on NaturalQuestions and TriviaQA and report the average. *Reading Comprehension:* For reading comprehension, we report the 0-shot average on SQuAD, QuAC, and BoolQ. *MATH:* We report the average of the GSM8K (8 shot) and MATH (4 shot) benchmarks at top 1.
 
-[More Information Needed]
+|||TruthfulQA|Toxigen|
+|---|---|---|---|
+|Llama 1|7B|27.42|23.00|
+|Llama 1|13B|41.74|23.08|
+|Llama 1|33B|44.19|22.57|
+|Llama 1|65B|48.71|21.77|
+|Llama 2|7B|33.29|**21.25**|
+|Llama 2|13B|41.86|26.10|
+|Llama 2|70B|**50.18**|24.60|
 
-## More Information [optional]
+**Evaluation of pretrained LLMs on automatic safety benchmarks.** For TruthfulQA, we present the percentage of generations that are both truthful and informative (the higher the better). For ToxiGen, we present the percentage of toxic generations (the smaller the better).
 
-[More Information Needed]
 
-## Model Card Authors [optional]
+|||TruthfulQA|Toxigen|
+|---|---|---|---|
+|Llama-2-Chat|7B|57.04|**0.00**|
+|Llama-2-Chat|13B|62.18|**0.00**|
+|Llama-2-Chat|70B|**64.14**|0.01|
 
-[More Information Needed]
+**Evaluation of fine-tuned LLMs on different safety datasets.** Same metric definitions as above.
 
-## Model Card Contact
+## Ethical Considerations and Limitations
+Llama 2 is a new technology that carries risks with use. Testing conducted to date has been in English, and has not covered, nor could it cover all scenarios. For these reasons, as with all LLMs, Llama 2’s potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate, biased or other objectionable responses to user prompts. Therefore, before deploying any applications of Llama 2, developers should perform safety testing and tuning tailored to their specific applications of the model.
 
-[More Information Needed]
+Please see the Responsible Use Guide available at [https://ai.meta.com/llama/responsible-use-guide/](https://ai.meta.com/llama/responsible-use-guide)
 
+## Reporting Issues
+Please report any software “bug,” or other problems with the models through one of the following means:
+- Reporting issues with the model: [github.com/facebookresearch/llama](http://github.com/facebookresearch/llama)
+- Reporting problematic content generated by the model: [developers.facebook.com/llama_output_feedback](http://developers.facebook.com/llama_output_feedback)
+- Reporting bugs and security concerns: [facebook.com/whitehat/info](http://facebook.com/whitehat/info)
 
+## Llama Model Index
+|Model|Llama2|Llama2-hf|Llama2-chat|Llama2-chat-hf|
+|---|---|---|---|---|
+|7B| [Link](https://huggingface.co/meta-llama/Llama-2-7b) | [Link](https://huggingface.co/meta-llama/Llama-2-7b-hf) | [Link](https://huggingface.co/meta-llama/Llama-2-7b-chat) | [Link](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf)|
+|13B| [Link](https://huggingface.co/meta-llama/Llama-2-13b) | [Link](https://huggingface.co/meta-llama/Llama-2-13b-hf) | [Link](https://huggingface.co/meta-llama/Llama-2-13b-chat) | [Link](https://huggingface.co/meta-llama/Llama-2-13b-chat-hf)|
+|70B| [Link](https://huggingface.co/meta-llama/Llama-2-70b) | [Link](https://huggingface.co/meta-llama/Llama-2-70b-hf) | [Link](https://huggingface.co/meta-llama/Llama-2-70b-chat) | [Link](https://huggingface.co/meta-llama/Llama-2-70b-chat-hf)|