weblab-10b-instruction-sft
Overview
This repository provides a Japanese-centric multilingual GPT-NeoX model of 10 billion parameters.
Library
The model was trained using code based on EleutherAI/gpt-neox.
Model architecture
A 36-layer, 4864-hidden-size transformer-based language model.
Pre-training
The model was trained on around 600B tokens from a mixture of the following corpora.
Instruction-supervised-finetuning
The model was finetuned on a subset records from a mixture of the following dataset. Training epoch: 1.
Model Series
Variant Link weblab-10b-instruction-sft https://huggingface.co/matsuo-lab/weblab-10b-instruction-sft weblab-10b https://huggingface.co/matsuo-lab/weblab-10b Authors
Takeshi Kojima
Benchmarking
Japanese benchmark : JGLUE 8-task (2023-08-27)
- We used Stability-AI/lm-evaluation-harness library for evaluation.
- The 8-task average accuracy is based on results of JCommonsenseQA-1.1, JNLI-1.1, MARC-ja-1.1, JSQuAD-1.1, jaqket_v2-0.2, xlsum_ja-1.0, xwinograd_ja, and mgsm-1.0.
- model loading is performed with float16, and evaluation is performed with template version 0.3 using the few-shot in-context learning.
- The number of few-shots is 3,3,3,2,1,1,0,5.
- special_tokens_map.json is modified to avoid errors during the evaluation of the second half benchmarks. As a result, the results of the first half benchmarks became slightly different.
model average jcommonsenseqa jnli marc_ja jsquad jaqket_v2 xlsum_ja xwinograd_ja mgsm weblab-10b-instruction-sft 59.11 74.62 66.56 95.49 78.34 63.32 20.57 71.95 2 weblab-10b 50.74 66.58 53.74 82.07 62.94 56.19 10.03 71.95 2.4 Japanese benchmark : JGLUE 4-task (2023-08-18)
- We used Stability-AI/lm-evaluation-harness library for evaluation.
- The 4-task average accuracy is based on results of JCommonsenseQA-1.1, JNLI-1.1, MARC-ja-1.1, and JSQuAD-1.1.
- model loading is performed with float16, and evaluation is performed with template version 0.3 using the few-shot in-context learning.
- The number of few-shots is 3,3,3,2.
Model Average JCommonsenseQA JNLI MARC-ja JSQuAD weblab-10b-instruction-sft 78.78 74.35 65.65 96.06 79.04 weblab-10b 66.38 65.86 54.19 84.49 60.98
How to use the model
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("matsuo-lab/weblab-10b-instruction-sft")
model = AutoModelForCausalLM.from_pretrained("matsuo-lab/weblab-10b-instruction-sft", torch_dtype=torch.float16)
if torch.cuda.is_available():
model = model.to("cuda")
text = "倧θ¦ζ¨‘θ¨θͺγ’γγ«γ«γ€γγ¦θͺ¬ζγγ¦γγ γγγ"
text = f'δ»₯δΈγ―γγΏγΉγ―γθͺ¬ζγγζη€Ίγ§γγθ¦ζ±γι©εγ«ζΊγγεΏηγζΈγγͺγγγ\n\n### ζη€Ί:\n{text}\n\n### εΏη:'
token_ids = tokenizer.encode(text, add_special_tokens=False, return_tensors="pt")
with torch.no_grad():
output_ids = model.generate(
token_ids.to(model.device),
max_new_tokens=100,
do_sample=True,
temperature=0.7,
top_p=0.95
)
output = tokenizer.decode(output_ids.tolist()[0])
print(output)
Licenese
- Downloads last month
- 725
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.