license: mit
language:
- it
Task: CHAT
Model: DIABLO 🔥
Lang: IT
Model description
This model is a conversational language model for the Italian language, based on a GPT-like [1] architecture (more specifically, the model has been obtained by modifying Meta's XGLM architecture [2] and exploiting its 1.7B checkpoint).
The model has been trained on a corpus of ~50K Italian conversational exchanges for ~3 epochs (~15K steps with a batch size of 10), using 3 different learning rates (1e-5, 2e-6, 1e-6) and exploiting FP16 quantization to manage the considerable size of the model. The training corpus has been built by using Meta's Blenderbot [3] to generate 50K conversational exchanges in English, and then translating them to the Italian language using a machine translation model.
The current release is designed for brief and informal conversations (small talk) covering light topics (mainly food, entertainment and holidays), but several generalizations and improvements will be introduced in future releases.
Quick usage
In order to use the model for inference, the following pipeline is needed:
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
import re
tokenizer = AutoTokenizer.from_pretrained("osiria/diablo-italian-chatbot-1.3b")
model = AutoModelForCausalLM.from_pretrained("osiria/diablo-italian-chatbot-1.3b")
device = torch.device("cpu")
model = model.to(device)
model.eval()
class Diablo:
def __init__(self, tokenizer, model):
self.tokenizer = tokenizer
self.model = model
def _check_sublist(self, lst, sub_lst, sep = " "):
l_type = type(lst[0])
lst = sep.join(list(map(str, lst)))
sub_lst = sep.join(list(map(str, sub_lst)))
return sub_lst in lst
def _exclude_sublist(self, lst, sub_lst, sep = " "):
l_type = type(lst[0])
lst = sep.join(list(map(str, lst)))
sub_lst = sep.join(list(map(str, sub_lst)))
lst = re.sub("\s+", " ", lst.replace(sub_lst, "")).strip().split(sep)
lst = list(map(l_type, lst))
return lst
def generate(self, prompt, sep = "|", max_tokens = 100, excluded = [[40, 19]],
lookback = 1, stop_tokens = [5, 27, 33], sample = False, top_k = 3):
tokens = tokenizer.encode(prompt + sep)
tokens_generated = []
while tokens[-1] not in stop_tokens and len(tokens) < max_tokens:
output = model.forward(input_ids=torch.tensor([tokens]).to(device)).logits[0,-1]
output = torch.softmax(output, dim = 0)
candidates = torch.topk(output, k = top_k)
if sample:
indices = candidates.indices
scores = candidates.values
next_token = indices[torch.multinomial(scores, 1)[0].item()]
else:
next_token = candidates.indices[0]
next_token = next_token.item()
sub_tokens = tokens_generated[-lookback:] + [next_token]
if len(tokens_generated) >= (lookback + 1) and next_token in tokens_generated[-(lookback + 1):]:
next_token = candidates.indices[1]
next_token = next_token.item()
elif len(tokens_generated) >= lookback and self._check_sublist(tokens_generated, sub_tokens):
next_token = candidates.indices[1]
next_token = next_token.item()
tokens = tokens + [next_token]
tokens_generated = tokens_generated + [next_token]
for ex_lst in excluded:
tokens = self._exclude_sublist(tokens, ex_lst)
output = tokenizer.decode(tokens, skip_special_tokens=True)
output = output.split(sep)[-1].strip()
output = output[0].upper() + output[1:]
if output[-1] == tokenizer.decode(stop_tokens[0]):
output = output[:-1]
return output
diablo = Diablo(tokenizer = tokenizer, model = model)
prompt = "Ciao, come stai?"
# setting "sample = True" the model will be more creative but occasionally less accurate
print("OUTPUT:", diablo.generate(prompt, sample = False))
# OUTPUT: Sto bene, grazie
Limitations
This model has been mainly trained on machine-translated (and synthetic) conversational data, so it might behave erratically when presented with prompts which are too far away from its training set. Moreover, the heterogeneous nature of the pretraining dataset, together with the limits of the conversational data, might lead the model to produce biased or offensive content with respect to gender, race, ideologies, and political or religious beliefs. These limitations imply that the model and its outputs should be used with caution, and should not be involved in situations that require the generated text to be fair or true.
References
[1] https://arxiv.org/abs/2005.14165
[2] https://arxiv.org/abs/2112.10668
[3] https://arxiv.org/pdf/2004.13637.pdf
License
The model is released under MIT license