|
--- |
|
library_name: transformers |
|
datasets: |
|
- Svngoku/xP3x-Kongo |
|
language: |
|
- kg |
|
metrics: |
|
- bleu |
|
pipeline_tag: text-generation |
|
tags: |
|
- africa |
|
- languages |
|
--- |
|
|
|
# Kongo Llama Experiment |
|
|
|
<!-- Provide a quick summary of what the model is/does. --> |
|
|
|
## Model Details |
|
|
|
- `Tokenizer` |
|
```py |
|
from transformers import PreTrainedTokenizerFast |
|
|
|
# Assuming your custom tokenizer is `tokenizer` |
|
wrapped_tokenizer = PreTrainedTokenizerFast( |
|
tokenizer_object=tokenizer, |
|
bos_token="[BOS]", # Replace with your special tokens |
|
eos_token="[EOS]", # Replace with your special tokens |
|
unk_token="[UNK]", |
|
pad_token="[PAD]" |
|
) |
|
|
|
# Ensure padding is applied to the right side (used in causal language modeling) |
|
wrapped_tokenizer.padding_side = "right" |
|
|
|
``` |
|
|
|
- `Model` |
|
```py |
|
from transformers import LlamaConfig, LlamaForCausalLM |
|
|
|
config = LlamaConfig( |
|
vocab_size=len(wrapped_tokenizer), # Get vocab size from the wrapped tokenizer |
|
hidden_size=512, # Adjust model size as needed |
|
intermediate_size=1024, |
|
num_hidden_layers=8, # Set number of layers and heads |
|
num_attention_heads=8, |
|
max_position_embeddings=512, |
|
rms_norm_eps=1e-6, |
|
initializer_range=0.02, |
|
use_cache=True, |
|
pad_token_id=wrapped_tokenizer.pad_token_id, |
|
bos_token_id=wrapped_tokenizer.bos_token_id, |
|
eos_token_id=wrapped_tokenizer.eos_token_id, |
|
) |
|
|
|
model = LlamaForCausalLM(config) |
|
``` |
|
|
|
- `Trainer` |
|
```py |
|
from transformers import TrainingArguments, Trainer |
|
|
|
# Define training arguments |
|
training_args = TrainingArguments( |
|
output_dir="kongo-llama", # Output directory for model and checkpoints |
|
num_train_epochs=1, |
|
per_device_train_batch_size=8, |
|
learning_rate=5e-5, |
|
warmup_steps=500, |
|
weight_decay=0.01, |
|
logging_dir="./logs", |
|
logging_steps=10, |
|
save_steps=1000, |
|
) |
|
|
|
|
|
trainer = Trainer( |
|
model=model, # Your model instance |
|
args=training_args, # Training arguments |
|
train_dataset=dataset, # Tokenized dataset with input_ids and labels |
|
tokenizer=wrapped_tokenizer, # Wrapped tokenizer |
|
data_collator=data_collator, # Data collator for causal language modeling |
|
) |
|
```` |
|
|
|
### Model Description |
|
|
|
<!-- Provide a longer summary of what this model is. --> |
|
|
|
This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated. |
|
|
|
- **Developed by:** [More Information Needed] |
|
- **Funded by [optional]:** [More Information Needed] |
|
- **Shared by [optional]:** [More Information Needed] |
|
- **Model type:** [More Information Needed] |
|
- **Language(s) (NLP):** [More Information Needed] |
|
- **License:** [More Information Needed] |
|
- **Finetuned from model [optional]:** [More Information Needed] |
|
|
|
### Model Sources [optional] |
|
|
|
<!-- Provide the basic links for the model. --> |
|
|
|
- **Repository:** [More Information Needed] |
|
- **Paper [optional]:** [More Information Needed] |
|
- **Demo [optional]:** [More Information Needed] |
|
|
|
## Uses |
|
|
|
```py |
|
# Use a pipeline as a high-level helper |
|
from transformers import pipeline |
|
pipe = pipeline("text-generation", model="Svngoku/kongo-llama") |
|
|
|
pipe( |
|
"Mbote, mono ", |
|
max_length=150, |
|
num_beams=5, |
|
temperature=0.7, |
|
do_sample=True, |
|
top_p=0.95 |
|
) |
|
|
|
``` |
|
|
|
```sh |
|
[{'generated_text': 'Mbote, mono na ngambu ya mpila ya bo ke monisa nde bantu yonso zole yina kaka na kati ya bo ke sadilaka yo mosi ve kana bo ke vandaka ti yo yina, to bima ya nkaka ya bo ke salaka sambu na bana ya zulu.'}] |
|
|
|
``` |
|
|
